Trials and tribulations

[unixronin]

So, let's see:  At the last point mentioned yesterday, about 1030, I'd just brought babylon5 back up after rather abruptly halting it by fumbling a power connector while hot-swapping in a SCSI-III disk as a replacement for one of the two mirrored boot disks, which was showing signs of impending spindle failure.  During the course of associated troubleshooting, it appeared the remaining CDROM drive in the machine had also failed.

Well, the good news is that once ALL the shuffling of SCSI devices was done with, it turned out the CDROM was still good.  In fact, what was going on was that the internal SCSI Zip drive on the same SCSI bus, always troublesome, had failed — and failed badly enough that it was screwing up both the Plextor CDROM and the Exabyte VXA1 tape drive that it shared the second SCSI controller with.  So I don't need to replace the CDROM.  (Not that it isn't a little tempting to get a CD/DVD writer into the machine.)

The bad news is that about 15 minutes after I started dd'ing data from the failing mirror disk (/dev/sda, aka /dev/sd/c0b0t0u0, aka /dev/scsi/host0/bus0/target0/lun0/disc) to the new disk (/dev/sdc, aka ... well, you get the idea), the system locked up for no apparent reason whatsoever.  In doing so, it apparently badly hosed the RAID superblock on /dev/sda1 ... and this caused ALL KINDS of trouble, especially since I don't have a bootable live-filesystem Linux CD around that's recent enough to grok ext3, let alone devfs.  I ended up having to recover the superblock on the partition, manually unmirror the disks, fix a couple of different issues on the disks, and eventually get the disks into good enough shape using a Slackware 7 live CD that I was able to boot the machine off its own disks again somewhere around 1900.  I still had a bunch of work to do after that before everything was working properly.  I compiled a new 2.4.34 kernel while I was at it, having discovered that one of the bizarre problems I'd been fighting is a known problem:  Sometimes, a Linux kernel compiled with both XFS support and metadevice support, using ext3 filesystems, will mistakenly try to mount ext3 filesystems on md devices as XFS filesystems, fail to find the necessary structures, and die from a kernel panic.  It appears at present the only way to avoid this problem is "Don't use ext3 filesystems on metadevices if you have XFS support in your kernel".  So when I installed the new kernel, I deleted both XFS and reiserfs support, which made for a significantly smaller kernel.  (A while back, I'd intended to do some performance comparisons between reiserfs, XFS and JFS, with a vague intention of switching to either XFS or JFS, and never got around to doing it.  I should probably have removed the JFS support as well while I was at it.)

In the meantime, someone had managed to overflow the toilet in the downstairs bathroom, flooding the bathroom and hallway (both tiled, fortunately, but there was plenty of stuff to get wet including the mat that's about an inch or two in each direction too big for said hallway).  This necessitated getting out the steam vac to clean up the flood, from which I learned that it's below freezing up in the attic (we hadn't emptied the wastewater tank last time we used it, and it was frozen solid).  I now also need to obtain a replacement for the short piece of hose that connects the back of the machine to the main body, because it had apparently hardened from age and it broke while I was vacuuming up the water.  (Fortunately that only affects the hand tools.  Unfortunately, I don't know whether I can actually get the replacement hose, or whether I can replace it without tearing apart the entire machine to do it.)

That's not the sum of the ways in which yesterday sucked.  But it's quite enough to be going on with.

Comments

[ithildae.livejournal.com]

Do I need to mention the steam vac and suck being used in the same post?

[unixronin.livejournal.com]

I have to admit that particular juxtaposition was not, this time, intentional.

[smandal.livejournal.com]

Rule of thumb:

Make a new rescue CD whenever you upgrade your kernel, or at least test the old one.

In my case, I just keep my laptop Gentoo profile and Gentoo LiveCD in sync (currently "2006.1").

[unix-jedi.livejournal.com]

Oooh.

And don't under any circumstances, use a old 2005 LiveCD when the overnight, overlooked grub installation stops booting (Hey, quick boot, --oneshot grub with /boot mounted, no problem).....

When / is reiser.

(2005 booted, mounted the reiserfs, and then *rolled back* all the existing changes..... meaing things were fooqued). Reiser fsck *did* eventually fix things, but then all these files weren't correct.. you know, like libc, little things like that. (Finally, just --oneshotted anything I'd installed recently and everything now works).

[unixronin.livejournal.com]

I've never used reiserfs, and I keep hearing things about it that convince me that I never should. libc had been updated HOW long ago and the changes hadn't been flushed to disk yet...?

[unix-jedi.livejournal.com]

Err, no, this wasn't reiser's fault.

It was mine. :) Maybe transactions were the wrong terminology - but basically, the old version saw the changelog, thought it in error, and rolled it back.

The problem was that the old version read it wrong... and there's something else I didn't mention that was really dumb of me.... This is a laptop, and I'd suspended it..... Yeah, I saw that grub was updating, but I didn't think about what it would mean...

So no, this was a complete dumbass stunt on my part... As much as I'd LIKE to blame somebody else. :)

I've never tried XFS or JFS - I've always had great success with Reiser.. It's always.. KILLED THE COMPETITION. Try the steak, I'll be here all week!

[unixronin.livejournal.com]

I haven't tried them yet, either, but I still want to. ReiserFS is pretty good, but I'm given to understand that it's performance actually isn't that great according to benchmarks, and some operations (deletes, for example) incur very high CPU usage. I've also heard of a lot of people getting bitten by Rieser bugs.

Both JFS and XFS out-perform Reiser overall, according to the benchmarks; which is the better of the two depends largely on what you're doing.