Arrays and RAIDZ and spares, oh my!

So ... when I first got my current main server, babylon4 (which was new to me, but not by any stretch new hardware — including its disks), I set it up with Solaris 10u5 x86.  I installed a mirrored pair of 2.5" 80GB SATA laptop drives in the single bay intended for a boot drive, and configured the main array of twelve 300GB SATA disks as a ZFS RAIDZ2 pool.  RAIDZ2, with two parity disks, should be able to survive any two drive failures and continue operating in degraded mode.

So, about two months later, we had an overnight cascade failure of three drives, and the array went down.

I reconfigured the remaining nine drives as an eight-drive RAIDZ2 plus a single hot spare, and restored all the data.  Over the next few months, one more drive failed; the hot spare was automatically pulled in as a replacement, just as it should.  When I got my hands on replacements for the (by now) four failed drives, I added them in as hot spares.  We've had no further failures.

Recently something hosed the boot archive and took the system down.  All the zpools were intact, so we didn't lose any user data, but I ended up reinstalling with Solaris 10u8.  Then, not long after, Sun ... er ... Oracle released Solaris 10u9 as a developer release.  Completely unsupported, sure, but I can't spare the money Oracle wants for a support contract anyway, so what's the difference?  So I live-upograded the machine to u9, and upgraded all the zpools to ZFS version 22.  But, ZFS 22 now supports RAIDZ3, an even-higher-reliability format for large disk pools, using three independent parity stripes.

So, yesterday I made one last differential backup, then blew away the RAIDZ2 zpool, reconfigured the array as an eleven-drive RAIDZ3 plus a single hot spare, and restored all the data overnight.  RAIDZ3 and a hot spare may be a little paranoid ... but I just increased the size of the working set by 600GB, and it should be able to survive up to four drive failures, as long as it finishes rebuilding the first failed drive on the hot spare before the fourth drive fails.

As an incidental side note, I note that the statfs() call in a 32-bit Linux kernel overflows when called on a filesystem with more than 2TB of free space...

Something new I learned today

Ignosticism, or igtheism, is the theological position that every other theological position (including agnosticism) assumes too much about the concept of god and many other theological concepts.

I've been an ignostic for many years, except that I called it "meta-agnosticism" because I didn't know that it had already been invented.  robhansen, aka cipherpunk, just pointed it out to me tonight.  I explained what I meant by meta-agnosticism, and he said, "Have you ever heard of ignosticism?"

Well, sort of, yes.  I just didn't know what it was called.  Now, I do.

The good, the bad, and the ...

There are good ideas, and there are bad ideas; and then there are ...

Well, and then there is this.

"This is the original Barrel Button.  It can be attached to any shoe or boot that has laces (snap closure).  It allows you to securely rest the muzzle of your unloaded shotgun.  [...]"

This is such an unspeakably awful idea that I really have no words to express the massive fail here.  Much though I'm opposed to frivolous liability suits, I have to agree with robhansen's comments:

If I were on a jury, I have to admit, I'd be torn about whether to hold the Barrel Button liable for someone losing their foot.

On the one hand, your own damn fault for letting the muzzle cross your foot.

On the other hand, given there is literally no safe way to use the product…

For those of you among my readers who are not initiates of the proper ways of safe gun-handling, allow me to cite the first two of the four firearms safety laws propounded by the late, lamented Col. Jeff Cooper (USMC, Ret.):

1. All guns are always loaded.  (Or, to clarify his intention, always treat all firearms as though loaded at all times.)
2. Never let the muzzle cover anything you are unwilling to destroy.

Which makes a device intentionally designed to rest the muzzle of a firearm (that you were CERTAIN was unloaded) securely upon a part of your body ... yeah.  It's that stupid.