There is a difference in terms of the severity of data loss after a cache failure and a power loss, but the basic problem is still the same.

The RAID is allowed to lose data on power loss. It is also allowed to ignore parity during reads, even if it’s still checking for consistency - thus reads do *not* get verified during the background check (this is beyond the guarantees made by a RAID system, which are all about protecting your data from a drive failure, not a system failure).

The guarantee is that the only time you can read corrupt data is immediately after a power failure; you are supposed to replay journals, or otherwise validate your data, to ensure that the corruption hasn’t damaged your data. If parity is damaged, *and* the background check does not reach the damaged parity before a drive failure (assuming the drive with the damaged parity block is not the drive that fails), you read correct data (and thus don’t correct it from the journal), but later see it corrupted when the data block is rebuilt from parity.

> you want to do the parity check in the background while the machine is back in service in many cases

In that case, you have to verify every read you do until the background check is complete.
But isn’t this supposed to be caught by replaying journals?

> within the guarantees of the RAID system.

I thought you just said a RAID was allowed to lose data on power loss?

When you’ve had an unexpected power fail event (whether we’re talking loss of system power without NVRAM cache, or NVRAM battery failure), the RAID is permitted to corrupt data; a *good* RAID implementation guarantees that it will only corrupt areas that have been written to in the last n seconds, for some (documented) value of n. For example, if my disks guarantee that a write has completed when the drive returns completed to a write command (and not that the write has been cached), the RAID may guarantee that only the last 8MB of writes are at-risk, assuming that the data has not been synced to disk (e.g. by fsync).

Higher levels of software build on these guarantees to avoid data loss, and do things like journalling writes (or BSD-style soft updates), to ensure that once the filesystem claims something’s on disk, it’s safe. In turn, application software like databases or mail servers ensures that it stays within these guarantees, and can do things like replaying journals to ensure that the data is correct, or at least consistent with claims made to the outside world (e.g. a mail server doesn’t give a 200 OK response to incoming mail until it’s confident that the mail is safely stored - fsync is often used on UNIX-likes).

Parity corruption just means that one disk failed to write all its blocks before power loss, and in this case it happened to be the disk writing parity, not the disk writing data. The trouble with insisting on a complete parity check before you bring the machine back up is that it’s slow; you want to do the parity check in the background while the machine is back in service in many cases (e.g. an outgoing SMTP server, where the RAID stores the outgoing queue), to reduce downtime to a minimum. If you don’t catch the faulty parity before a data drive fails, you risk losing data that you thought was safe, within the guarantees of the RAID system.

> The trouble with the situation where the parity is corrupt but not the data is that my data check after power loss shows that I’m A-OK.

What exactly does that check do? After power loss you should do a complete parity check.
And how did the parity get corrupt (while the data remained intact)?

A RAID system should provide a certain minimum set of guarantees; an important guarantee that the only time when data may be lost is when power goes unexpectedly. In particular, for models without NVRAM, any power loss may induce data loss. For models with NVRAM, data loss may occur if the NVRAM module loses power for more than a specified time (usually 30 days). If you go outside these conditions for any reason, you are advised to check your data, but once you have checked your data, no further corruption should occur.

The trouble with the situation where the parity is corrupt but not the data is that my data check after power loss shows that I’m A-OK. Some time later (possibly long enough that I’ve forgotten ever losing power), I have a disc failure while online; I hot-swap the drive, which *should* maintain my data (thanks to the data loss guarantee the RAID provides). However, the silently damaged parity means that at *this* stage, when I’m working within the limits of the guarantees RAID provides, I lose data. This is unacceptable behaviour, as the guarantee is broken; note that it *would* be acceptable to corrupt data when the power goes, so long as a data check then shows that there’s been trouble.

Fair enough, but I don’t see how that relates to my comments.

Simon: Scrubbing is good, but is typically only run from a cron job weekly (or some other infrequent interval). Of course if you do an entire RAID rebuild operation after every power failure (as Linux software RAID is prone to do) then you get some consistency at the cost of performance.

Ole: Auction sites have HP machines with hardware RAID at quite reasonable prices. HP supports them for a minimum of 5 years (within which time they guarantee that they will provide replacement hardware to read the disks). Also it’s not impossible to recreate an unknown RAID format. Some time ago I attended a lecture on how to determine the RAID format when you get a set of disks from an unknown machine. You have to recognise some patterns in the data, a large file with a known format is good for this. Then as there isn’t much variation in RAID formats you just work out the stripe size and the order of the disks and it’s simple to write a program to dump all the data to a single block device or file.

Olaf: When considering the mathematical issues we consider only a single stripe of a RAID-5 which has one block of data for parity and N-1 blocks of data. The fact that each successive stripe rotates the order by one is not relevant when considering single-stripe issues. Another lacking feature in Linux software RAID is the ability to read all disks and compare the result. Of course when you can’t actually do anything useful once you know the data is bad the deficiency isn’t so bad. It’s a pity that you can’t have a 3 disk mirror and take the majority vote or have RAID-6 read from all disks (with the ability to correct a single block error).