Copy-on-write, checksums and self-healing

1 Never overwrite a live block

When a block changes, ZFS writes the new version somewhere else and leaves the old one where it is. The pointer to the block changes, so its parent changes and is written elsewhere too, and so on up to the root. The old tree stays whole until a single write, the new uberblock, switches the pool from one tree to the other. Rewrite a block step by step, and cut the power before the commit.

Block pointers of the newest tree
Pointer	Address	Birth	Checksum

Changed pointers are marked. Checksums are real fletcher4 values over the demo blocks (last 8 hex digits of 64 shown).

Text version of this figure

The tree has an uberblock at the top, a root indirect block R, two indirect blocks I0 and I1, and four data blocks D0 to D3. Each block pointer holds the child’s address, its birth txg and its checksum.
Rewriting a data block writes a new copy at a new address; the old copy stays where it is.
Its parent now needs a new pointer (new address, new checksum), so the parent is copied too, and so on up to R.
Writing the new uberblock to the next slot of the ring is the commit. Before it, a power cut leaves the old tree intact and current; after it, the new tree is the pool.
Replaced blocks are freed, unless a snapshot still references them: then they stay on the deadlist.

2 The checksum lives in the parent

Each block pointer, the 128-byte blkptr_t, carries the 256-bit checksum of the block it points to. A block never vouches for itself: its parent does, and the parent’s parent vouches for the parent, up to the uberblock. Reading a block means recomputing its checksum and comparing it with the one in the pointer that led to it. On a mirror, a mismatch sends the read to the other disk, and the good copy is written back over the bad one.

Pointers in the parent block: the expected checksums

Text version of this figure

A file of four blocks, D0 to D3, sits on a two-way mirror: disk A and disk B each hold a full copy. The parent block holds, in each block pointer, the fletcher4 checksum of the block it points to.
Bit rot: one bit of D1 flips on disk A. The disk returns the bytes without error. Recomputing fletcher4 over them gives a different value from the one in the pointer: ECKSUM.
Lost write: disk A acknowledged the write of the new D2 but never performed it. Under copy-on-write that write went to a new address, not over the old D2, so the address still holds whatever was there before: stale bytes of an earlier block, since freed. Those bytes can be consistent with themselves; only the checksum in the parent, which describes the new D2, reveals them.
On a mismatch the mirror reads the other disk. If that copy matches, it is returned to the application and written back over the bad copy: the pool heals itself on a normal read.
If both copies are damaged, the read fails with an error rather than return wrong data. Each mismatch is counted against the disk that returned it.

The idea, at three depths

In one sentence

ZFS never overwrites live data and keeps each block’s checksum in the pointer to it, so a crash cannot leave the pool half written and a disk cannot return wrong data unnoticed.

In one paragraph

Every write goes to a new place: the block gets a new address, which changes its parent’s block pointer, which changes the parent, up to the root. Nothing the current uberblock reaches is ever modified, so at every instant the disks hold a complete, consistent tree, and writing one new uberblock commits the next one. Each block pointer also carries the checksum of its child, which makes the tree a Merkle tree: one trusted root vouches for every block below it. A mismatch on read is a detected error, never silent corruption, and with a second copy (a mirror, or up to three addresses in one block pointer) ZFS returns the good copy and repairs the bad one.

In detail

Where a write becomes a new block. In the write pipeline, zio_write_compress decides between a rewrite in place and a fresh allocation.

zio_write_compress Excerpt of module/zfs/zio.c, lines 2086–2105, OpenZFS zfs-2.4.4. View on GitHub

	/*
	 * The final pass of spa_sync() must be all rewrites, but the first
	 * few passes offer a trade-off: allocating blocks defers convergence,
	 * but newly allocated blocks are sequential, so they can be written
	 * to disk faster.  Therefore, we allow the first few passes of
	 * spa_sync() to allocate new blocks, but force rewrites after that.
	 * There should only be a handful of blocks after pass 1 in any case.
	 */
	if (!BP_IS_HOLE(bp) && BP_GET_BIRTH(bp) == zio->io_txg &&
	    BP_GET_PSIZE(bp) == psize &&
	    pass >= zfs_sync_pass_rewrite) {
		VERIFY3U(psize, !=, 0);
		enum zio_stage gang_stages = zio->io_pipeline & ZIO_GANG_STAGES;

		zio->io_pipeline = ZIO_REWRITE_PIPELINE | gang_stages;
		zio->io_flags |= ZIO_FLAG_IO_REWRITE;
	} else {
		BP_ZERO(bp);
		zio->io_pipeline = ZIO_WRITE_PIPELINE;
	}

L2094–L2101 In-place rewrite is allowed only for a block born in the very txg being synced (BP_GET_BIRTH(bp) == zio->io_txg), of unchanged size, from sync pass zfs_sync_pass_rewrite (2) on. No committed uberblock references such a block yet, so overwriting it touches no live data. This is the one exception to copy-on-write for blocks of the tree.
L2102–L2104 Every other write zeroes the block pointer and takes the full write pipeline, whose DVA_ALLOCATE stage normally gives the block a new address: zio_dva_allocate asserts the pointer is a hole before calling metaslab_alloc. Just below, three cases change the pipeline: a block that compresses to nothing becomes a hole, dedup takes the DDT pipeline, and nopwrite adds its own stage.

When the write completes, dbuf_write_done calls dsl_dataset_block_kill on the old pointer and dsl_dataset_block_born on the new one. The old block is freed only if it was born after the most recent snapshot; otherwise it goes on the dataset’s deadlist, and the snapshot keeps it.

Where the checksum is checked. On write, zio_checksum_compute stores the computed value in bp->blk_cksum. On read, the block is checked against that field.

zio_checksum_error_impl Excerpt of module/zfs/zio_checksum.c, lines 501–538, OpenZFS zfs-2.4.4. View on GitHub

	} else {
		byteswap = BP_SHOULD_BYTESWAP(bp);
		expected_cksum = bp->blk_cksum;
		ci->ci_func[byteswap](abd, size,
		    spa->spa_cksum_tmpls[checksum], &actual_cksum);
	}

	/*
	 * MAC checksums are a special case since half of this checksum will
	 * actually be the encryption MAC. This will be verified by the
	 * decryption process, so we just check the truncated checksum now.
	 * Objset blocks use embedded MACs so we don't truncate the checksum
	 * for them.
	 */
	if (bp != NULL && BP_USES_CRYPT(bp) &&
	    BP_GET_TYPE(bp) != DMU_OT_OBJSET) {
		if (!(ci->ci_flags & ZCHECKSUM_FLAG_DEDUP)) {
			actual_cksum.zc_word[0] ^= actual_cksum.zc_word[2];
			actual_cksum.zc_word[1] ^= actual_cksum.zc_word[3];
		}

		actual_cksum.zc_word[2] = 0;
		actual_cksum.zc_word[3] = 0;
		expected_cksum.zc_word[2] = 0;
		expected_cksum.zc_word[3] = 0;
	}

	if (info != NULL) {
		info->zbc_checksum_name = ci->ci_name;
		info->zbc_byteswapped = byteswap;
		info->zbc_injected = 0;
		info->zbc_has_cksum = 1;
	}

	if (!ZIO_CHECKSUM_EQUAL(actual_cksum, expected_cksum))
		return (SET_ERROR(ECKSUM));

	return (0);

L502–L505 The expected value is bp->blk_cksum, taken from the block pointer the reader followed; the actual value is recomputed over the bytes just read.
L515–L526 Encrypted blocks other than objsets: half of the 256 bits is the encryption MAC, which decryption verifies, so only the other 128 bits are compared here.
L535–L536 A mismatch returns ECKSUM. On a mirror this runs in each child read, not in the parent: zio_vdev_child_io moves the CHECKSUM_VERIFY stage down to the child. Each error is counted on the leaf vdev.

The chain stops at the uberblock: ub_rootbp has no parent pointer. Each uberblock slot is protected instead by an embedded SHA-256 “label” checksum whose verifier is the slot’s own offset on disk. Gang headers and ZIL blocks (zilog, zilog2) also carry embedded checksums, and checksum=off exists. The default, checksum=on, means fletcher4: four 64-bit running sums over the block read as 32-bit words. The figures above compute it for real.

Where the mirror heals. In vdev_mirror_io_done, if no child has returned good data yet, vdev_mirror_child_select picks another child and a new read is issued. If none is left, the read fails with an error. Repair needs at least one good copy, a writable pool, and either an unexpected error, a resilver, or a resilvering scrub (L883–L886). Then:

vdev_mirror_io_done Excerpt of module/zfs/vdev_mirror.c, lines 900–934, OpenZFS zfs-2.4.4. View on GitHub

			if (mc->mc_error == 0) {
				vdev_ops_t *ops = mc->mc_vd->vdev_ops;

				if (mc->mc_tried)
					continue;
				/*
				 * We didn't try this child.  We need to
				 * repair it if:
				 * 1. it's a scrub (in which case we have
				 * tried everything that was healthy)
				 *  - or -
				 * 2. it's an indirect or distributed spare
				 * vdev (in which case it could point to any
				 * other vdev, which might have a bad DTL)
				 *  - or -
				 * 3. the DTL indicates that this data is
				 * missing from this vdev
				 */
				if (!(zio->io_flags & ZIO_FLAG_SCRUB) &&
				    ops != &vdev_indirect_ops &&
				    ops != &vdev_draid_spare_ops &&
				    !vdev_dtl_contains(mc->mc_vd, DTL_PARTIAL,
				    zio->io_txg, 1))
					continue;
				mc->mc_error = SET_ERROR(ESTALE);
			}

			zio_nowait(zio_vdev_child_io(zio, zio->io_bp,
			    mc->mc_vd, mc->mc_offset,
			    zio->io_abd, zio->io_size, ZIO_TYPE_WRITE,
			    zio->io_priority == ZIO_PRIORITY_REBUILD ?
			    ZIO_PRIORITY_REBUILD : ZIO_PRIORITY_ASYNC_WRITE,
			    ZIO_FLAG_IO_REPAIR | (unexpected_errors ?
			    ZIO_FLAG_SELF_HEAL : 0), NULL, NULL));
		}

L900–L904 Children that returned good data are never rewritten: the comment just above warns that rewriting the only good copy during a power loss could lose it.
L905–L924 A child that was not tried is repaired only during a scrub, for indirect or dRAID-spare vdevs, or when its DTL says this txg is missing there.
L927–L933 The good buffer zio->io_abd is written to each damaged child as a ZIO_FLAG_IO_REPAIR write, flagged ZIO_FLAG_SELF_HEAL when the cause was an unexpected error.

What the figures leave out

The figures show one file, so the tree is short: in a pool it runs from the uberblock through the MOS, the dataset and the object’s dnode. They leave out sync passes, gang blocks, dedup, nopwrite, encryption MACs, and the mirror’s load-based choice of the side to read first (here, always disk A). The uberblock ring has 32 slots, as on 4K-sector disks. Checksums are real fletcher4 values over the demo blocks; addresses and txg numbers are made up.