Uberblok

Room 5

The vdevs

A pool is not a list of disks. It is mirrors, RAID-Z groups, dRAID, a log device, a special device, hot spares and disks being replaced, combined as the administrator chose, and any of them can change while the pool runs.

If the I/O path had a branch for each of these, and for each combination (a disk being replaced inside a raidz, a hot spare inside a mirror), it would never be finished. The code has no such branch. It builds one tree of nodes, vdev_t, hanging from spa_root_vdev, and gives each node a table of functions, vdev_ops_t. A write to the pool walks down the tree, and each node decides, with its own io_start, how the I/O goes on to its children. The allocator, for its part, sees only the top-level vdevs, grouped in classes by what they are for.

1 One tree, one table per node

The figure’s pool has three top-level vdevs for data, a mirror, a raidz1 of three disks and a dRAID of five, and can have a mirror for the log, a mirror as special vdev and a disk for dedup. Select a node to see the table it points to: its type is nothing but that table. “Log”, “special” and “dedup” are not types: they are the allocation bias of a top-level vdev. The hot spares and the cache device are not in the tree at all: they are two lists beside it. Write blocks, and follow each I/O down to the disks.

rootrootmirror-0mirror1sdadisk32 sectorssdbdisk32 sectorsraidz1-1raidzsdcdisksdddisksdediskdraid1:2d:5c:1s-2draidsdfdisksdgdisksdhdisksdidisksdjdiskmirror-3mirrorsdkdisksdldiskmirror-4mirror1sdmdisksdndisk

Beside the tree

spa_spares, hot spares code

  • sdp disk, available
  • draid1-2-0 dspare, available, distributed: space on every child of the dRAID

spa_l2cache, cache devices (L2ARC)

  • sdq disk

The last block

Block 2, a large file block, txg 101: the normal class.

  1. A data block of 128 KiB, larger than special_small_blocks (0): the normal class. code
  2. 1 DVA: a file’s data block with copies=1. code DVA 0 on mirror-0. code

The I/O, down the tree

The logical zio has no vdev: zio_vdev_io_start hands it to vdev_mirror_io_start, with one child per DVA (1 DVA). code

  • DVA 0 mirror-0 mirror vdev_mirror_io_start: to every child code
    • sda disk vdev_disk_io_start: 32 sectors
    • sdb disk vdev_disk_io_start: 32 sectors

The selected node

sdd code

FieldValue
vdev_ops&vdev_disk_ops
vdev_op_type"disk"
vdev_op_io_startvdev_disk_io_start
vdev_op_io_donevdev_disk_io_done
vdev_op_psize_to_asizevdev_default_asize
vdev_op_need_resilverNULL
vdev_op_leafB_TRUE
vdev_parentraidz1-1
vdev_topraidz1-1
vdev_children0
Metaslab groupsnone: only top-level vdevs are allocated from

Allocation classes

ClassTop-level vdevsBlocks
normalmirror-0, raidz1-1, draid1:2d:5c:1s-21
logmirror-30
specialmirror-41
dedupnone0
embedded logmirror-0, raidz1-1, draid1:2d:5c:1s-20
special embedded logmirror-40
Text version of this figure

The tree, from the root:

  • root root
  • mirror-0 mirror, normal
  • sda disk
  • sdb disk
  • raidz1-1 raidz, normal
  • sdc disk
  • sdd disk
  • sde disk
  • draid1:2d:5c:1s-2 draid, normal
  • sdf disk
  • sdg disk
  • sdh disk
  • sdi disk
  • sdj disk
  • mirror-3 mirror, log
  • sdk disk
  • sdl disk
  • mirror-4 mirror, special
  • sdm disk
  • sdn disk

Beside the tree: hot spares sdp and draid1-2-0; cache sdq.

The last block:

Block 2, a large file block, txg 101: the normal class.

  1. A data block of 128 KiB, larger than special_small_blocks (0): the normal class.
  2. 1 DVA: a file’s data block with copies=1. DVA 0 on mirror-0.

The I/O, down the tree

The logical zio has no vdev: zio_vdev_io_start hands it to vdev_mirror_io_start, with one child per DVA (1 DVA).

  • DVA 0 mirror-0 mirror vdev_mirror_io_start: to every child
    • sda disk vdev_disk_io_start: 32 sectors
    • sdb disk vdev_disk_io_start: 32 sectors

2 Where a block lands: allocation classes

Each top-level vdev joins one metaslab class, by its bias: normal, log, special or dedup. Every vdev but a log also gets a group in an “embedded log” class, room for the ZIL without a log device, but only a normal or special vdev with more than 64 metaslabs (a tunable) puts a metaslab in it: a dedup vdev’s group stays empty, and so does a small vdev’s. For each write, spa_preferred_class picks the class. A DDT block goes to dedup, else to special, else to normal. Metadata and the indirect blocks of files go to special. A file’s data block goes to special only if it is no larger than the dataset’s special_small_blocks, 0 by default, and only while special has used less than 75% of its space: the last 25% is kept for metadata. Everything else goes to normal. A ZIL block has its own order, log, special’s embedded log, special, embedded log, normal. When a class is full, the allocation asks again, and the class that failed is skipped.

A block can also have several copies, one DVA each, on consecutive vdevs of its class: metadata has one more copy than the dataset’s copies (with redundant_metadata=all, the default), and the DDT, which lives in the pool’s MOS, three, the cap. In the figure, drop each kind of block with and without a special vdev, set special_small_blocks to 16 KiB, then fill special.

3 A replacement is a mirror for a while

zpool replace does not swap one disk for another. It puts a new node in the old disk’s place, a replacing node, with the old disk as its first child and the new one as its last. A hot spare does the same with a spare node. Both tables use the mirror’s functions, so the I/O path needs no new code: a write goes to both disks, and a read skips the new disk for every block it does not have yet. When the resilver is done, the old disk is detached and the replacing node goes away. A spare node stays until zed brings the original disk back online or its fault is cleared, until the original disk is itself replaced, or until the administrator detaches one of the two; in the first two cases, the end of a resilver detaches the spare. Select a disk in the figure, replace it, write and read, then finish the resilver. Replace the dedup disk too: it is a top-level vdev, and the replacing node takes its role.

4 dRAID: permutations instead of fixed groups

A raidz group always uses the same disks. A dRAID spreads groups of a fixed width over all its children, through a table of permutations that the code generates from a fixed seed and checks against a checksum it stores, because “existing pools depend on the same mapping always being generated”. The figure’s draid1:2d:5c:1s has groups of three (one parity, two data) over five children, and one distributed spare: in every row of the table, one child holds no column but the spare’s space. The figure computes the same table as the code, and the same checksum. Write several large blocks and watch each group land on a different set of children; then replace a dRAID child with the distributed spare.

The idea, at three depths

In one sentence

Every kind of device and every replacement is a node of one tree with its own table of functions, so an I/O walks down the tree and each node fans it out its way, while the allocator chooses, by the block’s kind, a class of top-level vdevs to put it in.

In one paragraph

A vdev_t has a parent, children, the top-level vdev it belongs to, and a pointer to a vdev_ops_t that is its type. A block’s logical zio has no vdev: zio_vdev_io_start hands it to the mirror’s io_start with the block’s DVAs as children; each child zio then reaches its top-level vdev, whose io_start writes a mirror’s every child, a raidz’s columns, or a dRAID group placed by a fixed permutation of its children, until the leaves issue the disk I/O. Only top-level vdevs own metaslabs, in the class of their bias; spa_preferred_class sends DDT blocks to dedup, metadata to special, small data blocks to special while it is under 75%, and the rest to normal, and zio_dva_allocate falls back to another class when one is full. A replacement inserts a replacing or spare node above the old disk, with the mirror’s functions, until the resilver ends.

In detail

A node of the tree.

struct vdev Excerpt of include/sys/vdev_impl.h, lines 239–269, OpenZFS zfs-2.4.4. View on GitHub 
	uint64_t	vdev_state;	/* see VDEV_STATE_* #defines	*/
	uint64_t	vdev_prevstate;	/* used when reopening a vdev	*/
	vdev_ops_t	*vdev_ops;	/* vdev operations		*/
	spa_t		*vdev_spa;	/* spa for this vdev		*/
	void		*vdev_tsd;	/* type-specific data		*/
	vdev_t		*vdev_top;	/* top-level vdev		*/
	vdev_t		*vdev_parent;	/* parent vdev			*/
	vdev_t		**vdev_child;	/* array of children		*/
	uint64_t	vdev_children;	/* number of children		*/
	vdev_stat_t	vdev_stat;	/* virtual device statistics	*/
	vdev_stat_ex_t	vdev_stat_ex;	/* extended statistics		*/
	boolean_t	vdev_expanding;	/* expand the vdev?		*/
	boolean_t	vdev_reopening;	/* reopen in progress?		*/
	boolean_t	vdev_nonrot;	/* true if solid state		*/
	int		vdev_load_error; /* error on last load		*/
	int		vdev_open_error; /* error on last open		*/
	int		vdev_validate_error; /* error on last validate	*/
	kthread_t	*vdev_open_thread; /* thread opening children	*/
	kthread_t	*vdev_validate_thread; /* thread validating children */
	uint64_t	vdev_crtxg;	/* txg when top-level was added */
	uint64_t	vdev_root_zap;

	/*
	 * Top-level vdev state.
	 */
	uint64_t	vdev_ms_array;	/* metaslab array object	*/
	uint64_t	vdev_ms_shift;	/* metaslab size shift		*/
	uint64_t	vdev_ms_count;	/* number of metaslabs		*/
	metaslab_group_t *vdev_mg;	/* metaslab group		*/
	metaslab_group_t *vdev_log_mg;	/* embedded slog metaslab group	*/
	metaslab_t	**vdev_ms;	/* metaslab array		*/
  1. L241 vdev_ops: the node’s behaviour, its type.
  2. L243 Type-specific state: for a raidz, its parity; for a dRAID, its permutations and group sizes.
  3. L244–L247 The tree: the top-level vdev this node belongs to (a top-level vdev points at itself), the parent, the children.
  4. L264–L269 Only top-level vdevs own metaslabs and a metaslab group: only they receive allocations. Nodes below them only route I/O.
vdev_ops_t Excerpt of include/sys/vdev_impl.h, lines 101–126, OpenZFS zfs-2.4.4. View on GitHub 
typedef const struct vdev_ops {
	vdev_init_func_t		*vdev_op_init;
	vdev_fini_func_t		*vdev_op_fini;
	vdev_open_func_t		*vdev_op_open;
	vdev_close_func_t		*vdev_op_close;
	vdev_asize_func_t		*vdev_op_psize_to_asize;
	vdev_asize_func_t		*vdev_op_asize_to_psize;
	vdev_min_asize_func_t		*vdev_op_min_asize;
	vdev_min_alloc_func_t		*vdev_op_min_alloc;
	vdev_io_start_func_t		*vdev_op_io_start;
	vdev_io_done_func_t		*vdev_op_io_done;
	vdev_state_change_func_t	*vdev_op_state_change;
	vdev_need_resilver_func_t	*vdev_op_need_resilver;
	vdev_hold_func_t		*vdev_op_hold;
	vdev_rele_func_t		*vdev_op_rele;
	vdev_remap_func_t		*vdev_op_remap;
	vdev_xlation_func_t		*vdev_op_xlate;
	vdev_rebuild_asize_func_t	*vdev_op_rebuild_asize;
	vdev_metaslab_init_func_t	*vdev_op_metaslab_init;
	vdev_config_generate_func_t	*vdev_op_config_generate;
	vdev_nparity_func_t		*vdev_op_nparity;
	vdev_ndisks_func_t		*vdev_op_ndisks;
	vdev_kobj_post_evt_func_t	*vdev_op_kobj_evt_post;
	char				vdev_op_type[16];
	boolean_t			vdev_op_leaf;
} vdev_ops_t;
  1. L106–L107 Size translation: a raidz turns a block’s size into a larger allocated size.
  2. L110–L111 The I/O path. The root has neither: an I/O never starts at the root.
  3. L113 Each type decides whether a block touches a damaged child.
  4. L124–L125 The type’s name, and whether it is a leaf.

The figure’s types point to vdev_root_ops, vdev_mirror_ops, vdev_raidz_ops, vdev_draid_ops, vdev_disk_ops (on Linux), and the replacing, spare and distributed-spare tables below. The walk down. A block’s logical zio has no vdev, and zio_vdev_io_start hands it to the mirror’s function, with the block’s DVAs as children; every child zio, which has a vdev, goes to that vdev’s own io_start.

vdev_mirror_io_start Excerpt of module/zfs/vdev_mirror.c, lines 656–693, OpenZFS zfs-2.4.4. View on GitHub 
		/*
		 * For normal reads just pick one child.
		 */
		c = vdev_mirror_child_select(zio);
		children = (c >= 0);
	} else {
		ASSERT(zio->io_type == ZIO_TYPE_WRITE);

		/*
		 * Writes go to all children.
		 */
		c = 0;
		children = mm->mm_children;
	}

	while (children--) {
		mc = &mm->mm_child[c++];

		/*
		 * When sequentially resilvering and the integrity of the data
		 * is speculative (ZIO_FLAG_SPECULATIVE), issue write repair IOs
		 * only to the vdev which is being rebuilt. Existing data on
		 * other children must never be overwritten with unconfirmed
		 * data to avoid unrecoverable damage to the pool.
		 */
		if ((zio->io_priority == ZIO_PRIORITY_REBUILD) &&
		    (zio->io_flags & ZIO_FLAG_IO_REPAIR) &&
		    !(zio->io_flags & ZIO_FLAG_SCRUB) &&
		    (zio->io_flags & ZIO_FLAG_SPECULATIVE) &&
		    mm->mm_rebuilding && !mc->mc_rebuilding) {
			continue;
		}

		zio_nowait(zio_vdev_child_io(zio, zio->io_bp,
		    mc->mc_vd, mc->mc_offset, zio->io_abd, zio->io_size,
		    zio->io_type, zio->io_priority, 0,
		    vdev_mirror_child_done, mc));
	}
  1. L656–L660 A normal read picks one child: not one whose DTL says it is missing the block’s txg; a distributed spare first; else the least loaded.
  2. L662–L669 Writes go to all children.
  3. L671–L687 One exception: the repair writes of a sequential rebuild, when the data is speculative, go only to the child being rebuilt.
  4. L689–L692 One child zio per child, at the same offset and size.

A raidz’s io_start builds its map and issues one column per child, fewer for a small block: the RAID-Z room shows how; a normal read reads only the data columns. A dRAID is in part 4 below.

Classes. A top-level vdev joins a class when its metaslab group is created.

vdev_metaslab_group_create Excerpt of module/zfs/vdev.c, lines 1546–1579, OpenZFS zfs-2.4.4. View on GitHub 
	if (vd->vdev_mg == NULL) {
		metaslab_class_t *mc;

		if (vd->vdev_islog && vd->vdev_alloc_bias == VDEV_BIAS_NONE)
			vd->vdev_alloc_bias = VDEV_BIAS_LOG;

		ASSERT3U(vd->vdev_islog, ==,
		    (vd->vdev_alloc_bias == VDEV_BIAS_LOG));

		switch (vd->vdev_alloc_bias) {
		case VDEV_BIAS_LOG:
			mc = spa_log_class(spa);
			break;
		case VDEV_BIAS_SPECIAL:
			mc = spa_special_class(spa);
			break;
		case VDEV_BIAS_DEDUP:
			mc = spa_dedup_class(spa);
			break;
		default:
			mc = spa_normal_class(spa);
		}

		vd->vdev_mg = metaslab_group_create(mc, vd);

		if (!vd->vdev_islog) {
			if (mc == spa_special_class(spa)) {
				vd->vdev_log_mg = metaslab_group_create(
				    spa_special_embedded_log_class(spa), vd);
			} else {
				vd->vdev_log_mg = metaslab_group_create(
				    spa_embedded_log_class(spa), vd);
			}
		}
  1. L1549–L1550 A log device with no explicit bias gets the log bias.
  2. L1555–L1567 The bias picks the class: log, special, dedup, or normal. Four biases, six class pointers in the pool.
  3. L1571–L1578 Every vdev but a log gets a second group, in the embedded log class, or the special embedded log class for a special vdev: room for the ZIL without a log device. A metaslab is moved into it later, and only on a normal or special vdev with more than 64 metaslabs.
spa_preferred_class Excerpt of module/zfs/spa_misc.c, lines 2121–2156, OpenZFS zfs-2.4.4. View on GitHub 
spa_preferred_class(spa_t *spa, const zio_t *zio)
{
	metaslab_class_t *mc = zio->io_metaslab_class;
	boolean_t tried_dedup = (mc == spa_dedup_class(spa));
	boolean_t tried_special = (mc == spa_special_class(spa));
	const zio_prop_t *zp = &zio->io_prop;

	/* Gang children should always use the class of their parents. */
	if (zio->io_flags & ZIO_FLAG_GANG_CHILD) {
		ASSERT(mc != NULL);
		return (mc);
	}

	/*
	 * Override object type for the purposes of selecting a storage class.
	 * Primarily for DMU_OTN_ types where we can't explicitly control their
	 * storage class; instead, choose a static type most closely matches
	 * what we want.
	 */
	dmu_object_type_t objtype =
	    zp->zp_storage_type == DMU_OT_NONE ?
	    zp->zp_type : zp->zp_storage_type;

	/*
	 * ZIL allocations determine their class in zio_alloc_zil().
	 */
	ASSERT(objtype != DMU_OT_INTENT_LOG);

	if (DMU_OT_IS_DDT(objtype)) {
		if (spa_has_dedup(spa) && !tried_dedup && !tried_special)
			return (spa_dedup_class(spa));
		else if (spa_special_has_ddt(spa) && !tried_special)
			return (spa_special_class(spa));
		else
			return (spa_normal_class(spa));
	}
  1. L2123–L2125 The class an allocation already tried, if any: on a second call, the function knows what failed.
  2. L2144–L2147 ZIL blocks never come here.
  3. L2149–L2156 A DDT block: dedup if the pool has one and it was not tried; else special, if the pool has one (spa_special_has_ddt) and it was not tried; else normal. A pool “has” a class when it has at least one group in it.
spa_preferred_class Excerpt of module/zfs/spa_misc.c, lines 2158–2179, OpenZFS zfs-2.4.4. View on GitHub 
	if (!spa_has_special(spa) || tried_special)
		return (spa_normal_class(spa));

	if (DMU_OT_IS_METADATA(objtype) ||
	    (zfs_user_indirect_is_special && zp->zp_level > 0))
		return (spa_special_class(spa));

	/*
	 * Allow small blocks in special class.  However, leave a reserve of
	 * zfs_special_class_metadata_reserve_pct exclusively for metadata.
	 */
	if (zio->io_size <= zp->zp_zpl_smallblk) {
		metaslab_class_t *special = spa_special_class(spa);
		uint64_t limit = metaslab_class_get_space(special) *
		    (100 - zfs_special_class_metadata_reserve_pct) / 100;

		if (metaslab_class_get_alloc(special) < limit)
			return (special);
	}

	return (spa_normal_class(spa));
}
  1. L2158–L2159 No special vdev, or special already failed: normal.
  2. L2161–L2163 Metadata (an object type marked so, dnodes among them, file contents not) and the indirect blocks of files: special.
  3. L2169–L2176 A data block no larger than special_small_blocks, while special has allocated less than 75% of its space. Past that, the last 25% is for metadata only.
zio_dva_allocate Excerpt of module/zfs/zio.c, lines 4368–4407, OpenZFS zfs-2.4.4. View on GitHub 
	/*
	 * Try allocating the block in the usual metaslab class.
	 * If that's full, allocate it in some other class(es).
	 * If that's full, allocate as a gang block,
	 * and if all are full, the allocation fails (which shouldn't happen).
	 *
	 * Note that we do not fall back on embedded slog (ZIL) space, to
	 * preserve unfragmented slog space, which is critical for decent
	 * sync write performance.  If a log allocation fails, we will fall
	 * back to spa_sync() which is abysmal for performance.
	 */
	ASSERT(ZIO_HAS_ALLOCATOR(zio));
	error = metaslab_alloc(spa, mc, zio->io_size, bp,
	    zio->io_prop.zp_copies, zio->io_txg, NULL, flags,
	    &zio->io_alloc_list, zio->io_allocator, zio);

	/*
	 * When the dedup or special class is spilling into the normal class,
	 * there can still be significant space available due to deferred
	 * frees that are in-flight.  We track the txg when this occurred and
	 * back off adding new DDT entries for a few txgs to allow the free
	 * blocks to be processed.
	 */
	if (error == ENOSPC && spa->spa_dedup_class_full_txg != zio->io_txg &&
	    (mc == spa_dedup_class(spa) || (mc == spa_special_class(spa) &&
	    !spa_has_dedup(spa) && spa_special_has_ddt(spa)))) {
		spa->spa_dedup_class_full_txg = zio->io_txg;
		zfs_dbgmsg("%s[%llu]: %s class spilling, req size %llu, "
		    "%llu allocated of %llu",
		    spa_name(spa), (u_longlong_t)zio->io_txg,
		    metaslab_class_get_name(mc),
		    (u_longlong_t)zio->io_size,
		    (u_longlong_t)metaslab_class_get_alloc(mc),
		    (u_longlong_t)metaslab_class_get_space(mc));
	}

	/*
	 * Fall back to some other class when this one is full.
	 */
	if (error == ENOSPC && (newmc = spa_preferred_class(spa, zio)) != mc) {
  1. L4368–L4378 The order: the usual class, then other classes, then a gang block. Not the embedded log space, kept for the ZIL.
  2. L4380–L4382 The block’s copies, one DVA each, in one class.
  3. L4391–L4402 When dedup or special spills into normal, the txg is recorded, and new DDT entries back off for a few txgs.
  4. L4407 ENOSPC: ask spa_preferred_class again; it now knows which class failed, and the allocation starts over in the next one.

A ZIL block does not come here: zio_alloc_zil tries the log class, special’s embedded log, special, the embedded log on normal vdevs, then normal, with one DVA; the ZIL and SLOG room follows it. Ditto copies.

metaslab_alloc_dva_range Excerpt of module/zfs/metaslab.c, lines 5401–5427, OpenZFS zfs-2.4.4. View on GitHub 
	 * If we are doing ditto or log blocks, try to spread them across
	 * consecutive vdevs.  If we're forced to reuse a vdev before we've
	 * allocated all of our ditto blocks, then try and spread them out on
	 * that vdev as much as possible.  If it turns out to not be possible,
	 * gradually lower our standards until anything becomes acceptable.
	 * Also, allocating on consecutive vdevs (as opposed to random vdevs)
	 * gives us hope of containing our fault domains to something we're
	 * able to reason about.  Otherwise, any two top-level vdev failures
	 * will guarantee the loss of data.  With consecutive allocation,
	 * only two adjacent top-level vdev failures will result in data loss.
	 *
	 * If we are doing gang blocks (hintdva is non-NULL), try to keep
	 * ourselves on the same vdev as our gang block header.  It makes our
	 * fault domains something tractable.
	 */
	if (hintdva && DVA_IS_VALID(&hintdva[d])) {
		vd = vdev_lookup_top(spa, DVA_GET_VDEV(&hintdva[d]));
		mg = vdev_get_mg(vd, mc);
	}
	if (mg == NULL && d != 0) {
		vd = vdev_lookup_top(spa, DVA_GET_VDEV(&dva[d - 1]));
		mg = vdev_get_mg(vd, mc)->mg_next;
	}
	if (mg == NULL || mg->mg_class != mc || mg->mg_activation_count <= 0) {
		ASSERT(mca->mca_rotor != NULL);
		mg = mca->mca_rotor;
	}
  1. L5401–L5410 Spread copies across consecutive vdevs: only two adjacent top-level vdevs failing can lose such a block.
  2. L5420–L5423 DVA d starts at the group after DVA d − 1’s vdev: the next vdev of the class, round the class. A class of one vdev puts every copy on it.
  3. L5424–L5427 The first DVA starts at the class’s rotor, which moves on by block sizes, queue depths and free space: the metaslabs room.

How many copies: metadata one more than copies with redundant_metadata=all, the default; the MOS, which holds the DDT, spa_max_replication, 3, and a block’s copies are capped at that same 3.

A replacement. The replacing and spare tables are the mirror’s.

vdev_replacing_ops Excerpt of module/zfs/vdev_mirror.c, lines 997–1021, OpenZFS zfs-2.4.4. View on GitHub 
vdev_ops_t vdev_replacing_ops = {
	.vdev_op_init = NULL,
	.vdev_op_fini = NULL,
	.vdev_op_open = vdev_mirror_open,
	.vdev_op_close = vdev_mirror_close,
	.vdev_op_psize_to_asize = vdev_default_asize,
	.vdev_op_asize_to_psize = vdev_default_psize,
	.vdev_op_min_asize = vdev_default_min_asize,
	.vdev_op_min_alloc = NULL,
	.vdev_op_io_start = vdev_mirror_io_start,
	.vdev_op_io_done = vdev_mirror_io_done,
	.vdev_op_state_change = vdev_mirror_state_change,
	.vdev_op_need_resilver = vdev_default_need_resilver,
	.vdev_op_hold = NULL,
	.vdev_op_rele = NULL,
	.vdev_op_remap = NULL,
	.vdev_op_xlate = vdev_default_xlate,
	.vdev_op_rebuild_asize = vdev_mirror_rebuild_asize,
	.vdev_op_metaslab_init = NULL,
	.vdev_op_config_generate = NULL,
	.vdev_op_nparity = NULL,
	.vdev_op_ndisks = NULL,
	.vdev_op_type = VDEV_TYPE_REPLACING,	/* name of this vdev type */
	.vdev_op_leaf = B_FALSE			/* not a leaf vdev */
};
  1. L1000–L1007 The mirror’s open, close, size, io_start and io_done, as in vdev_mirror_ops and vdev_spare_ops.
  2. L1019 Only the name differs.
spa_vdev_attach Excerpt of module/zfs/spa.c, lines 8324–8360, OpenZFS zfs-2.4.4. View on GitHub 
	} else {
		/*
		 * Active hot spares can only be replaced by inactive hot
		 * spares.
		 */
		if (pvd->vdev_ops == &vdev_spare_ops &&
		    oldvd->vdev_isspare &&
		    !spa_has_spare(spa, newvd->vdev_guid))
			return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));

		/*
		 * If the source is a hot spare, and the parent isn't already a
		 * spare, then we want to create a new hot spare.  Otherwise, we
		 * want to create a replacing vdev.  The user is not allowed to
		 * attach to a spared vdev child unless the 'isspare' state is
		 * the same (spare replaces spare, non-spare replaces
		 * non-spare).
		 */
		if (pvd->vdev_ops == &vdev_replacing_ops &&
		    spa_version(spa) < SPA_VERSION_MULTI_REPLACE) {
			return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));
		} else if (pvd->vdev_ops == &vdev_spare_ops &&
		    newvd->vdev_isspare != oldvd->vdev_isspare) {
			return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));
		}

		if (spa_vdev_new_spare_would_cause_double_spares(newvd, pvd)) {
			vdev_dbgmsg(newvd,
			    "disk would create double spares, ignore.");
			return (spa_vdev_exit(spa, newrootvd, txg, EEXIST));
		}

		if (newvd->vdev_isspare)
			pvops = &vdev_spare_ops;
		else
			pvops = &vdev_replacing_ops;
	}
  1. L8325–L8332 An active hot spare can only be replaced by an inactive one.
  2. L8356–L8359 A hot spare gets a spare node, any other disk a replacing node. A distributed spare can only replace a child of its own dRAID.
vdev_add_parent Excerpt of module/zfs/vdev.c, lines 1418–1449, OpenZFS zfs-2.4.4. View on GitHub 
vdev_add_parent(vdev_t *cvd, vdev_ops_t *ops)
{
	spa_t *spa = cvd->vdev_spa;
	vdev_t *pvd = cvd->vdev_parent;
	vdev_t *mvd;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	mvd = vdev_alloc_common(spa, cvd->vdev_id, 0, ops);

	mvd->vdev_asize = cvd->vdev_asize;
	mvd->vdev_min_asize = cvd->vdev_min_asize;
	mvd->vdev_max_asize = cvd->vdev_max_asize;
	mvd->vdev_psize = cvd->vdev_psize;
	mvd->vdev_ashift = cvd->vdev_ashift;
	mvd->vdev_logical_ashift = cvd->vdev_logical_ashift;
	mvd->vdev_physical_ashift = cvd->vdev_physical_ashift;
	mvd->vdev_state = cvd->vdev_state;
	mvd->vdev_crtxg = cvd->vdev_crtxg;
	mvd->vdev_nonrot = cvd->vdev_nonrot;

	vdev_remove_child(pvd, cvd);
	vdev_add_child(pvd, mvd);
	cvd->vdev_id = mvd->vdev_children;
	vdev_add_child(mvd, cvd);
	vdev_top_update(cvd->vdev_top, cvd->vdev_top);

	if (mvd == mvd->vdev_top)
		vdev_top_transfer(cvd, mvd);

	return (mvd);
}
  1. L1426 The new node takes the old disk’s vdev_id…
  2. L1439–L1440 …and its place among its parent’s children.
  3. L1441–L1442 The old disk becomes its child 0; spa_vdev_attach then adds the new disk after it, as the last child.
  4. L1445–L1446 If the old disk was a top-level vdev, the new node becomes the top-level vdev and takes over its metaslabs, its group and its bias.

The new disk’s DTL is marked missing from TXG_INITIAL to the current txg plus TXG_CONCURRENT_STATES (this line, 3 and 4): a read of an older block skips it, which is what the resilver fixes.

spa_vdev_resilver_done_hunt Excerpt of module/zfs/spa.c, lines 9438–9456, OpenZFS zfs-2.4.4. View on GitHub 
	/*
	 * Check for a completed replacement.  We always consider the first
	 * vdev in the list to be the oldest vdev, and the last one to be
	 * the newest (see spa_vdev_attach() for how that works).  In
	 * the case where the newest vdev is faulted, we will not automatically
	 * remove it after a resilver completes.  This is OK as it will require
	 * user intervention to determine which disk the admin wishes to keep.
	 */
	if (vd->vdev_ops == &vdev_replacing_ops) {
		ASSERT(vd->vdev_children > 1);

		newvd = vd->vdev_child[vd->vdev_children - 1];
		oldvd = vd->vdev_child[0];

		if (vdev_dtl_empty(newvd, DTL_MISSING) &&
		    vdev_dtl_empty(newvd, DTL_OUTAGE) &&
		    !vdev_dtl_required(oldvd))
			return (oldvd);
	}
  1. L9439–L9441 The first child is the oldest disk, the last one the newest.
  2. L9446–L9455 When the new disk misses nothing and the old one is not required, the old disk is detached; the replacing node, left with one child, is removed.

A spare node is not collapsed by its own resilver. When the original disk is brought back online with the unspare flag, which zed passes, or its fault is cleared, it is marked vdev_unspare, and the end of its resilver detaches the spare. If the original disk is itself replaced, the end of that resilver detaches the spare as well. Otherwise the administrator detaches one of the two, and a spare node left with one child is removed.

dRAID. The table of permutations is generated, not stored.

vdev_draid_generate_perms Excerpt of module/zfs/vdev_draid.c, lines 506–526, OpenZFS zfs-2.4.4. View on GitHub 
	uint64_t draid_seed[2] = { VDEV_DRAID_SEED, map->dm_seed };
	uint8_t *current_row, *previous_row = initial_row;

	/*
	 * Perform a Fisher-Yates shuffle of each row using the previous
	 * row as the starting point.  An initial_row with known pattern
	 * is used as the input for the first row.
	 */
	for (int i = 0; i < nperms; i++) {
		current_row = &perms[i * children];
		memcpy(current_row, previous_row, rowsz);

		for (int j = children - 1; j > 0; j--) {
			uint64_t k = vdev_draid_rand(draid_seed) % (j + 1);
			uint8_t val = current_row[j];
			current_row[j] = current_row[k];
			current_row[k] = val;
		}

		previous_row = current_row;
	}
  1. L506 Seeded with VDEV_DRAID_SEED and the seed of the map for this number of children: for five, 256 permutations, seed 0xf46733b7f4d47dfd.
  2. L514–L526 Each row is a Fisher-Yates shuffle of the previous one, with xoroshiro128++. The result is checked against the stored checksum, 0x2a1648d74 for five children; the figure’s copy gives the same.
vdev_draid_map_alloc_row Excerpt of module/zfs/vdev_draid.c, lines 1042–1064, OpenZFS zfs-2.4.4. View on GitHub 
	uint8_t *base;
	uint64_t iter, asize = 0;
	vdev_draid_get_perm(vdc, perm, &base, &iter);
	for (uint64_t i = 0; i < groupwidth; i++) {
		raidz_col_t *rc = &rr->rr_col[i];
		uint64_t c = (groupstart + i) % ndisks;

		/* increment the offset if we wrap to the next row */
		if (i == wrap)
			physical_offset += VDEV_DRAID_ROWHEIGHT;

		rc->rc_devidx = vdev_draid_permute_id(vdc, base, iter, c);
		rc->rc_offset = physical_offset;

		if (q == 0 && i >= bc)
			rc->rc_size = 0;
		else if (i < bc)
			rc->rc_size = (q + 1) << ashift;
		else
			rc->rc_size = q << ashift;

		asize += rc->rc_size;
	}
  1. L1044 The row of the table and its rotation: each row serves once per child, shifted by one each time.
  2. L1047 Column i is position (groupstart + i) mod ndisks; ndisks is the children minus the distributed spares.
  3. L1049–L1051 A group that runs past the end of a row goes on in the next one.
  4. L1053 The child: the row’s entry at that position, plus the rotation, modulo the children.
  5. L1056–L1061 Column sizes as in RAID-Z; the first columns are parity.

The group of a block comes from its offset, and zpool create sets ngroups so that groups fill whole rows: four for draid1:2d:5c:1s, whose groups are three wide over four disks’ worth of five children, in rows of 16 MiB. The design comment draws the layout. A write fills every column of its group, an empty or short column padded with a skip sector, since “dRAID always allocates a full stripe width”; a normal read reads the data columns that hold data. Positions from ndisks to the last of each row take no column: that is the distributed spares’ space. A distributed spare is listed with the hot spares, and is a leaf whose io_start sends each write to the child holding the spare’s space in that row.

What the figure leaves out

Which vdev of a class gets a block’s first copy is decided by the allocator’s rotor, by load and free space: the figure moves the rotor on by one vdev per block, and places each dRAID copy at the start of the next group and each raidz copy after the last. Block sizes are inputs: 4 KiB for the ZIL and small file blocks, 16 KiB for metadata, 8 KiB for the DDT, 128 KiB for a large file block, after compression. A mirror’s read takes the first eligible child, where the code takes the least loaded. Only the special class fills up, and its embedded log space stays free. Normal and special vdevs are taken to be large enough to hold an embedded log metaslab. One replacement per disk at a time; the resilver itself, its order and the sequential rebuild are not drawn, only its end. Removed (indirect) vdevs, holes, RAID-Z expansion, gang blocks and errors are left out. The 3D view draws the same state as the 2D one; neither shows disk offsets.