Room 10

ZIL and SLOG

A program that calls fsync() asks for one promise: when the call returns, its data survives a power cut. A database asks for it on every commit.

ZFS writes its changes in transaction groups, and a txg is synced only every few seconds. Forcing a txg sync for each fsync() would rewrite large parts of the tree for a few bytes of data, and make the caller wait for all of it. ZFS does something else. Each dataset keeps an intent log, the ZIL: every change is noted as a small record, and on fsync() the records are written at once to a chain of log blocks. Once the txg holding the real change has synced, they are no longer needed and are thrown away. The log is read only after a crash. A separate log device, a SLOG, gives those blocks a home of their own.

1 Write, fsync, cut the power

Below, a fresh pool and one file, already several records long, whose blocks are 128 KiB; txg 100 is open, and the dataset has no log yet. Choose a log device, logbias, how the file is opened and a size: the first panel says what the code will decide for the next write(), and why. write() leaves a record in memory; fsync() writes it to the log, block by block, and returns once the disk has flushed. Sync the txg and the log blocks are freed. Cut the power whenever you like, then import the pool and mount the dataset.

The first fsync() creates the log, and that first commit waits for the txg to sync, so it logs nothing; from the second on, records go into the log. So start with a write and fsync(), then try a 4 KiB write and fsync() without a SLOG, then cut the power; the same with a sync before the cut; 128 KiB with and without a SLOG; logbias=throughput; a write without fsync() before the cut.

What `zil_write_state` decides for the next `write()`

WR_NEED_COPY 4 KiB

4 KiB is under zfs_immediate_write_sz, 32 KiB: not indirect. code
Not committed at once: WR_NEED_COPY, the data copied into the log block only when the record is written. code

This write() returns at once; fsync() commits it. code

The application’s file

No write yet.

In memory

itxs waiting for a commit code

None.

Log write blocks (zl_lwb_list) code

No log yet: the first commit creates it.

On disk

zil_header_t, in the dataset code

zh_log: hole
zh_claim_txg: 0
zh_flags: 0
zh_replay_seq: 0

The chain, from zh_log code

Empty.

Pool vdevs: embedded log space and data

nothing

Text version of this figure

A pool without a SLOG, logbias=latency. A file with 128 KiB blocks. Open txg 100, last synced 99.

Writes: none.
In memory, waiting for a commit: none.
Log write blocks: none.
On disk, from zh_log: empty.
Header: zh_log hole, zh_claim_txg 0, ZIL_REPLAY_NEEDED no.

Txg 100 open, txg 99 synced. 0 writes: none. Log: 0 blocks in the chain, zh_log hole.

The idea, at three depths

In one sentence

ZFS makes fsync() return early by writing at once only a record of the change, to a log kept per dataset, and the change itself later with its txg; the log is thrown away once that txg has synced, and read only after a crash.

In one paragraph

Every write() makes an in-memory record, an itx, under the open txg. fsync(), or a write to a file opened with O_DSYNC, commits the file’s records: they are packed into log write blocks, lwbs, each written as one block whose trailer points to the next block, so the log is a chain; the call returns when the block holding its last record is written and the disk caches are flushed. How much of the data goes into the log is decided write by write: copied into the record (WR_COPIED), copied when the block is written (WR_NEED_COPY), or written in place in the pool with the record only pointing to it (WR_INDIRECT), so that large data is not written twice. With a SLOG, unless logbias=throughput or Direct I/O, the data always goes into the log. The log’s blocks come from the SLOG first; a pool without one takes them from space reserved on its ordinary vdevs. When a txg syncs, the log blocks whose records all belong to synced txgs are freed. After a crash, importing the pool claims the chain, and mounting the dataset replays every record whose txg had not synced.

In detail

A record per write. In zfs_write, every write is logged by zfs_log_write, which makes an itx and queues it under the write’s txg, lrc_txg. Nothing is written yet. If the file was opened with O_SYNC or O_DSYNC, or the dataset has sync=always, the write is committed at once: zfs_write calls zil_commit before it returns. Otherwise the record waits for fsync(), which calls the same function. What the record holds is decided first:

zil_write_state Excerpt of module/zfs/zil.c, lines 2212–2246, OpenZFS zfs-2.4.4. View on GitHub

itx_wr_state_t
zil_write_state(zilog_t *zilog, uint64_t size, uint32_t blocksize,
    boolean_t o_direct, boolean_t commit)
{
	if (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT || o_direct)
		return (WR_INDIRECT);

	/*
	 * Don't use indirect for too small writes to reduce overhead.
	 * Don't use indirect if written less than a half of a block if
	 * we are going to commit it immediately, since next write might
	 * rewrite the same block again, causing inflation.  If commit
	 * is not planned, then next writes might coalesce, and so the
	 * indirect may be perfect.
	 */
	boolean_t indirect = (size >= zfs_immediate_write_sz &&
	    (size >= blocksize / 2 || !commit));

	if (spa_has_slogs(zilog->zl_spa)) {
		/* Dedicated slogs: never use indirect */
		indirect = B_FALSE;
	} else if (spa_has_special(zilog->zl_spa)) {
		/* Special vdevs: only when beneficial */
		boolean_t on_special = (blocksize <=
		    zilog->zl_os->os_zpl_special_smallblock);
		indirect &= (on_special || !zil_special_is_slog);
	}

	if (indirect)
		return (WR_INDIRECT);
	else if (commit)
		return (WR_COPIED);
	else
		return (WR_NEED_COPY);
}

L2216–L2217 logbias=throughput, or a write that took the Direct I/O path: always indirect, SLOG or not. The log block that holds the record still comes from the SLOG first: logbias decides where the data goes, not the log.
L2219–L2228 Indirect only from 32 KiB, and, for a write committed at once, only from half a block, 64 KiB for the figure’s file, because, as the comment above says, the next write might rewrite the same block.
L2230–L2232 A pool with a SLOG never logs indirect. spa_has_slogs asks whether the log class has a device, not whether it has room: a full SLOG still counts.
L2233–L2238 Special vdevs without a SLOG: indirect only when the file’s blocks go to the special class (or zil_special_is_slog is 0). The figure’s pool has no special vdev.
L2240–L2245 Indirect: the record carries a pointer. Otherwise WR_COPIED, the data copied into the record now, when the write is committed at once; or WR_NEED_COPY, the data read from the file when the log block is filled.

Then zfs_log_write turns WR_COPIED into WR_NEED_COPY when the data is over 7680 bytes, since a copied record must fit in one log block, and cuts an indirect write at each file block. O_DIRECT takes the Direct I/O path only for a write of at least a file block: a shorter one goes through the ARC, so in the figure only the 128 KiB write with O_DIRECT is Direct I/O. For WR_INDIRECT, the comment of zfs_get_data gives the reason: “for large writes it’s cheaper to sync the data and get a pointer to it (indirect) so that we don’t have to write the data twice”. The block is written in place by dmu_sync, or was already written by the Direct I/O write. In the figure, the panel at the top runs this function for every combination of the buttons; the badge on each write is its result.

Committing. zil_commit_impl, called by fsync():

zil_commit_impl Excerpt of module/zfs/zil.c, lines 4097–4122, OpenZFS zfs-2.4.4. View on GitHub

	zil_commit_waiter_t *zcw = zil_alloc_commit_waiter();
	zil_commit_itx_assign(zilog, zcw);

	uint64_t wtxg = zil_commit_writer(zilog, zcw);
	zil_commit_waiter(zilog, zcw);

	int err = 0;
	if (zcw->zcw_error != 0) {
		/*
		 * If there was an error writing out the ZIL blocks that
		 * this thread is waiting on, then we fallback to
		 * relying on spa_sync() to write out the data this
		 * thread is waiting on. Obviously this has performance
		 * implications, but the expectation is for this to be
		 * an exceptional case, and shouldn't occur often.
		 */
		ZIL_STAT_BUMP(zilog, zil_commit_error_count);
		DTRACE_PROBE2(zil__commit__io__error,
		    zilog_t *, zilog, zil_commit_waiter_t *, zcw);
		err = txg_wait_synced_flags(zilog->zl_dmu_pool, 0,
		    TXG_WAIT_SUSPEND);
	} else if (wtxg != 0) {
		ZIL_STAT_BUMP(zilog, zil_commit_suspend_count);
		err = txg_wait_synced_flags(zilog->zl_dmu_pool, wtxg,
		    TXG_WAIT_SUSPEND);
	}

L4097–L4098 A commit waiter is queued as a special itx, behind every record of the file (the file’s other records were moved to the sync list just before).
L4100 The pending records are packed into lwbs and the lwbs issued.
L4101 The caller sleeps until the lwb holding its waiter is written and flushed (zil_commit_waiter_done).
L4104–L4117 On a log write error, fall back to waiting for a full txg sync: just as durable, only slower. The figure has no I/O error.

An lwb’s life. Each lwb goes through seven states, in this order:

lwb_state_t Excerpt of include/sys/zil_impl.h, lines 83–92, OpenZFS zfs-2.4.4. View on GitHub

typedef enum {
    LWB_STATE_NEW,
    LWB_STATE_OPENED,
    LWB_STATE_CLOSED,
    LWB_STATE_READY,
    LWB_STATE_ISSUED,
    LWB_STATE_WRITE_DONE,
    LWB_STATE_FLUSH_DONE,
    LWB_NUM_STATES
} lwb_state_t;

L84–L85 New, then opened on its first record.
L86 Closed when full, or at the end of a commit, which also creates the next lwb, still empty.
L87–L88 Ready when filled with its data; issued once the next block is allocated.
L89–L90 Write done, then flush done when the disk caches are flushed: the waiters on it are woken and fsync() returns.

The comment above the enum describes the transitions and the locks they need. With one writer and nothing else in flight, as in the figure, the lwb holding a commit is closed and issued at once rather than kept open for more records. The figure draws each state on each lwb, in the memory panel.

Packing. How records go into lwbs:

zil_lwb_assign Excerpt of module/zfs/zil.c, lines 2334–2369, OpenZFS zfs-2.4.4. View on GitHub

	lwb_sp = lwb->lwb_nmax - lwb->lwb_nused;
	max_log_data = zil_max_log_data(zilog, sizeof (lr_write_t));
	if (reclen > lwb_sp || (reclen + dlen > lwb_sp &&
	    lwb_sp < zil_max_waste_space(zilog) &&
	    (dlen % max_log_data == 0 ||
	    lwb_sp < reclen + dlen % max_log_data))) {
		list_insert_tail(ilwbs, lwb);
		lwb = zil_lwb_write_close(zilog, lwb);
		if (lwb == NULL)
			return (NULL);
		zil_lwb_write_open(zilog, lwb);
		lwb_sp = lwb->lwb_nmax - lwb->lwb_nused;
	}

	/*
	 * There must be enough space in the log block to hold reclen.
	 * For WR_COPIED, we need to fit the whole record in one block,
	 * and reclen is the write record header size + the data size.
	 * For WR_NEED_COPY, we can create multiple records, splitting
	 * the data into multiple blocks, so we only need to fit one
	 * word of data per block; in this case reclen is just the header
	 * size (no data).
	 */
	ASSERT3U(reclen + MIN(dlen, sizeof (uint64_t)), <=, lwb_sp);

	dnow = MIN(dlen, lwb_sp - reclen);
	if (dlen > dnow) {
		ASSERT3U(lr->lrc_txtype, ==, TX_WRITE);
		ASSERT3U(itx->itx_wr_state, ==, WR_NEED_COPY);
		citx = zil_itx_clone(itx);
		clr = &citx->itx_lr;
		lr_write_t *clrw = (lr_write_t *)clr;
		clrw->lr_length = dnow;
		lrw->lr_offset += dnow;
		lrw->lr_length -= dnow;
		zilog->zl_cur_left -= dnow;

L2334–L2346 A record that does not fit closes the lwb and opens the next. The long test lets a WR_NEED_COPY record start in an almost full block only when that saves a split.
L2348–L2357 A WR_COPIED record must fit whole: it is never split.
L2359–L2369 A WR_NEED_COPY record larger than the room left is split: a clone takes what fits, the rest goes on to the next block.

Each record then gets the next sequence number, lrc_seq, and the lwb keeps the highest txg of its records. A write record’s header, lr_write_t, is 192 bytes; a log block’s trailer, zil_chain_t, 184 bytes; a log block at most 128 KiB. So a 128 KiB block holds 130,888 bytes of records, and a 128 KiB WR_NEED_COPY write is 130,696 bytes in one lwb and 376 bytes in the next: the figure shows it with a SLOG.

The chain, and where the SLOG comes in. When an lwb is issued, it allocates the block of the next lwb:

zil_lwb_write_issue Excerpt of module/zfs/zil.c, lines 2073–2092, OpenZFS zfs-2.4.4. View on GitHub

		error = zio_alloc_zil(spa, zilog->zl_os, txg, bp,
		    min_size, max_size, &slog, flexible);
		if (error == 0) {
			if (closed_slim)
				ASSERT3U(BP_GET_LSIZE(bp), ==, max_size);
			else if (flexible)
				nlwb->lwb_sz = BP_GET_LSIZE(bp);
			else
				ASSERT3U(BP_GET_LSIZE(bp), ==, nlwb->lwb_sz);
		}
		if (flexible)
			mutex_exit(&nlwb->lwb_lock);
	}
	if (error == 0) {
		ASSERT3U(BP_GET_BIRTH(bp), ==, txg);
		BP_SET_CHECKSUM(bp, (nlwb->lwb_flags & LWB_FLAG_SLIM) ?
		    ZIO_CHECKSUM_ZILOG2 : ZIO_CHECKSUM_ZILOG);
		bp->blk_cksum = lwb->lwb_blk.blk_cksum;
		bp->blk_cksum.zc_word[ZIL_ZC_SEQ]++;
	}

L2073–L2074 The next block is allocated now, and its pointer goes into this block’s trailer, zc_next_blk: each block points to the next, from zh_log in the header on.
L2087 The new block is born in an open txg.
L2088–L2091 It inherits this block’s checksum seed, with the sequence word incremented: at replay, that is how a valid next block is told from old data.

The first block of a chain, 4 KiB, comes from zil_create, through the same allocator, zio_alloc_zil:

zio_alloc_zil Excerpt of module/zfs/zio.c, lines 4561–4595, OpenZFS zfs-2.4.4. View on GitHub

	/* Try log class (dedicated slog devices) first */
	error = metaslab_alloc_range(spa, spa_log_class(spa), min_size,
	    max_size, new_bp, 1, txg, NULL, flags, &io_alloc_list, allocator,
	    NULL, &alloc_size);
	*slog = (error == 0);

	/* Try special_embedded_log class (reserved on special vdevs) */
	if (error != 0) {
		error = metaslab_alloc_range(spa,
		    spa_special_embedded_log_class(spa), min_size, max_size,
		    new_bp, 1, txg, NULL, flags, &io_alloc_list, allocator,
		    NULL, &alloc_size);
	}

	/* Try special class (general special vdev allocation) */
	if (error != 0) {
		error = metaslab_alloc_range(spa, spa_special_class(spa),
		    min_size, max_size, new_bp, 1, txg, NULL, flags,
		    &io_alloc_list, allocator, NULL, &alloc_size);
	}

	/* Try embedded_log class (reserved on normal vdevs) */
	if (error != 0) {
		error = metaslab_alloc_range(spa, spa_embedded_log_class(spa),
		    min_size, max_size, new_bp, 1, txg, NULL, flags,
		    &io_alloc_list, allocator, NULL, &alloc_size);
	}

	/* Finally fall back to normal class */
	if (error != 0) {
		ZIOSTAT_BUMP(ziostat_alloc_class_fallbacks);
		error = metaslab_alloc_range(spa, spa_normal_class(spa),
		    min_size, max_size, new_bp, 1, txg, NULL, flags,
		    &io_alloc_list, allocator, NULL, &alloc_size);
	}

L4561–L4565 The log class, the SLOG devices, first. Success sets slog, which becomes LWB_FLAG_SLOG on the lwb.
L4567–L4580 Then log space reserved on special vdevs, and the special class.
L4582–L4587 Then the embedded log: space every ordinary top-level vdev reserves for the ZIL. A pool without a SLOG, or with a full one, and without special vdevs, logs there.
L4589–L4595 Last, the normal class.

The first commit waits for a txg. zil_create is called by the commit when the dataset has no log. It cannot write the new block into zh_log itself: the header changes only in syncing context. So it allocates the block in a transaction and waits for that txg to sync. The writes of that txg are then in the tree, so the commit finds their txg synced and destroys their itxs instead of logging them; only the commit waiter goes into the first lwb. The first commit on a new log, an fsync() or an O_DSYNC write, therefore costs a txg sync, and that is why a log with nothing to replay keeps its first block.

In the figure, every allocation lists the classes it tried. The code says where a log block goes, not how fast the device is: what a SLOG buys depends on the device.

When the txg syncs. In syncing context, for each dataset whose log changed:

zil_sync Excerpt of module/zfs/zil.c, lines 4206–4224, OpenZFS zfs-2.4.4. View on GitHub

	while ((lwb = list_head(&zilog->zl_lwb_list)) != NULL) {
		zh->zh_log = lwb->lwb_blk;
		if (lwb->lwb_state != LWB_STATE_FLUSH_DONE ||
		    lwb->lwb_alloc_txg > txg || lwb->lwb_max_txg > txg)
			break;
		list_remove(&zilog->zl_lwb_list, lwb);
		if (!BP_IS_HOLE(&lwb->lwb_blk))
			zio_free(spa, txg, &lwb->lwb_blk);
		zil_free_lwb(zilog, lwb);

		/*
		 * If we don't have anything left in the lwb list then
		 * we've had an allocation failure and we need to zero
		 * out the zil_header blkptr so that we don't end
		 * up freeing the same block twice.
		 */
		if (list_is_empty(&zilog->zl_lwb_list))
			BP_ZERO(&zh->zh_log);
	}

L4207 The header’s zh_log follows the head of the list.
L4208–L4210 An lwb is freed once it is flushed, every record in it belongs to a synced txg, and its own block was allocated in this txg or earlier. The first that is not stops the loop.
L4211–L4214 Its block goes back to the allocator.

The records of a synced txg that were never written to the log are simply dropped by zil_clean: the txg made them durable. In the figure, freed blocks are greyed out and crossed through.

After a crash: claim. At import, in the pool’s first txg, for each dataset:

zil_claim Excerpt of module/zfs/zil.c, lines 1232–1253, OpenZFS zfs-2.4.4. View on GitHub

	 * Claim all log blocks if we haven't already done so, and remember
	 * the highest claimed sequence number.  This ensures that if we can
	 * read only part of the log now (e.g. due to a missing device),
	 * but we can read the entire log later, we will not try to replay
	 * or destroy beyond the last block we successfully claimed.
	 */
	ASSERT3U(zh->zh_claim_txg, <=, first_txg);
	if (zh->zh_claim_txg == 0) {
		(void) zil_parse(zilog, zil_claim_log_block,
		    zil_claim_log_record, tx, first_txg, B_FALSE);
		zh->zh_claim_txg = first_txg;
		zh->zh_claim_blk_seq = zilog->zl_parse_blk_seq;
		zh->zh_claim_lr_seq = zilog->zl_parse_lr_seq;
		if (zilog->zl_parse_lr_count || zilog->zl_parse_blk_count > 1)
			zh->zh_flags |= ZIL_REPLAY_NEEDED;
		zh->zh_flags |= ZIL_CLAIM_LR_SEQ_VALID;
		if (os->os_encrypted)
			os->os_next_write_raw[tx->tx_txg & TXG_MASK] = B_TRUE;
		dsl_dataset_dirty(dmu_objset_ds(os), tx);
	}

	ASSERT3U(first_txg, ==, (spa_last_synced_txg(zilog->zl_spa) + 1));

L1239–L1241 zil_parse walks the chain from zh_log, claiming each block so that the allocator does not hand it out again, and the blocks WR_INDIRECT records point to. The walk ends at the block allocated but never written: its checksum does not match.
L1242 The claim txg: the pool’s first txg.
L1245–L1246 ZIL_REPLAY_NEEDED if a record was found, or more than one block followed. The walk counts the unwritten last block, so a log with nothing in it counts one.
L1253 That first txg is the last synced txg plus one.

Replay. At mount, zil_replay runs each record through:

zil_replay_log_record Excerpt of module/zfs/zil.c, lines 4696–4712, OpenZFS zfs-2.4.4. View on GitHub

static int
zil_replay_log_record(zilog_t *zilog, const lr_t *lr, void *zra,
    uint64_t claim_txg)
{
	zil_replay_arg_t *zr = zra;
	const zil_header_t *zh = zilog->zl_header;
	uint64_t reclen = lr->lrc_reclen;
	uint64_t txtype = lr->lrc_txtype;
	int error = 0;

	zilog->zl_replaying_seq = lr->lrc_seq;

	if (lr->lrc_seq <= zh->zh_replay_seq)	/* already replayed */
		return (0);

	if (lr->lrc_txg < claim_txg)		/* already committed */
		return (0);

L4708–L4709 A record already replayed by an earlier, interrupted replay is skipped.
L4711–L4712 A record of a txg older than the claim txg is skipped: that txg synced, the change is already in the tree. Every other record is applied again.

Without ZIL_REPLAY_NEEDED, zil_replay only destroys the log, keeping its first block: the emptied header names it again, under a new chain GUID, and the next commit writes there without allocating. Otherwise it replays, then destroys the log. The header that holds all this, zil_header_t, lives in the dataset: zh_claim_txg, zh_replay_seq, zh_log, the claimed sequence numbers and the flags. So a crash before the sync brings back every record of the unsynced txg; a crash after it finds a log with nothing to replay, and a write that was never fsync()ed and never synced is lost, as its program allowed. After a replay the whole log is destroyed, so the next commit creates a new one and waits for a txg again.

What the figure leaves out

The figure runs the functions above on one file of one dataset, with one writer and one txg in flight, synced at once in one pass. The file already spans several records, so its blocks are 128 KiB; a file of a single block smaller than recordsize has a smaller block, and the half-block rule uses it. The first block of a new chain is 4 KiB, as in the code; every later log block is 128 KiB, where the code chooses each block’s size from the burst of records (zil_lwb_plan). So in a real chain records split at other places, and the byte counts in the figure are those of 128 KiB blocks. Each write() starts at the beginning of a file block. There is no I/O error, so no fallback to txg_wait_synced; the flush is one step for the whole lwb, without the per-vdev flush tree; the commit timeout never fires, because one writer closes its lwb at once. Writes over zil_slog_bulk per commit, 64 MiB, go to a SLOG at a lower priority; the Slim ZIL layout and encryption are not drawn. None of these changes a write state, a class, or which writes are replayed. When a txg syncs, see Transaction groups; how the classes share the pool’s space, Metaslabs and space maps.