Uberblok

Room 4

Transaction groups

Copy-on-write keeps the disks consistent at every instant, but only if the switch from the old tree to the new one is a single write. Writes never stop arriving, from many threads at once. If every one of them ended with its own commit, the pool would flush every disk and rewrite its labels and uberblock for every single write. And a commit has to cover a state in which no change is half done.

ZFS answers with batches. Every change joins a numbered transaction group, a txg, and ZFS “asserts consistency at the granularity of these transaction groups”.

1 Three txgs, one per stage

A txg is open, quiescing or syncing, and there is never more than one txg in each state: TXG_CONCURRENT_STATES is 3. Writes go into the open txg. To commit it, a thread closes it and opens the next one at once, so writers never wait for a disk. It then waits for the writes still under way in the closed txg to finish. A second thread writes the closed txg out and ends with the uberblock. Add writes, play, or go one event at a time.

Text version of this figure
  1. Every change to the pool joins a numbered transaction group (txg). Three txgs can be active at once, one in each state: open, quiescing, syncing.
  2. A write takes a hold on the open txg: txg_hold_open raises a counter of the CPU it runs on, for that txg. When the write is done, txg_rele_to_sync lowers it.
  3. The sync thread waits for one of three things: zfs_txg_timeout (5 seconds) since the last sync began, someone waiting for a sync (zpool sync, or txg_kick when the open txg holds more dirty data than zfs_dirty_data_sync_percent of zfs_dirty_data_max), or a txg already quiesced.
  4. It then asks the quiesce thread for the open txg. The quiesce thread closes it, opens the next txg at once so that writes never wait, and waits until every CPU’s counter of the closed txg is 0. It then hands the txg to the sync thread.
  5. The sync thread writes the txg in passes: the first writes the user data and the metadata above it; each pass allocates space, which changes space maps in the MOS, so it runs again until a pass leaves the MOS clean. Then vdev_config_sync writes the labels and the new uberblock: that one write commits the txg. A txg in which nothing changed writes nothing at all.
  6. Normally the next txg stays open while one syncs: the quiesce thread closes a txg only when someone asks. A caller that needs a new txg at once, such as dmu_free_long_range after a txg filled with frees, calls txg_wait_open, and then all three states are busy. There is never more than one txg per state.

The idea, at three depths

In one sentence

ZFS gathers every change into a numbered transaction group, closes it once no writer is still inside, and writes it out as a whole, so one uberblock write commits thousands of changes at a moment when none is half done.

In one paragraph

Every change to the pool is a transaction that joins the open txg. Two kernel threads per pool move txgs along. The sync thread wakes when zfs_txg_timeout (5 seconds) has passed since the last sync began, or sooner when someone needs a sync: zpool sync, or a txg that holds too much dirty data. It asks the quiesce thread for the open txg. The quiesce thread opens the next txg, waits until no transaction holds the old one, and hands it over. The sync thread then writes it out in passes until nothing is left to write, and commits it with the uberblock. Meanwhile the next txg is already taking writes. A txg in which nothing changed writes nothing, not even an uberblock. A program that needs its data on disk at once, with fsync, normally does not wait for a txg: it goes through the ZIL (room 10).

In detail

Joining the open txg. dmu_tx_assign calls txg_hold_open, which reads tx_open_txg and raises a counter of the CPU it runs on, tc_count[txg & TXG_MASK]. Per-txg state is kept in arrays of TXG_SIZE = 4, indexed by the low two bits of the txg number, so four consecutive txgs never share a slot. When the transaction is done, dmu_tx_commit calls txg_rele_to_sync, which lowers the counter and wakes the quiesce thread when it reaches 0. The counters are per CPU because a single one “would result in heavy lock contention”; the figure draws these counters for 4 CPUs and the txg numbers of each state above them.

When a txg ends. The sync thread decides, in its wait loop.

txg_sync_thread Excerpt of module/zfs/txg.c, lines 532–553, OpenZFS zfs-2.4.4. View on GitHub 
	for (;;) {
		clock_t timeout = zfs_txg_timeout * hz;
		clock_t timer;
		uint64_t txg;

		/*
		 * We sync when we're scanning, there's someone waiting
		 * on us, or the quiesce thread has handed off a txg to
		 * us, or we have reached our timeout.
		 */
		timer = (delta >= timeout ? 0 : timeout - delta);
		while (!dsl_scan_active(dp->dp_scan) &&
		    !tx->tx_exiting && timer > 0 &&
		    tx->tx_synced_txg >= tx->tx_sync_txg_waiting &&
		    !txg_has_quiesced_to_sync(dp)) {
			dprintf("waiting; tx_synced=%llu waiting=%llu dp=%p\n",
			    (u_longlong_t)tx->tx_synced_txg,
			    (u_longlong_t)tx->tx_sync_txg_waiting, dp);
			txg_thread_wait(tx, &cpr, &tx->tx_sync_more_cv, timer);
			delta = ddi_get_lbolt() - start;
			timer = (delta > timeout ? 0 : timeout - delta);
		}
  1. L533 The timeout is zfs_txg_timeout seconds, 5 by default, in clock ticks.
  2. L542 timer counts from start, set when the previous spa_sync began: the 5 seconds run from the start of the last sync, not from its end.
  3. L543–L546 The thread sleeps while all of these hold: no scan is active (a scrub, a resilver, or a background destroy or free), the timer has not run out, nobody waits for a txg past the last synced one, and no quiesced txg is waiting. Then it asks the quiesce thread for the open txg.

Two kinds of callers raise tx_sync_txg_waiting. zpool sync ends in txg_wait_synced(dp, 0), which waits for the open txg plus TXG_DEFER_SIZE, 2: in the figure, it syncs three txgs in a row. And each change adds its size to the dirty data of its txg; past zfs_dirty_data_sync_percent, 20%, of zfs_dirty_data_max (by default 10% of memory, capped), dsl_pool_dirty_space sees it and calls txg_kick, so a busy pool syncs well before 5 seconds.

Closing it. The quiesce thread runs txg_quiesce.

txg_quiesce Excerpt of module/zfs/txg.c, lines 395–428, OpenZFS zfs-2.4.4. View on GitHub 
	/*
	 * Grab all tc_open_locks so nobody else can get into this txg.
	 */
	for (c = 0; c < max_ncpus; c++)
		mutex_enter(&tx->tx_cpu[c].tc_open_lock);

	ASSERT(txg == tx->tx_open_txg);
	tx->tx_open_txg++;
	tx->tx_open_time = tx_open_time = gethrtime();

	DTRACE_PROBE2(txg__quiescing, dsl_pool_t *, dp, uint64_t, txg);
	DTRACE_PROBE2(txg__opened, dsl_pool_t *, dp, uint64_t, tx->tx_open_txg);

	/*
	 * Now that we've incremented tx_open_txg, we can let threads
	 * enter the next transaction group.
	 */
	for (c = 0; c < max_ncpus; c++)
		mutex_exit(&tx->tx_cpu[c].tc_open_lock);

	spa_txg_history_set(dp->dp_spa, txg, TXG_STATE_OPEN, tx_open_time);
	spa_txg_history_add(dp->dp_spa, txg + 1, tx_open_time);

	/*
	 * Quiesce the transaction group by waiting for everyone to
	 * call txg_rele_to_sync() for their open transaction handles.
	 */
	for (c = 0; c < max_ncpus; c++) {
		tx_cpu_t *tc = &tx->tx_cpu[c];
		mutex_enter(&tc->tc_lock);
		while (tc->tc_count[g] != 0)
			cv_wait(&tc->tc_cv[g], &tc->tc_lock);
		mutex_exit(&tc->tc_lock);
	}
  1. L398–L399 Taking every CPU’s tc_open_lock keeps new writers out of this txg: txg_hold_open takes the lock of its CPU before it reads tx_open_txg.
  2. L402 The open txg number goes up by one: from here on, new writes join the next txg.
  3. L412–L413 The locks are released at once. The next txg is open before the old one is drained, so writers do not wait.
  4. L422–L428 For each CPU in turn, wait until its count for this txg is 0. Writes that joined before the switch finish here. When the loop ends, nothing can change the txg any more.

One txg per state. The quiesce thread’s loop is where the rule is kept.

txg_quiesce_thread Excerpt of module/zfs/txg.c, lines 628–667, OpenZFS zfs-2.4.4. View on GitHub 
	for (;;) {
		uint64_t txg;

		/*
		 * We quiesce when there's someone waiting on us.
		 * However, we can only have one txg in "quiescing" or
		 * "quiesced, waiting to sync" state.  So we wait until
		 * the "quiesced, waiting to sync" txg has been consumed
		 * by the sync thread.
		 */
		while (!tx->tx_exiting &&
		    (tx->tx_open_txg >= tx->tx_quiesce_txg_waiting ||
		    txg_has_quiesced_to_sync(dp)))
			txg_thread_wait(tx, &cpr, &tx->tx_quiesce_more_cv, 0);

		if (tx->tx_exiting)
			txg_thread_exit(tx, &cpr, &tx->tx_quiesce_thread);

		txg = tx->tx_open_txg;
		dprintf("txg=%llu quiesce_txg=%llu sync_txg=%llu\n",
		    (u_longlong_t)txg,
		    (u_longlong_t)tx->tx_quiesce_txg_waiting,
		    (u_longlong_t)tx->tx_sync_txg_waiting);
		tx->tx_quiescing_txg = txg;

		mutex_exit(&tx->tx_sync_lock);
		txg_quiesce(dp, txg);
		mutex_enter(&tx->tx_sync_lock);

		/*
		 * Hand this txg off to the sync thread.
		 */
		dprintf("quiesce done, handing off txg %llu\n",
		    (u_longlong_t)txg);
		tx->tx_quiescing_txg = 0;
		tx->tx_quiesced_txg = txg;
		DTRACE_PROBE2(txg__quiesced, dsl_pool_t *, dp, uint64_t, txg);
		cv_broadcast(&tx->tx_sync_more_cv);
		cv_broadcast(&tx->tx_quiesce_done_cv);
	}
  1. L631–L641 It sleeps until someone asks for a txg past the open one, and as long as the last txg it quiesced has not been taken by the sync thread. One txg at most is quiescing or quiesced.
  2. L646–L654 It takes the open txg, records it in tx_quiescing_txg, and runs txg_quiesce with the pool’s sync lock released.
  3. L662–L666 The hand-off: the txg moves to tx_quiesced_txg and the sync thread is woken. If the sync thread is still writing the previous txg, this one waits there, quiesced.

Who asks? Normally the sync thread itself, when it is ready for more work. So while one txg syncs, the next usually stays open and keeps taking writes. One caller asks early: txg_wait_open(dp, 0, B_TRUE), which at this tag dmu_free_long_range calls “to avoid filling up a TXG with just frees”. Then the next txg quiesces during the sync, and the three states are busy at once: in the figure, “Free a large file” while a txg syncs.

The hand-off to the sync thread.

txg_sync_thread Excerpt of module/zfs/txg.c, lines 583–610, OpenZFS zfs-2.4.4. View on GitHub 
		/*
		 * Consume the quiesced txg which has been handed off to
		 * us.  This may cause the quiescing thread to now be
		 * able to quiesce another txg, so we must signal it.
		 */
		ASSERT(tx->tx_quiesced_txg != 0);
		txg = tx->tx_quiesced_txg;
		tx->tx_quiesced_txg = 0;
		tx->tx_syncing_txg = txg;
		DTRACE_PROBE2(txg__syncing, dsl_pool_t *, dp, uint64_t, txg);
		cv_broadcast(&tx->tx_quiesce_more_cv);

		dprintf("txg=%llu quiesce_txg=%llu sync_txg=%llu\n",
		    (u_longlong_t)txg, (u_longlong_t)tx->tx_quiesce_txg_waiting,
		    (u_longlong_t)tx->tx_sync_txg_waiting);
		mutex_exit(&tx->tx_sync_lock);

		txg_stat_t *ts = spa_txg_history_init_io(spa, txg, dp);
		start = ddi_get_lbolt();
		spa_sync(spa, txg);
		delta = ddi_get_lbolt() - start;
		spa_txg_history_fini_io(spa, ts);

		mutex_enter(&tx->tx_sync_lock);
		tx->tx_synced_txg = txg;
		tx->tx_syncing_txg = 0;
		DTRACE_PROBE2(txg__synced, dsl_pool_t *, dp, uint64_t, txg);
		cv_broadcast(&tx->tx_sync_done_cv);
  1. L588–L593 The sync thread takes the quiesced txg, records it as tx_syncing_txg, and wakes the quiesce thread, which can now quiesce another txg if one was asked for.
  2. L601–L603 spa_sync does all the I/O of the txg. start is the moment the timeout of the next wait counts from.
  3. L607–L610 The txg is synced: tx_synced_txg, and everyone in txg_wait_synced is woken.

Writing until nothing changes. Writing a block allocates space, which changes a space map, which is metadata in the MOS, which then has to be written too. So spa_sync runs passes. The first writes all the modified user data; the next ones write only metadata, what the previous pass dirtied. To make it converge, from pass 2 a block born in this txg is rewritten in place instead of reallocated (the copy-on-write room shows where), and from pass 2 frees are deferred to a later txg (without the log spacemap feature).

spa_sync_iterate_to_convergence Excerpt of module/zfs/spa.c, lines 10743–10766, OpenZFS zfs-2.4.4. View on GitHub 
		 * (e.g. if we have sync tasks but no dirty user data). We need
		 * to check the uberblock's rootbp because it is updated if we
		 * have synced out dirty data (though in this case the MOS will
		 * most likely also be dirty due to second order effects, we
		 * don't want to rely on that here).
		 */
		if (pass == 1 &&
		    BP_GET_LOGICAL_BIRTH(&spa->spa_uberblock.ub_rootbp) < txg &&
		    !dmu_objset_is_dirty(mos, txg)) {
			/*
			 * Nothing changed on the first pass, therefore this
			 * TXG is a no-op. Avoid syncing deferred frees, so
			 * that we can keep this TXG as a no-op.
			 */
			ASSERT(txg_list_empty(&dp->dp_dirty_datasets, txg));
			ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg));
			ASSERT(txg_list_empty(&dp->dp_sync_tasks, txg));
			ASSERT(txg_list_empty(&dp->dp_early_sync_tasks, txg));
			break;
		}

		spa_sync_deferred_frees(spa, tx);
	} while (dmu_objset_is_dirty(mos, txg));
}
  1. L10749–L10764 If pass 1 finds nothing to write, the root block pointer was not born in this txg and the MOS is clean: the txg is a no-op, and the loop stops.
  2. L10765 Otherwise, another pass as long as the MOS is dirty in this txg. The loop ends when a pass changes nothing.

Then spa_sync_rewrite_vdev_config calls vdev_config_sync, which flushes the write cache of every disk written in this txg, then writes the even labels, the uberblock and the odd labels. The labels are rewritten only on the vdevs whose configuration changed in this txg. The uberblock goes to every top-level vdev if the configuration changed, else to up to three top-level vdevs, taken in order from a random one, and then it is the only new thing in their labels. The uberblock write is the commit, and its cost (a flush per disk, label writes) is paid once per txg, not once per write. If nothing changed in the txg, the uberblock reports no change and nothing is written (unless multihost protection is on): an idle pool goes through a txg every 5 seconds without writing to its disks. The Hall shows where the uberblock lands in the ring.

What the figure leaves out

The figure runs the two threads’ wait conditions as the code writes them, over the same variables. It draws 4 CPUs; each write is one block, and the dirty-data threshold is scaled to 4 blocks (20% of a 20-block zfs_dirty_data_max). Time is slowed, and the length of a write or a pass is made up. It leaves out the write throttle (writers delayed or blocked as dirty data grows), the wait on tc_open_lock while a txg closes, scrubs and resilvers, synctasks, the ZIL, and pool suspension. Pass sizes are illustrative: a txg with data always takes three passes here.