Uberblok

A museum of ZFS, read in its source

Uberblok

Storage has always had the same enemies: disks that lie, writes cut short, a crash in the middle of an update. ZFS answers them with a handful of ideas. This museum shows them one room at a time, from the code that implements them.

Every animation maps to lines of OpenZFS zfs-2.4.4, frozen at commit 71a9f9578616, and every excerpt links to them. Nothing here is drawn from memory.

Why this museum?

Rendering of a strip of fan-fold listing paper printed with the OpenZFS file include/sys/uberblock_impl.h, its far end fading away. A round reading glass with a black rim and a black handle, held above it, enlarges the highlighted declaration of struct uberblock: uint64_t ub_magic, ub_version, ub_txg, ub_guid_sum, ub_timestamp and blkptr_t ub_rootbp stand out larger than the lines around them. The glass casts a soft shadow, with a bright spot, on the paper below.

Fig. 1. The Hall’s object under the glass: struct uberblock, lines 122–128 of include/sys/uberblock_impl.h at zfs-2.4.4, printed with lines 36–150 of its file.

  1. struct uberblock: the record that names the whole pool
  2. ub_txg: the transaction group of the last sync
  3. ub_rootbp: the block pointer to the root of the tree, the MOS
  4. The listing: lines 36 to 150 of the file, numbered as in the source

The object in the Hall: the uberblock

A pool is millions of blocks spread over several disks. After a crash, the system must find its current, consistent state without scanning any of them. So one small record names the root of everything: the uberblock. It holds a transaction group number, ub_txg, and one block pointer, ub_rootbp; every other block of the pool is reachable from that pointer.

Each disk carries four labels, two at its start and two at its end, and each label keeps a 128 KiB ring of uberblocks. A new uberblock never overwrites the previous one: the uberblock of txg N goes to slot N mod the number of slots (with multihost=on, the last slot is kept for MMP and the ring turns on one slot fewer). On import, ZFS reads every slot of every label and keeps the valid uberblock with the highest txg.

The ring is often said to hold 128 uberblocks. It holds 128 KiB: a slot takes one sector, rounded up to 1 KiB and capped at 8 KiB, so 128 slots on 512-byte sectors, 64 at ashift 11, 32 on the 4K-sector disks of today, 16 beyond. Change the sector size and watch the next txgs take their slots in turn.

txg 4211
ub_magic
0x00bab10c
ub_txg
4211
ub_rootbp
→ MOS

Ring of 32 slots of 4 KiB (ashift 12). The uberblock of txg 4211 is the newest: slot 19, because 4211 mod 32 = 19. The uberblock of txg 4210 is still intact in slot 18.

Every uberblock starts with the same magic number, 0x00bab10c: “oo-ba-bloc”.

Map of the visit

The rooms follow the data: the root and the tree, how the tree changes, where blocks land, what the tree makes possible, then the features built on top, and the engine room, where the museum says how it was made.

  1. Hall: the uberblock The one record that names the whole pool, and the ring that keeps it.
  2. History wing From the first lines of code to OpenZFS, every fact sourced.
  3. The tree and the uberblock Block pointers all the way down, and the write that makes a new tree the current one.
  4. Copy-on-write, checksums and self-healing Why a pool stays consistent without fsck, and how it catches a disk that lies.
  5. Transaction groups Open, quiescing, syncing: how thousands of writes become one commit.
  6. The vdevs Mirror, raidz, draid, log, cache, special: the tree of devices under a pool.
  7. RAID-Z Variable-width stripes, and why there is no write hole.
  8. Snapshots and clones Birth times and deadlists: which snapshot owns this block?
  9. The ARC A cache that learns from what it forgot.
  10. Metaslabs and space maps Finding free space fast, with an append-only log of allocations.
  11. ZIL and SLOG The path of a synchronous write.
  12. send / receive A dataset as a stream, and the increment between two snapshots.
  13. Compression, dedup, encryption Three transforms in the write pipeline, and the honest cost of dedup.
  14. Engine room How this museum was built, and from which commit.

Dedication

Why this museum?

Keeping data safe is critical work, and seldom seen. A file system is handed everything people keep, and it has to stay right through crashes, failing disks and writes cut short. One of the principles ZFS was designed on, in Bill Moore’s words: never, ever trust the underlying hardware.1

This museum pays homage to that piece of engineering, and to the people who built it. Jeff Bonwick led the development of ZFS, with Bill Moore as co-lead;1 Matt Ahrens was one of its two original authors.2 The work began in 2001 with two engineers at Sun Microsystems3 and was integrated into Solaris on 31 October 2005,4 a step Bonwick summed up as 584 files, 92,000 lines of change, 56 patents, 5 years.4 On 17 November he wrote: I’m incredibly proud of the ZFS team, and deeply grateful: no one person can do something on this scale.5

It pays homage, too, to the engineering at Sun in those years, where engineers published how their systems worked: the slab allocator, presented by Jeff Bonwick at USENIX in 1994;6 DTrace, by Bryan Cantrill, Michael Shapiro and Adam Leventhal, in 2004;7 and the source of ZFS itself, released under the CDDL as part of OpenSolaris.3

And to those who carry it on: OpenZFS, announced in 20132 and founded by members of the Linux, FreeBSD, Mac OS X and illumos communities.2 The release this museum reads lists more than 700 people in its AUTHORS file.

Rendering of a hard disk taken apart along its spindle: the base, the motor, three glossy brown platters, the clamp and, highest, the cover drawn in outline; the arms of the head stack reach over the platters.

Fig. 2. The underlying hardware: a 3.5-inch hard disk, exploded along its spindle.

  1. Top cover
  2. Disc clamp
  3. Platters
  4. Head stack: arms and heads
  5. Voice coil and its magnet
  6. Spindle motor
  7. Base casting
  1. 1 “A Conversation with Jeff Bonwick and Bill Moore”, ACM Queue 5(6), November 2007 (archived 2011-07-16)
  2. 2 OpenZFS launch announcement, open-zfs.org (archived 2013-09-20)
  3. 3 openzfs.org wiki, “History” (archived 2020-05-14)
  4. 4 Jeff Bonwick, “ZFS: The Last Word in Filesystems”, blog post of 31 October 2005 (archived 2007-02-07)
  5. 5 Jeff Bonwick, blog, posts of November 2005, “Thank You” of 17 November 2005 (archived 2006-12-19)
  6. 6 USENIX, abstract of “The Slab Allocator: An Object-Caching Kernel Memory Allocator”, Summer 1994 Technical Conference (archived 2001-02-21)
  7. 7 USENIX, abstract of “Dynamic Instrumentation of Production Systems”, 2004 Annual Technical Conference (archived 2006-01-07)