# ZFS **URL:** \ **Category:** home storage\ **Tags:** storage\ **Created:** [May 31, 2022, 2:00pm UTC](https://discuss.88.io/t/zfs/330 "2022-05-31T14:00:00Z")\ **Posts on this page:** 9\ **Page:** 1

**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [May 31, 2022, 2:00pm UTC](https://discuss.88.io/t/zfs/330/1 "2022-05-31T14:00:00Z")

# OpenZFS Infinite Disk's design is file system agnostic, it can be build by combining multiple file system technologies like LVM, XFS, Bcache, mdadm etc. However, [OpenZFS](https://openzfs.org) happens to have most of those technologies in one piece of software, so it is used to in the first implement of Infinite Disk. When we use the word "ZFS" we mean the original ZFS developed and open sourced by Sun and then continued as OpenZFS, not the closed source ZFS continued by Oracle. ZFS has been designed to be [the last word in file systems](https://web.archive.org/web/20060428092023/http://www.sun.com/2004-0914/feature/) and it is amazing! Good Introductions: 1. [https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/](https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/) 2. [https://openzfs.github.io/openzfs-docs/](https://openzfs.github.io/openzfs-docs/) # Making ZFS Distributed Infinite Disk took the solid full featured battle tested ZFS and made it into a distributed file system working across thousands of machines and made it work over the wide area network. A good comparison of some traditional distributed file systems is here: [https://arxiv.org/ftp/arxiv/papers/1904/1904.03997.pdf](https://arxiv.org/ftp/arxiv/papers/1904/1904.03997.pdf) Note the IBM zFS mentioned in the PDF above is NOT the same as the Sun developed [ZFS](https://en.wikipedia.org/wiki/ZFS) we are talking about here (that OpenZFS is based on). # Supported Versions The latest version of OpenZFS should be used, currently minimum supported OpenZFS version is 2.1.6 > **[Releases · openzfs/zfs](https://github.com/openzfs/zfs/releases)** > > OpenZFS on Linux and FreeBSD. Contribute to openzfs/zfs development by creating an account on GitHub. ---

**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [June 27, 2022, 10:18pm UTC](https://discuss.88.io/t/zfs/330/2 "2022-06-27T22:18:28Z")

# Lustre and ZFS [Lustre](https://www.lustre.org/) is the [dominant file system](https://en.wikipedia.org/wiki/Lustre_(file_system)) in the super computing and large scale data storage. It is increasingly using OpenZFS as it backend storage: [https://whamcloud.github.io/Online-Help/docs/Create\_and\_manage\_ZFS\_based\_LFS\_8\_0.html](https://whamcloud.github.io/Online-Help/docs/Create_and_manage_ZFS_based_LFS_8_0.html) > **[Bolstering Lustre on ZFS: Highlights of Continuing Work](https://www.nextplatform.com/2017/01/11/bolstering-lustre-zfs-highlights-continuing-work/)** > > The Zettabyte File System (ZFS), as a back-end file system to Lustre, has had support in Lustre for a long time. But in the last few years it has gained **High Performance Computing** Similar to the high performance computing efforts above, ZFS is used in the backend, but [Infinite Disk](https://infinitedisk.com) also take advantage of ZFS features in the frontend and in intermediate distribution layer, giving Infinite Disk substantial advantages over traditional distributed file system like Lustre and Spectrum Scale in a lot of high performance computing use cases. ---

# ZFS Snapshot Replication A popular way of using Infinite Disk is as replication target for ZFS snapshots using the ZFS send protocol. [https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html](https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html) Due to Infinite Disk's use of OpenZFS Native Encryption, all zfs send command MUST have the "-w" or "-raw" option to enable raw send so encryption is preserved. Some tools to make "zfs send" easier: 1. [GitHub - zrepl/zrepl: One-stop ZFS backup & replication solution](https://github.com/zrepl/zrepl) 2. [GitHub - psy0rz/zfs\_autobackup: ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable.](https://github.com/psy0rz/zfs_autobackup) 3. [https://www.znapzend.org/](https://www.znapzend.org/) [Zrepl](https://zrepl.github.io/) is the default snapshot replication. ---

# ZFS for Geeks ZFS can be tuned easily for almost every application. ## 1. On Disk Format 1. [Zfs\_ondiskformat.pdf](https://discuss.88.io/uploads/short-url/fbE0HQTBagqaygOWyfuYyzTT0VN.pdf) (495.5 KB) 2. [Your browser is suspiciously without a Sec-Fetch-Mode header](https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSBlockPointers) 3. [Your browser is suspiciously without a Sec-Fetch-Mode header](https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSLogicalVsPhysicalBlockSizes) ## 2. Reliability 1. [https://www.usenix.org/legacy/events/fast10/tech/full\_papers/zhang.pdf](https://www.usenix.org/legacy/events/fast10/tech/full_papers/zhang.pdf) ## 3. RAID 1. [draid Declustered RAID for ZFS Installation and Configuration Guide High Performance Data Division.pdf](https://discuss.88.io/uploads/short-url/h1RGwjl8dknLOzGpZIf3x3y663z.pdf) (1.8 MB) ## 4. Recovery If you cannot import a zpool, the following command might help: ```plaintext zpool import -FfmX zpool clear ``` If the above does not work, you will can try the really hardcore -T option and roll-back to a good tgx - this is not for the faint-hearted: Find a good tgx with ```plaintext zpool history -il or zdb ``` Rollback to the good tgx ```plaintext zpool import -f -F -T ``` Mount it and dd some zeros to clear out the more recent bad tgx. There is a script called [zfs\_revert-0.1.py](https://gist.github.com/mkhon/34d979c78077a20648456272d7f2cc15) which follows a similar way of recovery. The comment at the end of that page is especially good at showing how to find a "good" tgx. ## 5. Encryption 1. [How-To: Using ZFS Encryption at Rest in OpenZFS (ZFS on Linux, ZFS on FreeBSD, …) | Philipp's Tech Blog](https://blog.heckel.io/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/) 2. [https://klarasystems.com/articles/openzfs-native-encryption/](https://klarasystems.com/articles/openzfs-native-encryption/) 3. [https://github.com/illumos/illumos-gate/commit/eb633035c80613ec93d62f90482837adaaf21a0a](https://github.com/illumos/illumos-gate/commit/eb633035c80613ec93d62f90482837adaaf21a0a) | Encrypted | Not Encrypted | | --- | --- | | File data and metadata | Everything in RAM | | ACLs, names, permissions, attrs | Dataset / snapshot names | | Directory listings | Dataset properties | | All Zvol data | Pool layout | | FUID Mappings | ZFS Structure | | Master encryption keys | Dedup tables | | All of the above in the L2ARC | | | All of the above in the ZIL | | In case zfs encryption can be broken in some way, infinite disk provides [additional layers of security](https://discuss.88.io/t/infinite-disk/331/9) on top of zfs encryption. ---

# ZFS Metadata ## 1. Metadata Size ```auto zpool list yourpool -v ``` Look under special device ```auto # zpool list -v m440qv2qrt9t0r3v8euv NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT m440qv2qrt9t0r3v8euv 64.0G 20.2G 43.8G - - 15% 31% 1.00x ONLINE - mirror-0 63.5G 20.1G 43.4G - - 15% 31.6% - ONLINE nbd1 64G - - - - - - - ONLINE nbd0 64G - - - - - - - ONLINE special - - - - - - - - - mirror-1 480M 154M 326M - - 54% 32.0% - ONLINE nbd97 488M - - - - - - - ONLINE nbd98 488M - - - - - - - ONLINE logs - - - - - - - - - sdb2 488M 260K 480M - - 0% 0.05% - ONLINE ``` ## 2. Metadata Size (different calculation - interesting but not accurate) ```auto zdb -PLbbbs yourpool | tee ~/yourpool_metadata_output.txt ``` Sum the relevant sections in the ASIZE column ```auto ( cat ~/yourpool_metadata_output.txt \ | grep -B 9999 'L1 Total' \ | grep -A 9999 'ASIZE' \ | grep -v \ -e 'L1 object array' -e 'L0 object array' \ -e 'L1 bpobj' -e 'L0 bpobj' \ -e 'L2 SPA space map' -e 'L1 SPA space map' -e 'L0 SPA space map' \ -e 'L5 DMU dnode' -e 'L4 DMU dnode' -e 'L3 DMU dnode' -e 'L2 DMU dnode' -e 'L1 DMU dnode' -e 'L0 DMU dnode' \ -e 'L0 ZFS plain file' -e 'ZFS plain file' \ -e 'L2 ZFS directory' -e 'L1 ZFS directory' -e 'L0 ZFS directory' \ -e 'L3 zvol object' -e 'L2 zvol object' -e 'L1 zvol object' -e 'L0 zvol object' \ -e 'L1 SPA history' -e 'L0 SPA history' \ -e 'L1 deferred free' -e 'L0 deferred free' \ | awk \ '{sum+=$4} \ END {printf "\nTotal Metadata\n %.0f Bytes\n" " %.2f GiB\n",sum,sum/1073741824}' \ ) ``` Sample output: ```auto Total Metadata 57844416512 Bytes 53.87 GiB ``` Reference: > **[ZFS Metadata Special Device: Z](https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954/96)** > > 2023 Edit: This script seemed to work with a freshly made pool, but doesn’t seem to work with my pool which is now more “mature” and has more data on the special vdevs than this script reports. As such this script should be considered ultimately... ## 3. Metadata Device Tips > **[ZFS Metadata Special Device: Z](https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954)** > > Introduction ZFS Allocation Classes: It isn’t storage tiers or caching, but gosh darn it, you can really REALLY speed up your zfs pool. From the manual: Special Allocation Class The allocations in the special class are dedicated to specific... ---

# ZFS Management Infinite Disk comes with its own Zabbix based management console. Other ZFS related management tools: 1. [GitHub - AnalogJ/scrutiny: Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds](https://github.com/AnalogJ/scrutiny) 2. [GitHub - chadmiller/zpool-iostat-viz: "zpool iostats" for humans; find the slow parts of your ZFS pool](https://github.com/chadmiller/zpool-iostat-viz) ---

# L2ARC Original comments inside Sun ZFS source code usr/src/uts/common/fs/zfs/arc.c on how L2ARC works. ```auto /* * Level 2 ARC * * The level 2 ARC (L2ARC) is a cache layer in-between main memory and disk. * It uses dedicated storage devices to hold cached data, which are populated * using large infrequent writes. The main role of this cache is to boost * the performance of random read workloads. The intended L2ARC devices * include short-stroked disks, solid state disks, and other media with * substantially faster read latency than disk. * * +-----------------------+ * | ARC | * +-----------------------+ * | ^ ^ * | | | * l2arc_feed_thread() arc_read() * | | | * | l2arc read | * V | | * +---------------+ | * | L2ARC | | * +---------------+ | * | ^ | * l2arc_write() | | * | | | * V | | * +-------+ +-------+ * | vdev | | vdev | * | cache | | cache | * +-------+ +-------+ * +=========+ .-----. * : L2ARC : |-__-| * : devices : | Disks | * +=========+ `-__-' * * Read requests are satisfied from the following sources, in order: * * 1) ARC * 2) vdev cache of L2ARC devices * 3) L2ARC devices * 4) vdev cache of disks * 5) disks * * Some L2ARC device types exhibit extremely slow write performance. * To accommodate for this there are some significant differences between * the L2ARC and traditional cache design: * * 1. There is no eviction path from the ARC to the L2ARC. Evictions from * the ARC behave as usual, freeing buffers and placing headers on ghost * lists. The ARC does not send buffers to the L2ARC during eviction as * this would add inflated write latencies for all ARC memory pressure. * * 2. The L2ARC attempts to cache data from the ARC before it is evicted. * It does this by periodically scanning buffers from the eviction-end of * the MFU and MRU ARC lists, copying them to the L2ARC devices if they are * not already there. It scans until a headroom of buffers is satisfied, * which itself is a buffer for ARC eviction. The thread that does this is * l2arc_feed_thread(), illustrated below; example sizes are included to * provide a better sense of ratio than this diagram: * * head --> tail * +---------------------+----------+ * ARC_mfu |:::::#:::::::::::::::|o#o###o###|-->. # already on L2ARC * +---------------------+----------+ | o L2ARC eligible * ARC_mru |:#:::::::::::::::::::|#o#ooo####|-->| : ARC buffer * +---------------------+----------+ | * 15.9 Gbytes ^ 32 Mbytes | * headroom | * l2arc_feed_thread() * | * l2arc write hand <--[oooo]--' * | 8 Mbyte * | write max * V * +==============================+ * L2ARC dev |####|#|###|###| |####| ... | * +==============================+ * 32 Gbytes * * 3. If an ARC buffer is copied to the L2ARC but then hit instead of * evicted, then the L2ARC has cached a buffer much sooner than it probably * needed to, potentially wasting L2ARC device bandwidth and storage. It is * safe to say that this is an uncommon case, since buffers at the end of * the ARC lists have moved there due to inactivity. * * 4. If the ARC evicts faster than the L2ARC can maintain a headroom, * then the L2ARC simply misses copying some buffers. This serves as a * pressure valve to prevent heavy read workloads from both stalling the ARC * with waits and clogging the L2ARC with writes. This also helps prevent * the potential for the L2ARC to churn if it attempts to cache content too * quickly, such as during backups of the entire pool. * * 5. After system boot and before the ARC has filled main memory, there are * no evictions from the ARC and so the tails of the ARC_mfu and ARC_mru * lists can remain mostly static. Instead of searching from tail of these * lists as pictured, the l2arc_feed_thread() will search from the list heads * for eligible buffers, greatly increasing its chance of finding them. * * The L2ARC device write speed is also boosted during this time so that * the L2ARC warms up faster. Since there have been no ARC evictions yet, * there are no L2ARC reads, and no fear of degrading read performance * through increased writes. * * 6. Writes to the L2ARC devices are grouped and sent in-sequence, so that * the vdev queue can aggregate them into larger and fewer writes. Each * device is written to in a rotor fashion, sweeping writes through * available space then repeating. * * 7. The L2ARC does not store dirty content. It never needs to flush * write buffers back to disk based storage. * * 8. If an ARC buffer is written (and dirtied) which also exists in the * L2ARC, the now stale L2ARC buffer is immediately dropped. * * The performance of the L2ARC can be tweaked by a number of tunables, which * may be necessary for different workloads: * * l2arc_write_max max write bytes per interval * l2arc_write_boost extra write bytes during device warmup * l2arc_noprefetch skip caching prefetched buffers * l2arc_headroom number of max device writes to precache * l2arc_feed_secs seconds between L2ARC writing * * Tunables may be removed or added as future performance improvements are * integrated, and also may become zpool properties. */ ``` ---

**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [September 2, 2023, 5:56am UTC](https://discuss.88.io/t/zfs/330/8 "2023-09-02T05:56:10Z")

# ZIL > **[Solaris ZFS, Synchronous Writes and the ZIL Explained – Tech, AI and...](https://constantin.glez.de/posts/2010-07-20-solaris-zfs-synchronous-writes-and-the-zil-explained/)** > > When talking to customers, partners and colleagues about Oracle Solaris ZFS performance, one topic almost always seems to pop up: Synchronous writes and the ZIL. > In fact, most ZFS performance problems… - zilstat [Richard Elling Consulting - zilstat](https://www.richardelling.com/richard-elling-consulting/scripts-and-programs/zilstat) ---

**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [May 28, 2024, 3:08am UTC](https://discuss.88.io/t/zfs/330/9 "2024-05-28T03:08:44Z")

# ZFS LXC There are increasing support of ZFS in LXC: 1. [https://github.com/canonical/lxd/pull/12056](https://github.com/canonical/lxd/pull/12056)