VMware NFS datastores on ZFS – Why so slow?

Posted: March 22nd, 2010 | Author: | Filed under: Storage, Virtualization | Tags: , , , | 3 Comments »

I will go straight to the point, your problem is ZIL (ZFS Intent Log). But what does it mean and how can I avoid that?

Well the answer is not simple, first things first:

  • VMware will ALWAYS write to the NFS datastore using FSYNC and you cannot change its behaviour.
  • FSYNC is used to protect your data, so circumventing it can impact data integrity in the event of a failure.
  • An FSYNC call will force the NFS Server to issue a FSYNC call to ZFS as well, it means that the system will not return an acknowledge on the write until the data is written to “Stable Storage” (i.e. the disks)

Now, to increase performances we can go through two paths:

  • Add a ZIL device on a fast medium (like a SSD or battery-backed NVRAM).
  • Disable ZIL.

Obviously the former is the RIGHT way to do it, because it doesn’t compromise the data path, everything is protected and production safe.

*BUT*, it’s expensive. *VERY* expensive.

If you’re running an home lab environment, or a test environment maybe you can disable the ZIL, but what are the trade-offs?

  • Possible data corruption if a server crash occurs between the acknowledge of a write and the effective commit on stable storage.
  • The knob to disable the ZIL is system-wise so every ZFS on the system will have its ZIL disabled.

If you’re conscious of the risks involved you can proceed to disable the ZIL with this simple command:

echo zil_disable/W0t1 | mdb -kw

After issuing the command you need to umount/mount or export/import the pools, also the config is not persistent across reboots, if you want to make it persistent you need to add this line to your /etc/system file:

set zfs:zil_disable=1

That will made the modification persistent.

This parameter will be obsoleted if this Opensolaris Bug will be fixed: CR 6280630 – zil synchronicity.

Instead, if you have a fast device handy, like a small SLC 18G SSD, you can just add it as a separate ZIL device to your ZPOOL using this command:

zpool add <pool> log <log devices>

This will move all the ZIL processing (which is the “journaling” of ZFS) to this device, resulting in a dramatic improvement in synchronous operations, without the data corruption :) .

Feedback and Comments are very welcomed!

Related Posts