Ytdyklly

I'm trying to check how my ZFS pool's ARC is used, motivated by "do I need more RAM, even though expensive".

I have 128GB fast ECC and using NVMe SSDs for L2ARC, but it's still doing a lot of small IO read thrashing related to a mix of DDT/spacemaps. The system is specced for DDT and has about 3.5x ~ 4x dedup ratio, so please no "DDT is baad" replies, I know that, but need DDT, so I'm working to minimise the remaining read-thrash by ensuring at least that ddt/spacemap metadata is pretty much all retained in ARC once the system is warm.

The way I'm expecting RAM to be used is - DDT is about 35-40GB, and I've used sysctls to reserve 85GB of ARC for metadata. I've also set spacemap block size to a larger size, and defragged the pool (copied to new pool), which looks like it's helping a lot. But because I can't see metrics for how much of different types of metadata is loaded or evicted (ddt/spacemap/other), and there aren't tools to set ZFS ddt block size or preload DDT entries to ARC, I'm in the dark as to the exact impact, or whether more RAM will help, or other systematic ways to do better.

I've looked for solutions. zdb, arc-stats etc don't expose a breakdown of the metadata ARC situation, just a lump sum for all metadata.

Is there a straightforward way to get a sense what's going on, in order to assess whether more RAM will help, even if it's not precise, or to get some better sense (even if imprecise) of breakdown of the amounts of ddt/spacemap/"other" metadata MRU/MFU being loaded/cached/evicted?

I don’t think there’s any built-in tool like arcstats for this purpose, especially since you’re on FreeBSD, which I’m guessing doesn’t have mdb (from illumos / Solaris).

The simplest solution would be giving a little bit more RAM and seeing if that helps. Of course, that costs money, but it might be less expensive than your time trying to figure out the answer (if you’re being paid for this work).

The next easiest thing to try would be fussing with the ARC memory limits while running some test workload. The impact these have on workloads is often unintuitive, so my recommendation would be to start with all default settings, then gradually change stuff to make it more complicated. It’s possible that e.g. when you tried to set the min memory reserved for metadata, you actually set the max by mistake, which would cause thrashing like you described — so make sure to read the “docs” for these settings carefully.

Finally, if you’re a bit intrepid and fairly confident in your ability to analyze complicated data to determine the answer to your question, you can also use DTrace for this. This probe is triggered each time a cache miss happens in the ARC:

DTRACE_PROBE4(arc__miss, arc_buf_hdr_t *, hdr, blkptr_t *, bp,

    uint64_t, lsize, zbookmark_phys_t *, zb);

So, you could write a D script that listens for the :::arc_miss probe, and use args[0], args[1], and / or the backtrace to figure out what type of requests are missing the cache.

I suspect the easiest way to go is looking at the type of the blockptr_t in args[1]. Unfortunately that’s somewhat annoying to extract because it’s part of a bitfield. The definition of the block pointer object can be found here, and you want your DTrace script to output the same thing that BP_GET_TYPE(args[1]) would output, and then interpret those values by comparing to values of dmu_object_type from here.

Alternately, I can recommend a simpler script that’s potentially more involved to interpret. It would collect the backtrace every time the probe fires, and then you can post-process the traces to make a flame graph for easier interpretation. The method names in ZFS are pretty descriptive in general (at least, they all have acronyms, e.g. ddt for “dedup table” that you can search online), so you can probably figure out what the callers are doing that way.

If there’s a ton of values showing up that aren’t file or directory data, you probably need to keep more metadata in cache. You can do that either by dedicating more space to it using the tunables, or by giving the machine more RAM.