Ytdyklly

I am trying to build small CEPH cluster on single box - testing in the lab, before switching bigger probject to it. Run into memory problems and can not find way to control it.

We have old Core2Duo CPU, 4gb ram. Creating 3 OSDs with 3 local 8tb disks.

ceph-deploy will make it and everything runs nice, expcet that for each of 3 OSDs 1 tmpfs partition is being created, which is 2gb and after copying ~50gb of data to CephFS Bluestore - box starts agressively using RAM and ends up with using all the swap. I failed to find correct setting to control, how much RAM OSD process is allowed to use.

It is ok to use more IO, instead of RAM. Looking for help (if that is possible). :) I am building everything on Centos 7.

This is a common issue, the ceph mailing list is a very helpful archive for these questions, see this. You can reduce the bluestore_cache_size values, the defaults are 3GB for a SSD and 1 GB for a HDD OSD:

# If bluestore_cache_size is zero, bluestore_cache_size_hdd or bluestore_cache_size_ssd will be used instead.



host1:~ # ceph daemon osd.3 config show | grep bluestore_cache_size

    "bluestore_cache_size": "0",

    "bluestore_cache_size_hdd": "1073741824",

    "bluestore_cache_size_ssd": "3221225472",

But according to our monitoring in a production cluster the residual memory is about 3 GB for a HDD-OSD, so you'll have to tweak those values to your needs. The config reference is here.

I would recommend to start with one OSD first then watch the performance of your node; then add another OSD if the memory (or other measurements) is ok. If you hit the limits with one or two OSDs already you'll have to adjust the configs according to your needs.
The values can be changed online running:

host1:~ # ceph daemon osd.<ID> config set bluestore_cache_size[_hdd|_ssd] <VALUE>

Permanent changes of configs have to be stored in /etc/ceph/ceph.conf

The Ceph 13.2.2 release notes says the following...

The bluestore_cache_* options are no longer needed. They are replaced
by osd_memory_target, defaulting to 4GB. BlueStore will expand and
contract its cache to attempt to stay within this limit. Users
upgrading should note this is a higher default than the previous
bluestore_cache_size of 1GB, so OSDs using BlueStore will use more
memory by default. For more details, see the BlueStore docs.

This caught me by surprise. My osds were going absolutely wild with resident memory usage. The kernel was oom-killing osd processes.

Changing over to the new key and bouncing the osd processes has given me stable performance.