circuit_breaking_exception in SO

[rampart51]

I am once again receiving the circuit_breaking_exception error when reviewing Alerts and/or Hunt data. I've increased the ES Heap but have found that after a period of a few weeks the error kicks up again. The ES heap is adjusted after each occurrence of the circuit_breaking_exception.

I am running SO 2.3.2 on CentOS 7 with 4 Xeon cores 3.00GHz; 64GB memory; 11TB for /nsm with ~20% used.

Any thoughts? Is this a potential memory leak?

[TOoSmOotH]

This is not a memory leak it is the lucene/ES indices. Since SO uses a daily index, each day will load the lucene index for that day into memory. Once those indices run out of heap you need to not have as many loaded aka fewer days in an open state, or increase the heap size. There are diminishing returns at some point with how large you can make the heap. You can control the number of open days per index if you would like to leave say 45 days of ids alerts open and back zeek down to say 3 weeks etc in global.sls.

[rampart51]

I was able to prevent this for some time with changes to heap. I now see the "diminishing" returns with constantly doing that. This is a sample from my global.sls. I assume the entries (except shards) are 'days'. My question now is... is the 'warm' setting the one that should be adjusted for modules of concern?

so-beats:
shards: 1
warm: 7
close: 30
delete: 365

[weslambert]

You can ignore the warm setting at this time. This is for a specialized use case and wouldn't come into play here.