SOC Request failed with status 500

[ivantn21]

Deployed SO 2.3.61 today, configured fleet agent (3 in total) and no other services collecting data (no network monitoring, no other endpoint agents) and from nowhere got a Request failed with status 500 on SOC. I know the problem is with ElasticSearch and after hours of trying to troubleshoot looking at the log (/opt/so/log/elasticsearch/... and so on I can find the problem, has anyone seen this in the past?

I already added the transport.ping_request option to the salt elasticsearch config since it was suggested to a similar error in the past??

[ivantn21]

When accessing Kibana I can see that the problem might be related to the license?

{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}

[facyber]

Hi @ivantn21

I had similar issue recently and the problem was in my Nginx configuration file, as I copied to a local salt folder and edit it for my needs, but in the meantime it was updated and changed, so I had to create a new local copy of it and add what I need.

If you have manually changed some files or created copies of it in local folder, to review them to see what are different.

[mmsystems]

Deployed SO 2.3.61 today, configured fleet agent (3 in total) and no other services collecting data (no network monitoring, no other endpoint agents) and from nowhere got a Request failed with status 500 on SOC. I know the problem is with ElasticSearch and after hours of trying to troubleshoot looking at the log (/opt/so/log/elasticsearch/... and so on I can find the problem, has anyone seen this in the past?

I already added the transport.ping_request option to the salt elasticsearch config since it was suggested to a similar error in the past??

After a "soup" update I had this error code 500 almost every time I do a query. I modified "esheap" value and now it works fine.
To do it, edit the file:
/opt/so/saltstack/local/pillar/minions/securityonion_standalone.sls

And edit the esheap parameter in the blocks manager: and elasticsearch:. The default values in my installation are esheap: '2668' and I edited this number and set esheap: 4668 in the two locations.

(...)
manager:
mainip: 'X.X.X.X'
mainint: 'eth0'
proxy: ''
no_proxy: ''
esheap: '4668m'
esclustername: '{{ grains.host }}'
freq: 0
domainstats: 0
mtu: 1500
elastalert: 1
es_port: 9200
cur_close_days: 30
grafana: 1
osquery: 1
thehive: 1
playbook: 1

elasticsearch:
mainip: 'X.X.X.X'
mainint: 'eth0'
esheap: '4668m'
esclustername: '{{ grains.host }}'
node_type: ''
es_port: 9200
log_size_limit: 63
node_route_type: 'hot'
(...)

Then I rebooted the machine.

The value 4668 is purely trivial, I added 2000Mb to test it and it apparently works.