so-elasticsearch container crashes/exits

[netsecninja]

This crashing/exiting seems to happen every couple of days.

• Running "so-status" shows only so-elasticsearch with an ERROR status.
• Ran "so-elasticsearch-restart", get a "Connection attempt timed out. Unable to connect to ElasticSearch." message each time.
• Reviewing /opt/so/elasticsearch/securityonion.log does not show any error messages.
• Ran "docker ps", and elasticsearch is not listed.
• Ran "docker ps -a" shows the container as "Exited (1)".
• Ran "docker container start so-elasticsearch", container exits immediately
• The only workaround I've found is a reboot.

Please see the attached docs for more details.
so-elasticsearch-restart.txt
elasticsearch_securityonion.log

SO 2.3.50, Standalone (ISO)

[dougburks]

Is it possible that you're sending more data to Elasticsearch than it can handle with it's current heap size?
https://docs.securityonion.net/en/2.3/elasticsearch.html#heap-size

[netsecninja]

Just checked, and my heap is set to 4GB, which is the recommended size for my 16GB RAM. Was there something in the logs you saw that would indicate I need to increase the heap size more?

[dougburks]

The most likely reason for Elasticsearch exiting randomly like that is due to heap or memory issues. The heap is set solely based on your RAM with no knowledge of how much heap is actually necessary for the ingestion rate. So if this is a production deployment monitoring a busy network, it's quite possible that you may need more RAM in the box and/or more heap for Elasticsearch. Please note that 16GB RAM is the bare minimum for a production deployment:
https://docs.securityonion.net/en/2.3/hardware.html#standalone-deployments

[netsecninja]

I wiped and reinstalled SO, just to make sure it wasn't an installation/upgrade problem. Less than 16 hours up and it crashed again. This is only run on my home network, so 16GB might be the minimum, it should be enough to work on my small network. I don't have any more RAM to throw at SO.

Below you can see the moment of crash, nothing was spiking. My previous install had run since SO 2 came out, and it wasn't until recently it started crashing.