SALT issue - Pillar render error: Rendering SLS 'global' failed. Please see master log for details.

[szym3n]

I have a distributed environment runnign 2.3.2.
I was adding new sensors and another search node, however salt was having issues on the sensors. After troubleshooting on the master node by trying to delete the key, and then adding the sensors more times to the master, something corrupted.
I check the /opt/so/log/salt/master on the master node and this is what i see:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/pillar/init.py", line 885, in render_pstate
**defaults
File "/usr/lib/python3.6/site-packages/salt/template.py", line 106, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 74, in render
raise SaltRenderError(exc)
salt.exceptions.SaltRenderError: while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2
2020-11-11 22:24:28,157 [salt.pillar :1209][CRITICAL][1857] Pillar render error: Rendering SLS 'global' failed. Please see master log for details.
2020-11-11 22:24:28,863 [salt.pillar :889 ][CRITICAL][1856] Rendering SLS 'global' failed, render error:
while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 68, in render
data = yamlloader.load(yaml_data, Loader=get_yaml_loader(argline))
File "/usr/lib/python3.6/site-packages/salt/utils/yamlloader.py", line 178, in load
return yaml.load(stream, Loader=Loader)
File "/usr/lib64/python3.6/site-packages/yaml/init.py", line 72, in load
return loader.get_single_data()
File "/usr/lib64/python3.6/site-packages/yaml/constructor.py", line 35, in get_single_data
node = self.get_single_node()
File "_yaml.pyx", line 707, in _yaml.CParser.get_single_node (ext/_yaml.c:8040)
File "_yaml.pyx", line 725, in _yaml.CParser._compose_document (ext/_yaml.c:8295)
File "_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:8983)
File "_yaml.pyx", line 892, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:10544)
File "_yaml.pyx", line 905, in _yaml.CParser._parse_next_event (ext/_yaml.c:10722)
yaml.parser.ParserError: while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/pillar/init.py", line 885, in render_pstate
**defaults
File "/usr/lib/python3.6/site-packages/salt/template.py", line 106, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 74, in render
raise SaltRenderError(exc)
salt.exceptions.SaltRenderError: while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2
2020-11-11 22:24:28,867 [salt.pillar :1209][CRITICAL][1856] Pillar render error: Rendering SLS 'global' failed. Please see master log for details.
2020-11-11 22:39:29,969 [salt.pillar :889 ][CRITICAL][1856] Rendering SLS 'global' failed, render error:
while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 68, in render
data = yamlloader.load(yaml_data, Loader=get_yaml_loader(argline))
File "/usr/lib/python3.6/site-packages/salt/utils/yamlloader.py", line 178, in load
return yaml.load(stream, Loader=Loader)
File "/usr/lib64/python3.6/site-packages/yaml/init.py", line 72, in load
return loader.get_single_data()
File "/usr/lib64/python3.6/site-packages/yaml/constructor.py", line 35, in get_single_data
node = self.get_single_node()
File "_yaml.pyx", line 707, in _yaml.CParser.get_single_node (ext/_yaml.c:8040)
File "_yaml.pyx", line 725, in _yaml.CParser._compose_document (ext/_yaml.c:8295)
File "_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:8983)
File "_yaml.pyx", line 892, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:10544)
File "_yaml.pyx", line 905, in _yaml.CParser._parse_next_event (ext/_yaml.c:10722)
yaml.parser.ParserError: while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/pillar/init.py", line 885, in render_pstate
**defaults
File "/usr/lib/python3.6/site-packages/salt/template.py", line 106, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 74, in render
raise SaltRenderError(exc)
salt.exceptions.SaltRenderError: while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2
2020-11-11 22:39:29,974 [salt.pillar :1209][CRITICAL][1856] Pillar render error: Rendering SLS 'global' failed. Please see master log for details.
2020-11-11 22:42:34,216 [salt.pillar :889 ][CRITICAL][1855] Rendering SLS 'global' failed, render error:
while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 68, in render
data = yamlloader.load(yaml_data, Loader=get_yaml_loader(argline))
File "/usr/lib/python3.6/site-packages/salt/utils/yamlloader.py", line 178, in load
return yaml.load(stream, Loader=Loader)
File "/usr/lib64/python3.6/site-packages/yaml/init.py", line 72, in load
return loader.get_single_data()
File "/usr/lib64/python3.6/site-packages/yaml/constructor.py", line 35, in get_single_data
node = self.get_single_node()
File "_yaml.pyx", line 707, in _yaml.CParser.get_single_node (ext/_yaml.c:8040)
File "_yaml.pyx", line 725, in _yaml.CParser._compose_document (ext/_yaml.c:8295)
File "_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:8983)
File "_yaml.pyx", line 892, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:10544)
File "_yaml.pyx", line 905, in _yaml.CParser._parse_next_event (ext/_yaml.c:10722)
yaml.parser.ParserError: while parsing a block mapping
in "", line 1, column 1
did not find expected key
in "", line 119, column 2

During handling of the above exception, another exception occurred:

[nitsewg]

I am having a similar issue. Everything seemed to be working on a fresh standalone install Friday, but when I checked this morning, Suricata had crashed. When trying to restart the service, I got the SLS 'global' failed message. Tried rebooting the server, and almost all of the services were at 'wait_start'. Here is the message from the master log:

  File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 892, in render_pstate
    **defaults
  File "/usr/lib/python3.6/site-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1235, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2268, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2283, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/renderers/yaml.py", line 74, in render
    raise SaltRenderError(exc)
salt.exceptions.SaltRenderError: while constructing a mapping
  in "<unicode string>", line 123, column 3
found conflicting ID 'sids'
  in "<unicode string>", line 130, column 3
2021-06-07 13:03:40,497 [salt.pillar      :1214][CRITICAL][1879] Pillar render error: Rendering SLS 'global' failed. Please see master log for details.
2021-06-07 13:03:40,573 [salt.utils.job   :88  ][INFO    ][1878] Got return from hs-so_standalone for job 20210607130340572360

The only services showing 'OK' are Docker, so-aptcacherng, and so-dockerregistry. Security onion is fully up to date.

Let's see if we need to update Security Onion. Checking to see if there are hotfixes needed You are already running the latest version of Security Onion.

[nitsewg]

Update: Turns out I did something wrong while trying to add thresholding. Once I removed that from global.sls, suricata started fine, along with the rest of the services. I may need to read up more on how to do that.