Skip to content

Commit

Permalink
Update routing-by-type.rst
Browse files Browse the repository at this point in the history
  • Loading branch information
aantn committed Jan 6, 2025
1 parent 62e8755 commit 719e7ba
Showing 1 changed file with 67 additions and 30 deletions.
97 changes: 67 additions & 30 deletions docs/notification-routing/routing-by-type.rst
Original file line number Diff line number Diff line change
@@ -1,29 +1,29 @@
Route By Type
Route By Alert Name
=============================

By default, all Robusta notifications are sent to all :ref:`sinks <Sinks Reference>`.
By default, Robusta notifications are sent to all :ref:`sinks <Sinks Reference>`.

It can be useful to send alerts of a *specific type* to a dedicated channel.
In this guide, you'll learn to route alerts by their name:

In this guide, we'll show how to route notifications for crashing pods to a specific Slack channel. All other notifications
will be sent to the usual channel.
* *KubePodCrashLooping* alerts will be sent to a *#crash-alerts* channel
* All other alerts will be sent to *#general-alerts*

Prerequisites
----------------

All least one existing :ref:`sink <Sinks Reference>` must be configured.
All least one existing :ref:`sink <Sinks Reference>` must be configured. Below, we'll assume it's a Slack sink.

Setting Up Routing
----------------------

This guide applies to all sink types. For simplicity's sake we'll assume you have an existing Slack sink:
Assume you have an existing Slack sink as follows:

.. code-block:: yaml
sinksConfig:
- slack_sink:
name: slack_app_sink
slack_channel: main-notifications
name: sink1
slack_channel: general-alerts
api_key: secret-key
The first step is to duplicate your sink. You need two unique sinks - one for each channel:
Expand All @@ -32,43 +32,80 @@ The first step is to duplicate your sink. You need two unique sinks - one for ea
sinksConfig:
- slack_sink:
name: main_sink
slack_channel: main-notifications
name: sink2
slack_channel: crash-alerts
api_key: secret-key
- slack_sink:
name: crashloopbackoff_slack_sink
slack_channel: crashpod-notifications
name: sink1
slack_channel: general-alerts
api_key: secret-key
The sinks are nearly identical - only the ``name`` and ``slack_channel`` parameters vary.
The sinks are nearly identical - only the ``name`` and ``slack_channel`` parameters vary:

Now lets add a :ref:`matcher <sink-matchers>` to each sink, so it receives a subset of notifications:
* The ``name`` field identifies this sink in Robusta and can be chosen arbitrarily - so long as it is unique between sinks
* The ``slack_channel`` field should match a channel in your Slack account

The next step is to update the configuration so that ``#crash-alerts`` receives a subset of alerts:

.. code-block:: yaml
sinksConfig:
- slack_sink:
name: main_sink
slack_channel: main-notifications
name: sink2
slack_channel: crash-alerts
api_key: secret-key
- scope:
exclude:
# don't send notifications related to image pull backoff
- identifier: [ImagePullBackoff]
include:
# only send notifications for the KubePodCrashLooping alert
- identifier: [KubePodCrashLooping]
- slack_sink:
name: sink1
slack_channel: general-alerts
api_key: secret-key
We added an :ref:`inclusion scope <sink-matchers>` for the ``#crash-alerts`` channel. To filter alerts by their name, use the ``identifier`` field which corresponds to the Prometheus alert name.

One final step: we must update the default sink to exclude *KubePodCrashLooping*. You can do this two ways:

- slack_sink:
name: crashloopbackoff_slack_sink
slack_channel: crash-notifications
**Option 1:** add an exclusion scope:

.. code-block:: yaml
sinksConfig:
- slack_sink:
name: sink2
slack_channel: crash-alerts
api_key: secret-key
- scope:
include:
# only send notifications related to crashing pods and CPU throttling
- identifier: [CrashLoopBackoff, CPUThrottlingHigh]
# only send notifications for the KubePodCrashLooping alert
- identifier: [KubePodCrashLooping]
- slack_sink:
name: sink1
slack_channel: general-alerts
api_key: secret-key
- scope:
exclude:
# don't send notifications for the KubePodCrashLooping alert
- identifier: [KubePodCrashLooping]
.. note::
**Option 2:** use ``stop: true`` to prevent alerts from propogating after a match:

For Prometheus alerts use the Alert name. Example, ``CPUThrottlingHigh``, ``KubeContainerWaiting``.
For other events, use the name as it appears on the Robusta timeline. Example, ``report_crash_loop`` and ``image_pull_backoff_reporter``
.. code-block:: yaml
sinksConfig:
- slack_sink:
name: sink2
slack_channel: crash-alerts
api_key: secret-key
# add the following line!
stop: true
- scope:
include:
- identifier: [KubePodCrashLooping]
- slack_sink:
name: sink1
slack_channel: general-alerts
api_key: secret-key
Now the ``crash-notifications`` channel will receive crashpod notifications and all other notifications will go to the
``main-notifications`` channel.
Whichever way you chose, now *KubePodCrashLooping* alerts are sent to ``#crash-alerts``. Other alerts go to ``#general-alerts``.

0 comments on commit 719e7ba

Please sign in to comment.