From 3523b039efd2775355d48e0d7b4499bd0d8a972b Mon Sep 17 00:00:00 2001 From: Natan Yellin Date: Tue, 7 Jan 2025 11:24:16 +0200 Subject: [PATCH 1/2] Update routing-by-type.rst (#1679) --- docs/index.rst | 2 +- .../configuring-sinks.rst | 45 ++++++--- .../implementing-monitoring-shifts.rst | 4 +- .../notification-routing-examples.rst | 2 +- .../routing-by-namespace.rst | 22 +++-- docs/notification-routing/routing-by-type.rst | 97 +++++++++++++------ .../routing-to-multiple-slack-channels.rst | 1 + 7 files changed, 118 insertions(+), 55 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 671077004..f19c95ec1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -88,7 +88,7 @@ Better Prometheus Alerts (and more) for Kubernetes * :ref:`AI Investigation ` - Kickstart your alert investigations with AI (optional) * :ref:`Alert Enrichment ` - see pods log and other data alongside your alerts * :ref:`Self-Healing ` - define auto-remediation rules for faster fixes - * :ref:`Advanced Routing ` by team, namespace, k8s metadata and more + * :ref:`Advanced Routing ` by team, namespace, k8s metadata and more * :ref:`K8s Problem-Detection ` - alert on OOMKills or failing Jobs without PromQL * :ref:`Change Tracking ` - correlate alerts and Kubernetes rollouts * :ref:`Auto-Resolve ` - send alerts, resolve them when updated (e.g. in Jira) diff --git a/docs/notification-routing/configuring-sinks.rst b/docs/notification-routing/configuring-sinks.rst index bf5f14dbf..bc8218671 100644 --- a/docs/notification-routing/configuring-sinks.rst +++ b/docs/notification-routing/configuring-sinks.rst @@ -1,31 +1,47 @@ .. _sinks-overview: -Notification Basics +Defining Sinks ========================== -Robusta can send notifications to various destinations, called sinks. For a list of all sinks, refer to :ref:`Sinks Reference`. +Robusta can send notifications to various destinations, called sinks. -Defining Sinks -^^^^^^^^^^^^^^^^^^ +For a list of all sinks, refer to :ref:`Sinks Reference`. + +A Simple Sink Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sinks are defined in Robusta's Helm chart, using the ``sinksConfig`` value. -For example, lets add a :ref:`Microsoft Teams `: +For example, lets add a :ref:`Microsoft Teams Sink `: + +.. code-block:: yaml + + sinksConfig: + - ms_teams_sink: # sink type + name: my_teams_sink # name that uniquely identifies this sink in Robusta + webhook_url: # the webhook URL for MSTeams - each sink has different parameters like this + +More Sink Options +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Here are some more options you can set on sinks: .. code-block:: yaml sinksConfig: - ms_teams_sink: # sink type - name: my_teams_sink # arbitrary name - webhook_url: # a sink-specific parameter - stop: false # optional (see `Routing Alerts to only one Sink`) - scope: {} # optional routing rules - default: true # optional (see below) + name: my_teams_sink # name that uniquely identifies this sink in Robusta + stop: false # optional - covered in the next tutorial + scope: {} # optional - covered in the next tutorial + default: true # optional - covered in the next tutorial + + # sink-specific parameters + webhook_url: -In addition to the above attributes, some sinks have additional attributes that can be found in their documentation. +The ``stop``, ``scope``, and ``default`` fields are used for :ref:`routing (scopes) `. Defining Multiple Sinks ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -You can define multiple sinks and by default, notifications will be sent to all of them. +You can define multiple sinks. By default, notifications will be sent to all of them. In the following example, we define a :ref:`Slack sink ` and a :ref:`MS Teams sink ` without any routing rules, so both sinks receive all notifications: @@ -42,7 +58,8 @@ In the following example, we define a :ref:`Slack sink ` and a :ref:`MS T To selectively send notifications to different sinks, refer to :ref:`routing (scopes) `. -See Also +Learn More ^^^^^^^^^^^^ -🔔 :ref:`All Sinks ` +* 🔔 :ref:`All Sinks ` +* ↳ :ref:`Routing (scopes) ` \ No newline at end of file diff --git a/docs/notification-routing/implementing-monitoring-shifts.rst b/docs/notification-routing/implementing-monitoring-shifts.rst index 53dca9b00..aca479dac 100644 --- a/docs/notification-routing/implementing-monitoring-shifts.rst +++ b/docs/notification-routing/implementing-monitoring-shifts.rst @@ -1,4 +1,4 @@ -Monitoring Shifts +Route By Time of Day #################### In this example we'll route alerts to different support teams based on the time of day: @@ -33,4 +33,4 @@ In this example we'll route alerts to different support teams based on the time - start: 12:00 # 12 PM end: 23:59 # 11:59 PM -For more information on time-based routing, :ref:`view ` +For more information on time-based routing, view :ref:`Route by Time`. diff --git a/docs/notification-routing/notification-routing-examples.rst b/docs/notification-routing/notification-routing-examples.rst index cb1a82407..849042043 100644 --- a/docs/notification-routing/notification-routing-examples.rst +++ b/docs/notification-routing/notification-routing-examples.rst @@ -7,8 +7,8 @@ Routing Cookbook routing-by-namespace routing-by-type - routing-to-multiple-slack-channels implementing-monitoring-shifts + routing-to-multiple-slack-channels routing-exclusion In this section you'll find example configurations for common routing patterns. diff --git a/docs/notification-routing/routing-by-namespace.rst b/docs/notification-routing/routing-by-namespace.rst index a56745e1e..01617ee36 100644 --- a/docs/notification-routing/routing-by-namespace.rst +++ b/docs/notification-routing/routing-by-namespace.rst @@ -11,7 +11,7 @@ Prerequisites Setting Up Routing ---------------------- -Assume you have an existing sink, in this case Slack: +Assume you have an existing sink: .. code-block:: yaml @@ -21,7 +21,9 @@ Assume you have an existing sink, in this case Slack: slack_channel: app-notifications api_key: secret-key -By default, the sink will receive notifications for all namespaces. Let's duplicate the sink and change only the Slack channel: +By default, the sink will receive notifications for all namespaces. + +Let's create a 2nd copy of the sink and change the Slack channel: .. code-block:: yaml @@ -35,7 +37,9 @@ By default, the sink will receive notifications for all namespaces. Let's duplic slack_channel: system-notifications api_key: secret-key -We now have two sinks, both receiving all notifications. Restrict the notifications for each sink by adding :ref:`scopes `: +Now we have two sinks, both receiving all notifications. + +The final step is to restrict notifications for each sink by adding :ref:`scopes `: .. code-block:: yaml @@ -58,9 +62,13 @@ We now have two sinks, both receiving all notifications. Restrict the notificati Alerts will be now routed according to Kubernetes namespace. -You can apply this method with as many sinks as you like. If the number of sinks is large, consider setting the channel dynamically. See instructions for :ref:`Slack ` or :ref:`MS Teams `. +Routing for many namespaces +----------------------------------- +Using the above method, you can create one sink for each namespace. + +If you have a large number of namespaces, there is an alternative method you can consider: you can define a single sink and set the channel dynamically according to alert metadata. See instructions for :ref:`Slack ` or :ref:`MS Teams `. -Troubleshooting Issues ------------------------- +Fallback for alerts without a namespace +-------------------------------------------------- -For this guide to work, alerts must be tagged with ``namespace`` metadata. It is recommended that you setup a :ref:`Fallback Sink ` to catch alerts that don't have a namespace. \ No newline at end of file +The above example assumes that all alerts have ``namespace`` metadata. It is recommended that you setup a :ref:`Fallback Sink ` to catch alerts that don't have a namespace. \ No newline at end of file diff --git a/docs/notification-routing/routing-by-type.rst b/docs/notification-routing/routing-by-type.rst index f1941e005..5914091f0 100644 --- a/docs/notification-routing/routing-by-type.rst +++ b/docs/notification-routing/routing-by-type.rst @@ -1,29 +1,29 @@ -Route By Type +Route By Alert Name ============================= -By default, all Robusta notifications are sent to all :ref:`sinks `. +By default, Robusta notifications are sent to all :ref:`sinks `. -It can be useful to send alerts of a *specific type* to a dedicated channel. +In this guide, you'll learn to route alerts by their name: -In this guide, we'll show how to route notifications for crashing pods to a specific Slack channel. All other notifications -will be sent to the usual channel. +* *KubePodCrashLooping* alerts will be sent to a *#crash-alerts* channel +* All other alerts will be sent to *#general-alerts* Prerequisites ---------------- -All least one existing :ref:`sink ` must be configured. +All least one existing :ref:`sink ` must be configured. Below, we'll assume it's a Slack sink. Setting Up Routing ---------------------- -This guide applies to all sink types. For simplicity's sake we'll assume you have an existing Slack sink: +Assume you have an existing Slack sink as follows: .. code-block:: yaml sinksConfig: - slack_sink: - name: slack_app_sink - slack_channel: main-notifications + name: sink1 + slack_channel: general-alerts api_key: secret-key The first step is to duplicate your sink. You need two unique sinks - one for each channel: @@ -32,43 +32,80 @@ The first step is to duplicate your sink. You need two unique sinks - one for ea sinksConfig: - slack_sink: - name: main_sink - slack_channel: main-notifications + name: sink2 + slack_channel: crash-alerts api_key: secret-key - slack_sink: - name: crashloopbackoff_slack_sink - slack_channel: crashpod-notifications + name: sink1 + slack_channel: general-alerts api_key: secret-key -The sinks are nearly identical - only the ``name`` and ``slack_channel`` parameters vary. +The sinks are nearly identical - only the ``name`` and ``slack_channel`` parameters vary: -Now lets add a :ref:`matcher ` to each sink, so it receives a subset of notifications: +* The ``name`` field identifies this sink in Robusta and can be chosen arbitrarily - so long as it is unique between sinks +* The ``slack_channel`` field should match a channel in your Slack account + +The next step is to update the configuration so that ``#crash-alerts`` receives a subset of alerts: .. code-block:: yaml sinksConfig: - slack_sink: - name: main_sink - slack_channel: main-notifications + name: sink2 + slack_channel: crash-alerts api_key: secret-key - scope: - exclude: - # don't send notifications related to image pull backoff - - identifier: [ImagePullBackoff] + include: + # only send notifications for the KubePodCrashLooping alert + - identifier: [KubePodCrashLooping] + - slack_sink: + name: sink1 + slack_channel: general-alerts + api_key: secret-key + +We added an :ref:`inclusion scope ` for the ``#crash-alerts`` channel. To filter alerts by their name, use the ``identifier`` field which corresponds to the Prometheus alert name. + +One final step: we must update the default sink to exclude *KubePodCrashLooping*. You can do this two ways: - - slack_sink: - name: crashloopbackoff_slack_sink - slack_channel: crash-notifications +**Option 1:** add an exclusion scope: + +.. code-block:: yaml + + sinksConfig: + - slack_sink: + name: sink2 + slack_channel: crash-alerts api_key: secret-key - scope: include: - # only send notifications related to crashing pods and CPU throttling - - identifier: [CrashLoopBackoff, CPUThrottlingHigh] + # only send notifications for the KubePodCrashLooping alert + - identifier: [KubePodCrashLooping] + - slack_sink: + name: sink1 + slack_channel: general-alerts + api_key: secret-key + - scope: + exclude: + # don't send notifications for the KubePodCrashLooping alert + - identifier: [KubePodCrashLooping] -.. note:: +**Option 2:** use ``stop: true`` to prevent alerts from propogating after a match: - For Prometheus alerts use the Alert name. Example, ``CPUThrottlingHigh``, ``KubeContainerWaiting``. - For other events, use the name as it appears on the Robusta timeline. Example, ``report_crash_loop`` and ``image_pull_backoff_reporter`` +.. code-block:: yaml + + sinksConfig: + - slack_sink: + name: sink2 + slack_channel: crash-alerts + api_key: secret-key + # add the following line! + stop: true + - scope: + include: + - identifier: [KubePodCrashLooping] + - slack_sink: + name: sink1 + slack_channel: general-alerts + api_key: secret-key -Now the ``crash-notifications`` channel will receive crashpod notifications and all other notifications will go to the -``main-notifications`` channel. +Whichever way you chose, now *KubePodCrashLooping* alerts are sent to ``#crash-alerts``. Other alerts go to ``#general-alerts``. diff --git a/docs/notification-routing/routing-to-multiple-slack-channels.rst b/docs/notification-routing/routing-to-multiple-slack-channels.rst index c9431a701..691d09add 100644 --- a/docs/notification-routing/routing-to-multiple-slack-channels.rst +++ b/docs/notification-routing/routing-to-multiple-slack-channels.rst @@ -22,3 +22,4 @@ In this example, we'll route alerts to different Slack channels depending on an include: - namespace: [backend] +For more details, refer to :ref:`scopes `. \ No newline at end of file From e363f6aecdd38ba2c5bdcbd1f8ff78d4ef1ab6b4 Mon Sep 17 00:00:00 2001 From: Natan Yellin Date: Tue, 7 Jan 2025 19:17:12 +0200 Subject: [PATCH 2/2] more docs fixes (#1682) --- .../configuring-sinks.rst | 68 ++++++++++++++----- docs/setup-robusta/installation-faq.rst | 2 +- .../_helm_install_with_prometheus.inc.rst | 2 +- 3 files changed, 52 insertions(+), 20 deletions(-) diff --git a/docs/notification-routing/configuring-sinks.rst b/docs/notification-routing/configuring-sinks.rst index bc8218671..9dac6719f 100644 --- a/docs/notification-routing/configuring-sinks.rst +++ b/docs/notification-routing/configuring-sinks.rst @@ -20,24 +20,7 @@ For example, lets add a :ref:`Microsoft Teams Sink `: name: my_teams_sink # name that uniquely identifies this sink in Robusta webhook_url: # the webhook URL for MSTeams - each sink has different parameters like this -More Sink Options -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Here are some more options you can set on sinks: - -.. code-block:: yaml - - sinksConfig: - - ms_teams_sink: # sink type - name: my_teams_sink # name that uniquely identifies this sink in Robusta - stop: false # optional - covered in the next tutorial - scope: {} # optional - covered in the next tutorial - default: true # optional - covered in the next tutorial - - # sink-specific parameters - webhook_url: - -The ``stop``, ``scope``, and ``default`` fields are used for :ref:`routing (scopes) `. +For all options, refer to :ref:`All Sink Options`. Defining Multiple Sinks ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -58,6 +41,55 @@ In the following example, we define a :ref:`Slack sink ` and a :ref:`MS T To selectively send notifications to different sinks, refer to :ref:`routing (scopes) `. + +All Sink Options +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Here is an example showing common sink options: + +.. code-block:: yaml + + sinksConfig: + - slack_sink: # sink type + name: my_sink_name # name that uniquely identifies this sink in Robusta + scope: {} # optional - filter notifications sent to this sink + activity: {} # optional - enable/disable sink according to time of day/week + stop: false # optional - stop notifications from continuing to subsequent sinks + grouping: {} # optional - use grouping to reduce the number of notifications (i.e. group into slack threads) + default: true # optional - disable this sink by default + + # sink-specific parameters - e.g. for Slack, some options are shown below + # api_key: xoxb-112... + # slack_channel: general-alerts + +Description of each option: + ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| Parameter Name | Description | Default | Docs | ++==================+=========================================================+==========================================================+===============================================+ +| name | A unique name for this sink in Robusta | - | - | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| scope | Filters the notifications sent to this sink | *undefined* - all notifications are sent (unless already | | +| | | sent a previou sink that set `stop: true`) | :ref:`Routing (scopes) ` | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| activity | Controls the hours this sink is active | *undefined* - active all hours and all days of the week | :ref:`Route by Time` | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| stop | Should notifications continue to subsequent sinks? | false - notification sent to this sink will continue to | :ref:`Stop Further Notifications` | +| | | subsequent sinks | | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| grouping | Currently only impacts the Slack sink, where it controls| *undefined* (disabled) | :ref:`Grouping ` | +| | the creation of threads and the grouping of many | | | +| | notifications into one message | | | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| default | Is this sink enabled by default? When false, this sink | true - this sink is enabled by default | :ref:`Alternative Routing Methods` | +| | only accepts notifications from customPlaybooks which | | | +| | explicitly named this sink (if scope is set, it will | | | +| | still filter those notifications) | | | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ +| *sink specific* | Parameters specific to the sink type, like api_key for | - | :ref:`sink-specific docs ` | +| *parameters* | Slack and webhook_url for MSTeams | | | ++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+ + Learn More ^^^^^^^^^^^^ diff --git a/docs/setup-robusta/installation-faq.rst b/docs/setup-robusta/installation-faq.rst index 825155468..9d5f4b72f 100644 --- a/docs/setup-robusta/installation-faq.rst +++ b/docs/setup-robusta/installation-faq.rst @@ -39,7 +39,7 @@ For example, to install in the ``robusta`` namespace: helm install robusta robusta/robusta -f ./generated_values.yaml -n robusta --create-namespace --set clusterName= -Verify success by checking that two Robusta pods are running: +Verify success by checking that Robusta pods are running: .. code-block:: bash :name: cb-get-pods-robusta-logs-custom diff --git a/docs/setup-robusta/installation/_helm_install_with_prometheus.inc.rst b/docs/setup-robusta/installation/_helm_install_with_prometheus.inc.rst index 2150f9e8e..ab0205081 100644 --- a/docs/setup-robusta/installation/_helm_install_with_prometheus.inc.rst +++ b/docs/setup-robusta/installation/_helm_install_with_prometheus.inc.rst @@ -91,7 +91,7 @@ On some clusters this can take a while, so don't panic if it appears stuck: Verifying Installation ------------------------------ -Confirm that two Robusta pods are running with no errors in the logs: +Confirm that Robusta pods are running with no errors in the logs: .. code-block:: bash :name: cb-get-pods-robusta-logs