Merge branch 'master' into docs/update-old-ui-links

docs/index.rst

@@ -88,7 +88,7 @@ Better Prometheus Alerts (and more) for Kubernetes
         * :ref:`AI Investigation <AI Analysis>` - Kickstart your alert investigations with AI (optional)
         * :ref:`Alert Enrichment <Automatically Investigate a Prometheus Alert>` - see pods log and other data alongside your alerts
         * :ref:`Self-Healing <Remediate Prometheus Alerts>` - define auto-remediation rules for faster fixes
-        * :ref:`Advanced Routing <Notification Basics>` by team, namespace, k8s metadata and more
+        * :ref:`Advanced Routing <Defining Sinks>` by team, namespace, k8s metadata and more
         * :ref:`K8s Problem-Detection <Triggers Reference>` - alert on OOMKills or failing Jobs without PromQL
         * :ref:`Change Tracking <Track Kubernetes Changes>` - correlate alerts and Kubernetes rollouts
         * :ref:`Auto-Resolve <Jira>` - send alerts, resolve them when updated (e.g. in Jira)

docs/notification-routing/configuring-sinks.rst

@@ -1,31 +1,30 @@
 .. _sinks-overview:
 
-Notification Basics
+Defining Sinks
 ==========================
 
-Robusta can send notifications to various destinations, called sinks. For a list of all sinks, refer to :ref:`Sinks Reference`.
+Robusta can send notifications to various destinations, called sinks.
 
-Defining Sinks
-^^^^^^^^^^^^^^^^^^
+For a list of all sinks, refer to :ref:`Sinks Reference`.
+
+A Simple Sink Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Sinks are defined in Robusta's Helm chart, using the ``sinksConfig`` value.
 
-For example, lets add a :ref:`Microsoft Teams <MS Teams sink>`:
+For example, lets add a :ref:`Microsoft Teams Sink <MS Teams sink>`:
 
 .. code-block:: yaml
 
     sinksConfig:
     - ms_teams_sink:                  # sink type
-        name: my_teams_sink           # arbitrary name
-        webhook_url: <placeholder>    # a sink-specific parameter
-        stop: false                   # optional (see `Routing Alerts to only one Sink`)
-        scope: {}                     # optional routing rules
-        default: true                 # optional (see below)
+        name: my_teams_sink           # name that uniquely identifies this sink in Robusta
+        webhook_url: <placeholder>    # the webhook URL for MSTeams - each sink has different parameters like this
 
-In addition to the above attributes, some sinks have additional attributes that can be found in their documentation.
+For all options, refer to :ref:`All Sink Options`.
 
 Defining Multiple Sinks
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-You can define multiple sinks and by default, notifications will be sent to all of them.
+You can define multiple sinks. By default, notifications will be sent to all of them.
 
 In the following example, we define a :ref:`Slack sink <Slack>` and a :ref:`MS Teams sink <MS Teams>` without any routing rules, so both sinks receive all notifications:
 
@@ -42,7 +41,57 @@ In the following example, we define a :ref:`Slack sink <Slack>` and a :ref:`MS T
 
 To selectively send notifications to different sinks, refer to :ref:`routing (scopes) <sink-scope-matching>`.
 
-See Also
+
+All Sink Options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here is an example showing common sink options:
+
+.. code-block:: yaml
+
+    sinksConfig:
+    - slack_sink:                     # sink type
+        name: my_sink_name            # name that uniquely identifies this sink in Robusta
+        scope: {}                     # optional - filter notifications sent to this sink
+        activity: {}                  # optional - enable/disable sink according to time of day/week
+        stop: false                   # optional - stop notifications from continuing to subsequent sinks
+        grouping: {}                  # optional - use grouping to reduce the number of notifications (i.e. group into slack threads)
+        default: true                 # optional - disable this sink by default
+
+        # sink-specific parameters - e.g. for Slack, some options are shown below
+        # api_key: xoxb-112...
+        # slack_channel: general-alerts
+
+Description of each option:
+
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| Parameter Name   | Description                                             | Default                                                  | Docs                                          |
++==================+=========================================================+==========================================================+===============================================+
+| name             | A unique name for this sink in Robusta                  | -                                                        | -                                             |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| scope            | Filters the notifications sent to this sink             | *undefined* - all notifications are sent (unless already |                                               |
+|                  |                                                         | sent a previou sink that set `stop: true`)               | :ref:`Routing (scopes) <sink-scope-matching>` |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| activity         | Controls the hours this sink is active                  | *undefined* - active all hours and all days of the week  | :ref:`Route by Time`                          |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| stop             | Should notifications continue to subsequent sinks?      | false - notification sent to this sink will continue to  | :ref:`Stop Further Notifications`             |
+|                  |                                                         | subsequent sinks                                         |                                               |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| grouping         | Currently only impacts the Slack sink, where it controls| *undefined* (disabled)                                   | :ref:`Grouping <notification-grouping>`       | 
+|                  | the creation of threads and the grouping of many        |                                                          |                                               |
+|                  | notifications into one message                          |                                                          |                                               |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| default          | Is this sink enabled by default? When false, this sink  | true - this sink is enabled by default                   | :ref:`Alternative Routing Methods`            |
+|                  | only accepts notifications from customPlaybooks which   |                                                          |                                               |
+|                  | explicitly named this sink (if scope is set, it will    |                                                          |                                               |
+|                  | still filter those notifications)                       |                                                          |                                               |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+| *sink specific*  | Parameters specific to the sink type, like api_key for  | -                                                        | :ref:`sink-specific docs <Sinks Reference>`   |
+| *parameters*     | Slack and webhook_url for MSTeams                       |                                                          |                                               |
++------------------+---------------------------------------------------------+----------------------------------------------------------+-----------------------------------------------+
+
+Learn More
 ^^^^^^^^^^^^
 
-🔔 :ref:`All Sinks <Sinks Reference>`
+* 🔔 :ref:`All Sinks <Sinks Reference>`
+* ↳ :ref:`Routing (scopes) <sink-scope-matching>`

docs/notification-routing/implementing-monitoring-shifts.rst

@@ -1,4 +1,4 @@
-Monitoring Shifts
+Route By Time of Day
 ####################
 
 In this example we'll route alerts to different support teams based on the time of day:
@@ -33,4 +33,4 @@ In this example we'll route alerts to different support teams based on the time
             - start: 12:00  # 12 PM
                 end: 23:59  # 11:59 PM
 
-For more information on time-based routing, :ref:`view <Route by Time>`
+For more information on time-based routing, view :ref:`Route by Time`.

docs/notification-routing/notification-routing-examples.rst

@@ -7,8 +7,8 @@ Routing Cookbook
 
    routing-by-namespace
    routing-by-type
-   routing-to-multiple-slack-channels
    implementing-monitoring-shifts
+   routing-to-multiple-slack-channels
    routing-exclusion
 
 In this section you'll find example configurations for common routing patterns.

docs/notification-routing/routing-by-namespace.rst

@@ -11,7 +11,7 @@ Prerequisites
 Setting Up Routing
 ----------------------
 
-Assume you have an existing sink, in this case Slack:
+Assume you have an existing sink:
 
 .. code-block:: yaml
 
@@ -21,7 +21,9 @@ Assume you have an existing sink, in this case Slack:
         slack_channel: app-notifications
         api_key: secret-key
 
-By default, the sink will receive notifications for all namespaces. Let's duplicate the sink and change only the Slack channel:
+By default, the sink will receive notifications for all namespaces.
+
+Let's create a 2nd copy of the sink and change the Slack channel:
 
 .. code-block:: yaml
 
@@ -35,7 +37,9 @@ By default, the sink will receive notifications for all namespaces. Let's duplic
         slack_channel: system-notifications
         api_key: secret-key
 
-We now have two sinks, both receiving all notifications. Restrict the notifications for each sink by adding :ref:`scopes <sink-matchers>`:
+Now we have two sinks, both receiving all notifications.
+
+The final step is to restrict notifications for each sink by adding :ref:`scopes <sink-scope-matching>`:
 
 .. code-block:: yaml
 
@@ -58,9 +62,13 @@ We now have two sinks, both receiving all notifications. Restrict the notificati
 
 Alerts will be now routed according to Kubernetes namespace.
 
-You can apply this method with as many sinks as you like. If the number of sinks is large, consider setting the channel dynamically. See instructions for :ref:`Slack <Dynamic Alert Routing>` or :ref:`MS Teams <Dynamically Route MS Teams Alerts>`.
+Routing for many namespaces
+-----------------------------------
+Using the above method, you can create one sink for each namespace.
+
+If you have a large number of namespaces, there is an alternative method you can consider: you can define a single sink and set the channel dynamically according to alert metadata. See instructions for :ref:`Slack <Dynamic Alert Routing>` or :ref:`MS Teams <Dynamically Route MS Teams Alerts>`.
 
-Troubleshooting Issues
-------------------------
+Fallback for alerts without a namespace
+--------------------------------------------------
 
-For this guide to work, alerts must be tagged with ``namespace`` metadata. It is recommended that you setup a :ref:`Fallback Sink <Stop Further Notifications>` to catch alerts that don't have a namespace.
+The above example assumes that all alerts have ``namespace`` metadata. It is recommended that you setup a :ref:`Fallback Sink <Stop Further Notifications>` to catch alerts that don't have a namespace.

docs/notification-routing/routing-by-type.rst

@@ -1,29 +1,29 @@
-Route By Type
+Route By Alert Name
 =============================
 
-By default, all Robusta notifications are sent to all :ref:`sinks <Sinks Reference>`.
+By default, Robusta notifications are sent to all :ref:`sinks <Sinks Reference>`.
 
-It can be useful to send alerts of a *specific type* to a dedicated channel.
+In this guide, you'll learn to route alerts by their name:
 
-In this guide, we'll show how to route notifications for crashing pods to a specific Slack channel. All other notifications
-will be sent to the usual channel.
+* *KubePodCrashLooping* alerts will be sent to a *#crash-alerts* channel
+* All other alerts will be sent to *#general-alerts*
 
 Prerequisites
 ----------------
 
-All least one existing :ref:`sink <Sinks Reference>` must be configured.
+All least one existing :ref:`sink <Sinks Reference>` must be configured. Below, we'll assume it's a Slack sink.
 
 Setting Up Routing
 ----------------------
 
-This guide applies to all sink types. For simplicity's sake we'll assume you have an existing Slack sink:
+Assume you have an existing Slack sink as follows:
 
 .. code-block:: yaml
 
     sinksConfig:
     - slack_sink:
-        name: slack_app_sink
-        slack_channel: main-notifications
+        name: sink1
+        slack_channel: general-alerts
         api_key: secret-key
 
 The first step is to duplicate your sink. You need two unique sinks - one for each channel:
@@ -32,43 +32,80 @@ The first step is to duplicate your sink. You need two unique sinks - one for ea
 
     sinksConfig:
     - slack_sink:
-        name: main_sink
-        slack_channel: main-notifications
+        name: sink2
+        slack_channel: crash-alerts
         api_key: secret-key
     - slack_sink:
-        name: crashloopbackoff_slack_sink
-        slack_channel: crashpod-notifications
+        name: sink1
+        slack_channel: general-alerts
         api_key: secret-key
 
-The sinks are nearly identical - only the ``name`` and ``slack_channel`` parameters vary.
+The sinks are nearly identical - only the ``name`` and ``slack_channel`` parameters vary:
 
-Now lets add a :ref:`matcher <sink-matchers>` to each sink, so it receives a subset of notifications:
+* The ``name`` field identifies this sink in Robusta and can be chosen arbitrarily - so long as it is unique between sinks
+* The ``slack_channel`` field should match a channel in your Slack account
+
+The next step is to update the configuration so that ``#crash-alerts`` receives a subset of alerts:
 
 .. code-block:: yaml
 
     sinksConfig:
     - slack_sink:
-        name: main_sink
-        slack_channel: main-notifications
+        name: sink2
+        slack_channel: crash-alerts
         api_key: secret-key
         - scope:
-            exclude:
-            # don't send notifications related to image pull backoff
-              - identifier: [ImagePullBackoff]
+            include:
+            # only send notifications for the KubePodCrashLooping alert
+            - identifier: [KubePodCrashLooping]
+    - slack_sink:
+        name: sink1
+        slack_channel: general-alerts
+        api_key: secret-key
+
+We added an :ref:`inclusion scope <sink-matchers>` for the ``#crash-alerts`` channel. To filter alerts by their name, use the ``identifier`` field which corresponds to the Prometheus alert name.
+
+One final step: we must update the default sink to exclude *KubePodCrashLooping*. You can do this two ways:
 
-   - slack_sink:
-        name: crashloopbackoff_slack_sink
-        slack_channel: crash-notifications
+**Option 1:** add an exclusion scope:
+
+.. code-block:: yaml
+
+    sinksConfig:
+    - slack_sink:
+        name: sink2
+        slack_channel: crash-alerts
         api_key: secret-key
         - scope:
             include:
-            # only send notifications related to crashing pods and CPU throttling
-              - identifier: [CrashLoopBackoff, CPUThrottlingHigh]
+            # only send notifications for the KubePodCrashLooping alert
+            - identifier: [KubePodCrashLooping]
+    - slack_sink:
+        name: sink1
+        slack_channel: general-alerts
+        api_key: secret-key
+        - scope:
+            exclude:
+            # don't send notifications for the KubePodCrashLooping alert
+            - identifier: [KubePodCrashLooping]
 
-.. note::
+**Option 2:** use ``stop: true`` to prevent alerts from propogating after a match:
 
-    For Prometheus alerts use the Alert name. Example, ``CPUThrottlingHigh``, ``KubeContainerWaiting``.
-    For other events, use the name as it appears on the Robusta timeline. Example, ``report_crash_loop`` and ``image_pull_backoff_reporter``
+.. code-block:: yaml
+
+    sinksConfig:
+    - slack_sink:
+        name: sink2
+        slack_channel: crash-alerts
+        api_key: secret-key
+        # add the following line!
+        stop: true
+        - scope:
+            include:
+            - identifier: [KubePodCrashLooping]
+    - slack_sink:
+        name: sink1
+        slack_channel: general-alerts
+        api_key: secret-key
 
-Now the ``crash-notifications`` channel will receive crashpod notifications and all other notifications will go to the
-``main-notifications`` channel.
+Whichever way you chose, now *KubePodCrashLooping* alerts are sent to ``#crash-alerts``. Other alerts go to ``#general-alerts``.

docs/notification-routing/routing-to-multiple-slack-channels.rst

@@ -22,3 +22,4 @@ In this example, we'll route alerts to different Slack channels depending on an
             include:
             - namespace: [backend]
 
+For more details, refer to :ref:`scopes <sink-scope-matching>`.

docs/setup-robusta/installation-faq.rst

@@ -39,7 +39,7 @@ For example, to install in the ``robusta`` namespace:
 
     helm install robusta robusta/robusta -f ./generated_values.yaml -n robusta --create-namespace --set clusterName=<YOUR_CLUSTER_NAME>
 
-Verify success by checking that two Robusta pods are running:
+Verify success by checking that Robusta pods are running:
 
 .. code-block:: bash
    :name: cb-get-pods-robusta-logs-custom

docs/setup-robusta/installation/_helm_install_with_prometheus.inc.rst

@@ -91,7 +91,7 @@ On some clusters this can take a while, so don't panic if it appears stuck:
 Verifying Installation
 ------------------------------
 
-Confirm that two Robusta pods are running with no errors in the logs:
+Confirm that Robusta pods are running with no errors in the logs:
 
 .. code-block:: bash
     :name: cb-get-pods-robusta-logs