Alerting with Prometheus in Kubernetes — A Beginners Guide

This blog will cover the Alert Manager setup and its integration with Prometheus deployed on a kubernetes cluster. This architecture will be using slack, as a notification channel.

Alertmanager is an open-source alerting system. The prime purpose is to manage alerts sent by client applications such as the Prometheus server. Moreover, it takes care of deduplicating, grouping, and routing them to the correct receiver integration such as Slack, Email, PagerDuty, OpsGenie, etc. It also manages silencing and inhibition of alerts.

This blog will be showing you the installation of Prometheus and Alertmanager via helm as initially, when I was trying to do so, I struggled a lot as there were only a few resources available on the internet. Hence this gave me motivation to write up this blog. Have a look-

The main steps involved in setting up the alerting and notifications are:

• Setup and configure the Alertmanager, You can configure the alertmanager with different notification channels.
• Create alerting rules in Prometheus (Like if any pod is not running)

Architecture

Below is the basic architecture of alertmanager:

Installation

If you install a stable/prometheus helm chart, alertmanager is also installed with it.

Setting up Slack Alerts

The first thing you need to do :

1. Be a part of the Slack workspace.
2. Go to Slack -> Administration -> Manage apps.

The Alertmanager uses the Incoming Webhooks feature of Slack, so we need to set that up first.

Manage -> Custom Integrations -> Incoming Webhooks

Here you need to add the channel for receiving the alerts. Once you confirm and add Incoming WebHooks integration, the webhook URL (which is your Slack api_url) is displayed.

Now copy the webhook url, this is your Slack api_url which will be used in the configmap of alertmanager.

Alertmanager Setup

Changes are to be made in the configmaps of alertmanager to configure it with slack. We need to edit the alertmanger.yml file.

Step 1: View the alertmanager configmap.

Step 2: Store the alertmanager configmap on local.
We can even directly edit the configmap file or the best approach is to save the cm(configmap) on local and then edit it using vs code or any text editor.

Step 3: Edit the alertmanager.yml file

• Various alertmanager receivers can be integrated such as email, slack, PagerDuty, OpsGenie, etc.
• Below is the setup for slack, edit the configmap accordingly

Step 4: Apply the configmap manifest file.

Step 5: Validate the configmap

Alerting Rules file in Prometheus

Alerting rules can be set in the configmap of prometheus-server.

Step 1: Store the cm on your local

Step 2: Add the rules in alerting_rules.yml file

Step 3: Apply the prometheus-server configmap manifest file

Relax and check your Slack Channel

Here you go, now you are all set to get notified via the firing alerts in slack, till then relax !!