Introduction to Alerting

Alerting in Prometheus is separated into 2 parts
- Alerting rules in Prometheus server
- Alertmanager

Alerting - Alerting rules

Rules live in Prometheus server config
Best practice to separate the alerts form the Prometheus config
- Add an include to /etc/prometheus/prometheus.yml
  - rule_files:
    - "/etc/prometheus/alert.rules"
Alert format
Alert example:
Alerting rule allow you to define the alert conditions
Alerting rules sent the alerts being fired to an external service
The format of these alerts is in the Prometheus expression language
Example:

Alertmanager Configuration (/etc/alertmanager/alertmanager.yml)
Prometheus Configuration (/etc/prometheus/promethes.yml)
Concepts:
- Grouping: Groups similar alerts into 1 notification
- Inhibition: Silence other alerts if one specified alert is already fired
- Silences: A simple way to mute certain notifications

High availability
- You can create a high available Alertmanager cluster using mesh config
- Do not load balance this service!
  - Use a list of Alertmanager nodes in Prometheus config
- All alerts are sent to all known Alertmanager nodes
- No need to monitor the monitoring
- Guarantees the notification is at least send once
Alert states:
- Inactive - No rule is met
- Pending - Rule is met but can be suppressed due to validations
- Firing - Alert is sent to the configured channel (mail, Slack, ...)
Runs on port :9093
Notifying multiple destinations