Introduction to AlertingÂ
- Alerting in Prometheus is separated into 2 parts
- Alerting rules in Prometheus server
- Alertmanager
Alerting - Alerting rules
- Rules live in Prometheus server config
- Best practice to separate the alerts form the Prometheus config
- Add an include to /etc/prometheus/prometheus.yml
- rule_files:
- "/etc/prometheus/alert.rules"
- rule_files:
- Add an include to /etc/prometheus/prometheus.yml
- Alert format
- Alert example:
- Alerting rule allow you to define the alert conditions
- Alerting rules sent the alerts being fired to an external service
- The format of these alerts is in the Prometheus expression language
- Example:
Alertmanager
- Alertmanager Configuration (/etc/alertmanager/alertmanager.yml)
- Prometheus Configuration (/etc/prometheus/promethes.yml)
- Concepts:
- Grouping: Groups similar alerts into 1 notification
- Inhibition: Silence other alerts if one specified alert is already fired
- Silences: A simple way to mute certain notifications
- High availability
- You can create a high available Alertmanager cluster using mesh config
- Do not load balance this service!
- Use a list of Alertmanager nodes in Prometheus config
- All alerts are sent to all known Alertmanager nodes
- No need to monitor the monitoring
- Guarantees the notification is at least send once
- Alert states:
- Inactive - No rule is met
- Pending - Rule is met but can be suppressed due to validations
- Firing - Alert is sent to the configured channel (mail, Slack, ...)
- Runs on port :9093
- Notifying multiple destinations
1 Comment
Anonymous