Versions Compared

Old Version 2

changes.mady.by.user Sansae

Saved on

compared with

New Version Current

changes.mady.by.user Sansae

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

Table of Contents

Introduction to Alerting 

Info
  • Alerting in Prometheus is separated into 2 parts
    • Alerting rules in Prometheus server
    • Alertmanager

Alerting - Alerting rules

Info
  • Rules live in Prometheus server config
  • Best practice to separate the alerts form the Prometheus config
    • Add an include to /etc/prometheus/prometheus.yml
      • rule_files:
        - "/etc/prometheus/alert.rules"
  • Alert format
  • Alert example:
  • Alerting rule allow you to define the alert conditions
  • Alerting rules sent the alerts being fired to an external service
  • The format of these alerts is in the Prometheus expression language
  • Example:

Alertmanager

Info
  • Alertmanager Configuration (/etc/alertmanager/alertmanager.yml)
  • Prometheus Configuration (/etc/prometheus/promethes.yml) 
  • Concepts:
    • Grouping: Groups similar alerts into 1 notification
    • Inhibition: Silence other alerts if one specified alert is already fired
    • Silences: A simple way to mute certain notifications
  • High availability
    • You can create a high available Alertmanager cluster using mesh config
    • Do not load balance this service!
      • Use a  list of Alertmanager nodes in Prometheus config
    • All alerts are sent to all known Alertmanager nodes
    • No need to monitor the monitoring
    • Guarantees the notification is at least send once
  • Alert states:
    • Inactive - No rule is met
    • Pending - Rule is met but can be suppressed due to validations
    • Firing - Alert is sent to the configured channel (mail, Slack, ...)
  • Runs on port :9093
  • Notifying multiple destinations