Versions Compared

Old Version 3

changes.mady.by.user Sansae

Saved on

compared with

New Version Current

changes.mady.by.user Sansae

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

Table of Contents

Setting

...

Info
  • Install Alertmanager
  • Create config fot the Alertmanager
    • Mail
    • Slack
    • ...
  • Alter prometheus config
  • Setup an alert
    • See the notification coming in when an alert is fired

Setting - Slack

Info

Run Alertmanager on Docker

Info

  • https://github.com/prometheus/alertmanager

    Code Block
    sansae@sansaeubuntu:~$ docker run --name alertmanager -d -p 9093:9093 quay.io/prometheus/alertmanager
Unable to find image 'quay.io/prometheus/alertmanager:latest' locally
latest: Pulling from prometheus/alertmanager
0f8c40e1270f: Pull complete
626a2a3fee8c: Pull complete
74ad1ee664e6: Pull complete
addd0e4e1dc5: Pull complete
012b22e92c79: Pull complete
518f5393dbbc: Pull complete
Digest: sha256:24a5204b418e8fa0214cfb628486749003b039c279c56b5bddb5b10cd100d926
Status: Downloaded newer image for quay.io/prometheus/alertmanager:latest
8177cb714ac6256558917a0c3b165f526baf591f7e295cc12f295e2ab4ef6598
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$
    Code Block
     docker exec -it alertmanager sh
/alertmanager $ vi /etc/alertmanager/alertmanager.yml
/alertmanager $ cp /etc/alertmanager/alertmanager.yml /etc/alertmanager/alertmanager.yml.orig
/alertmanager $
/alertmanager $ vi /etc/alertmanager/alertmanager.yml
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'sooabia@gmail.com'
  smtp_auth_username: ''
  smtp_auth_password: ''

templates:
- '/etc/alertmanager/temlate/*.tmpl'

route:
  repeat_interval: 1h
  receiver: operations-team

receivers:
- name: 'operations-team'
  email_configs:
  - to: 'sooabia@gmail.com'
  slack_configs:
  - api_url: https://hooks.slack.com/services/T01JVFP1UCCXXXXXX/B020ED9QYUD/RvFVPTzArYTOJhgQqp1kKA3URvFVPTzArYTXXXXXXXX
    channel: '#alertmanager-demo'
    send_resolved: true
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/alertmanager $ exit
sansae@sansaeubuntu:~$ docker stop alertmanager
alertmanager
sansae@sansaeubuntu:~$ docker start alertmanager
alertmanager
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$ docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS         PORTS
                              NAMES
8177cb714ac6   quay.io/prometheus/alertmanager   "/bin/alertmanager -…"   17 minutes ago   Up 3 seconds   0.0.0.0:9093->9093/tcp, :::9093->9093/tcp   alertmanager
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$ curl ifconfig.me
20.41.96.191

   Image Added

Setting Prometheus on Docker

Info
Code Block
sansae@sansaeubuntu:~$ docker run -d --name prometheus --link exporter --link alertmanager -p 9090:9090 prom/prometheus
8125a3d79028fca48963236c2f5a3c1ca12fe2287af0e5331253bebf8bb55434
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$ docker exec -it prometheus sh
/prometheus $ vi /etc/prometheus/prometheus.yml
/prometheus $
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - alertmanager:9093
rule_files:
  - "/etc/prometheus/alert.urles"

scrape_configs:
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
    - targets: ['exporter:9100']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/prometheus $
/prometheus $
/prometheus $ vi /prometheus/alert.rules
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
groups:
- name: example
  rules:
  - alert: cpuUsge
    expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{job='node_exporter',mode="idle"}[5m])) * 100) > 45
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: Machine under healvy load
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

/prometheus $ exit
sansae@sansaeubuntu:~$
sansae@sansaeubuntu:~$ docker stop prometheus
prometheus
sansae@sansaeubuntu:~$ docker start prometheus
prometheus
sansae@sansaeubuntu:~$

Image Added


Test Webhook - Slack

Info

Image Added