Prometheus

소개

• Metric collecting 

  • Push
    • Coupling metric backend system
    •  require (agent | login) per application
  • Pull
    
    • require service discovery
    • easy to update setting

• Prometheus

  • Features

    • multi-dimensional data model
    • PromQL
    • no reliance on distributed storage
    • via pull method over HTTP
    • pushing time series is support (gateway)
    • targets r discovered via service discovery or static conf

Metric types

• Counter

  • cumulative metric (reset 0)
  • ex)
    • total req
    • total send/receive bytes
    • uptime

• Gauge

  • single numerical value (up & down)
  • ex)
    • cpu / memory usage
    • temperature
    • concurrent req
    • thread cnt

• Histogram

  • samples observations (샘플링을 관찰)
  • ex)
    • <basename>_bucket{le="<upper inclusive bound>"}
      • SLO to serve 95% of req within 300ms
        • http_request_duration_seconds_bucket{le="0.3"}
    • <basename>_sum
    • <basename>_count

• Summary

  • samples observations

  • ex)
    • <basename>{quantile="<Ψ>"}
    • <basename>_sum
    • <basename>_count

Grafana

Prometheus metric

Prometheus expression language

• Sample
  • Prometheus metric 1건
• Instant vector
  • 동일 시간대의 sample들의 집합
  • prometheus_http_requests_total
  • 
• Range vector
  • prometheus_http_requests_total [1m]
  • 
• Scalar

PromQL

• Selector

  • Instant vector selector
    • regex
    • prometheus_http_requests_total {code = "200"}
    • 
  • Range vector selector
    • prometheus_http_requests_total [1m]
      • (1분 간격으로 그룹핑)
  • Offset
    • prometheus_http_requests_total offset 1m
      • (현재 시점에서 1분전 데이타)
  • @modifier
    • Enable Configuration : "--enable-feature=promql-at-modifier" flag
    • http_requests_total @160974600 offset 5m
      • (특정 시간의 5분전 데이터 추출)

• Operation (Instance Vector를 위한)

  • sum
    • sum(prometheus_http_requests_totla{})
  • min
  • max
  • avg
  • stddev (표준편차)
  • stdvar (표준분산)
  • count
    • count(prometheus_http_requests_totla{})
  • count_values
    • count_values("xxx", prometheus_http_requests_total{})
      • (동일한 values값을 그룹화하여 몇개가 있는지)
  • bottomk
    • bottomk(2, prometheus_http_requests_total{})
      • (순차 정렬하여 작은 값 2개 추출)
  • topk
    • topk(2, prometheus_http_requests_total{})
      • (내림차순 정렬하여 큰값 2개 추출)
  • quantile
    • 분기수를 구할때 사용

• Group by

  • by
    • sum(prometheus_http_requests_totla{}) by (code)
      
      • (code로 그룹화 해서 sum)
  • without
    • sum(prometheus_http_requests_totla{}) without (code)
      • (code는 무시하고 sum)

• Join

  • one-to-one
    • method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m
  • one-to-many
    • method_code:http_errors:rate5m{code="500"} / ignoring(code) group_left method:http_requests:rate5m