Prometheus

소개

  • Metric collecting 

    • Push
      • Coupling metric backend system
      •  require (agent | login) per application
    • Pull
      • require service discovery
      • easy to update setting

  • Prometheus

    • Features

      • multi-dimensional data model
      • PromQL
      • no reliance on distributed storage
      • via pull method over HTTP
      • pushing time series is support (gateway)
      • targets r discovered via service discovery or static conf

Metric types

  • Counter

    • cumulative metric (reset 0)
    • ex)
      • total req
      • total send/receive bytes
      • uptime

  • Gauge

    • single numerical value (up & down)
    • ex)
      • cpu / memory usage
      • temperature
      • concurrent req
      • thread cnt

  • Histogram

    • samples observations (샘플링을 관찰)
    • ex)
      • <basename>_bucket{le="<upper inclusive bound>"}
        • SLO to serve 95% of req within 300ms
          • http_request_duration_seconds_bucket{le="0.3"}
      • <basename>_sum
      • <basename>_count

  • Summary

    • samples observations

    • ex)
      • <basename>{quantile="<Ψ>"}
      • <basename>_sum
      • <basename>_count

Grafana


Prometheus metric

Prometheus expression language

  • Sample
    • Prometheus metric 1건
  • Instant vector
    • 동일 시간대의 sample들의 집합
    • prometheus_http_requests_total
  • Range vector
    • prometheus_http_requests_total [1m]
  • Scalar

PromQL

  • Selector

    • Instant vector selector
      • regex
      • prometheus_http_requests_total {code = "200"}
    • Range vector selector
      • prometheus_http_requests_total [1m]
        • (1분 간격으로 그룹핑)
    • Offset
      • prometheus_http_requests_total offset 1m
        • (현재 시점에서 1분전 데이타)
    • @modifier
      • Enable Configuration : "--enable-feature=promql-at-modifier" flag
      • http_requests_total @160974600 offset 5m
        • (특정 시간의 5분전 데이터 추출)

  • Operation (Instance Vector를 위한)

    • sum
      • sum(prometheus_http_requests_totla{})
    • min
    • max
    • avg
    • stddev (표준편차)
    • stdvar (표준분산)
    • count
      • count(prometheus_http_requests_totla{})
    • count_values
      • count_values("xxx", prometheus_http_requests_total{})
        • (동일한 values값을 그룹화하여 몇개가 있는지)
    • bottomk
      • bottomk(2, prometheus_http_requests_total{})
        • (순차 정렬하여 작은 값 2개 추출)
    • topk
      • topk(2, prometheus_http_requests_total{})
        • (내림차순 정렬하여 큰값 2개 추출)
    • quantile
      • 분기수를 구할때 사용

  • Group by

    • by
      • sum(prometheus_http_requests_totla{}) by (code)
        • (code로 그룹화 해서 sum)
    • without
      • sum(prometheus_http_requests_totla{}) without (code)
        • (code는 무시하고 sum)

  • Join

    • one-to-one
      • method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m
    • one-to-many
      • method_code:http_errors:rate5m{code="500"} / ignoring(code) group_left method:http_requests:rate5m




  • No labels