# Pathrule Pattern: Prometheus and Grafana Operations (1.0.0) # ::pathrule:package:prometheus-grafana ### [RULE] Keep metric labels finite and operational (path: /observability/metrics) Every unique label set creates a time series with memory, storage, and query cost. An unbounded user ID, URL, exception message, or queue item can grow without a fixed ceiling. - Allowlist labels from stable service, operation, status class, region, version, and other bounded dimensions tied to an operational question. - Normalize route templates instead of raw paths and classify failures instead of labeling full exception text. - Put high-cardinality identities in sampled traces or structured logs with privacy controls, not in metric labels. - Measure active series, churn, samples, scrape size, and top label combinations before and after every new dimension. See /observability/rules for the adjacent decision or procedure that completes this constraint. --- ### [RULE] Alerts express sustained user-impacting conditions (path: /observability/rules) A threshold crossed for one scrape is often noise, while a slow-burning failure can hide under a single average. Alerts should describe an actionable service condition and survive missing data intentionally. - Start from error, latency, availability, freshness, saturation, or correctness symptoms that an owner can act on. - Use an evaluation window and persistence period that match the failure dynamics and data resolution. - Define how absent, stale, reset, and partial metrics affect the expression; missing series must not silently look healthy. - Attach owner, severity, service, environment, summary, and runbook labels or annotations and test the route before relying on the page. See /observability/dashboards for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] Histograms preserve aggregatable latency distributions (path: /observability/metrics) Client-side quantiles cannot be averaged across instances or longer windows. Histograms retain cumulative bucket counts that can be aggregated when boundaries are consistent. - Choose bucket boundaries around user-visible objectives and meaningful tail regions rather than copying a generic sequence without observing the service. - Use identical boundaries for series that must aggregate and keep the number of buckets compatible with cardinality and scrape cost. - Calculate rates before histogram quantiles over the selected window and group by only the dimensions needed for the question. - Measure count and sum alongside tail estimates and preserve units in metric names and dashboard labels. See /observability/dashboards for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Recording rules are a reviewed query API (path: /observability/rules) Dashboards and alerts that each repeat a complex raw query create cost and semantic drift. A recording rule can centralize the computation but becomes a public derived metric that must be versioned carefully. - Create recording rules for frequently reused rates, service-level indicators, and expensive aggregations after confirming their label and window semantics. - Name the rule to expose level, metric, operation, and aggregation convention without embedding an arbitrary dashboard range. - Drop labels that no longer carry meaning after aggregation and retain only dimensions downstream users are allowed to split. - Test rules against fixture series with counter resets, missing targets, multiple instances, and zero traffic before alerts and dashboards depend on them. See /observability/dashboards for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Dashboards preserve query meaning across time ranges (path: /observability/dashboards) A dashboard can look plausible while changing meaning when the time range or resolution changes. Fixed rate windows, raw counters, and unlabeled aggregates create false comparisons. - Use range-aware rate intervals large enough for scrape cadence and keep query windows visible in panel descriptions where interpretation depends on them. - Label units, aggregation, scope, and missing-data behavior; do not mix seconds and milliseconds or totals and per-second rates on one axis without clarity. - Lead with service health and objectives, then provide traffic, errors, latency, saturation, deployment, and dependency drilldowns. - Link to related traces, logs, runbooks, deployments, and owner context using bounded identifiers instead of copying high-cardinality data into metrics. See /observability/rules for the rule or workflow that puts this decision into practice. --- ### [SKILL] review-prometheus-metric (path: /) --- name: review-prometheus-metric description: Review a new or changed Prometheus metric and its Grafana use. --- # Review Prometheus Metric Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. - [ ] State the operational question, metric type, unit, event or observation point, and which component owns emission. - [ ] Enumerate every label and its maximum values, churn, attacker control, privacy class, and aggregation meaning. - [ ] Calculate projected series and sample volume across replicas, environments, status values, buckets, and deployment overlap. - [ ] Test counter reset, missing target, no traffic, duplicate scrape, restart, and mixed-version behavior in recording and dashboard queries. - [ ] Deploy to a small scope, inspect actual series and query cost, then remove experimental labels and panels that do not answer a real question. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks. --- ### [SKILL] validate-prometheus-alert (path: /) --- name: validate-prometheus-alert description: Validate a Prometheus alert and Grafana evidence before it pages an owner. --- # Validate Prometheus Alert Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. 1. Evaluate the expression over representative history with incidents, deployments, quiet periods, missing targets, counter resets, and seasonal load. 2. Force the condition in a test environment and confirm pending duration, firing labels, annotations, grouping, routing, inhibition, and notification delivery. 3. Open the linked dashboard and runbook from the notification and verify they identify scope, user impact, likely causes, and safe first actions. 4. Resolve the condition and confirm recovery timing, state retention, and notification behavior without flapping. 5. Record expected false-positive and false-negative tradeoffs, owner, review date, and the exact signal that would justify retuning. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.