Prometheus and Grafana Operations

Pathrule2 Rules • 3 Memories • 2 Skills

Prometheus reliability depends on label cardinality, scrape and query cost, series churn, retention, rule evaluation, and alert semantics, while Grafana can hide expensive queries and ambiguous units behind a polished dashboard. This pattern constrains metric labels and alert evidence, records histogram, recording-rule, and dashboard ownership, and supplies separate metric-review and alert-validation procedures. It deepens the general observability pattern with concrete PromQL, time-series cardinality, scrape, rule, and visualization operations rather than traces and logs.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

review-prometheus-metric

validate-prometheus-alert

observability/

metrics/

Keep metric labels finite and operational

Histograms preserve aggregatable latency distributions

rules/

Alerts express sustained user-impacting conditions

Recording rules are a reviewed query API

dashboards/

Dashboards preserve query meaning across time ranges

Rules

Keep metric labels finite and operational/observability/metricshighstrictUse bounded dimensions that support aggregation and diagnosis; never encode user, request, payload, or raw path identity in labels.

1	Every unique label set creates a time series with memory, storage, and query cost. An unbounded user ID, URL, exception message, or queue item can grow without a fixed ceiling.
2
3	- Allowlist labels from stable service, operation, status class, region, version, and other bounded dimensions tied to an operational question.
4	- Normalize route templates instead of raw paths and classify failures instead of labeling full exception text.
5	- Put high-cardinality identities in sampled traces or structured logs with privacy controls, not in metric labels.
6	- Measure active series, churn, samples, scrape size, and top label combinations before and after every new dimension.
7
8	See /observability/rules for the adjacent decision or procedure that completes this constraint.

Alerts express sustained user-impacting conditions/observability/ruleshighstrictCombine symptom, duration, scope, and routing metadata with a runbook instead of paging on raw resource noise.

1	A threshold crossed for one scrape is often noise, while a slow-burning failure can hide under a single average. Alerts should describe an actionable service condition and survive missing data intentionally.
2
3	- Start from error, latency, availability, freshness, saturation, or correctness symptoms that an owner can act on.
4	- Use an evaluation window and persistence period that match the failure dynamics and data resolution.
5	- Define how absent, stale, reset, and partial metrics affect the expression; missing series must not silently look healthy.
6	- Attach owner, severity, service, environment, summary, and runbook labels or annotations and test the route before relying on the page.
7
8	See /observability/dashboards for the adjacent decision or procedure that completes this constraint.

Memories

Histograms preserve aggregatable latency distributions/observability/metricsChoose buckets from service objectives and observed shape, then calculate rates and quantiles from bucket counters.

1	Client-side quantiles cannot be averaged across instances or longer windows. Histograms retain cumulative bucket counts that can be aggregated when boundaries are consistent.
2
3	- Choose bucket boundaries around user-visible objectives and meaningful tail regions rather than copying a generic sequence without observing the service.
4	- Use identical boundaries for series that must aggregate and keep the number of buckets compatible with cardinality and scrape cost.
5	- Calculate rates before histogram quantiles over the selected window and group by only the dimensions needed for the question.
6	- Measure count and sum alongside tail estimates and preserve units in metric names and dashboard labels.
7
8	See /observability/dashboards for the rule or workflow that puts this decision into practice.

Recording rules are a reviewed query API/observability/rulesPrecompute stable expensive aggregations with explicit names, units, labels, ownership, and tests.

1	Dashboards and alerts that each repeat a complex raw query create cost and semantic drift. A recording rule can centralize the computation but becomes a public derived metric that must be versioned carefully.
2
3	- Create recording rules for frequently reused rates, service-level indicators, and expensive aggregations after confirming their label and window semantics.
4	- Name the rule to expose level, metric, operation, and aggregation convention without embedding an arbitrary dashboard range.
5	- Drop labels that no longer carry meaning after aggregation and retain only dimensions downstream users are allowed to split.
6	- Test rules against fixture series with counter resets, missing targets, multiple instances, and zero traffic before alerts and dashboards depend on them.
7
8	See /observability/dashboards for the rule or workflow that puts this decision into practice.

Dashboards preserve query meaning across time ranges/observability/dashboardsUse rates and intervals that adapt correctly, show units and scope, and provide drill paths from service symptoms to evidence.

1	A dashboard can look plausible while changing meaning when the time range or resolution changes. Fixed rate windows, raw counters, and unlabeled aggregates create false comparisons.
2
3	- Use range-aware rate intervals large enough for scrape cadence and keep query windows visible in panel descriptions where interpretation depends on them.
4	- Label units, aggregation, scope, and missing-data behavior; do not mix seconds and milliseconds or totals and per-second rates on one axis without clarity.
5	- Lead with service health and objectives, then provide traffic, errors, latency, saturation, deployment, and dependency drilldowns.
6	- Link to related traces, logs, runbooks, deployments, and owner context using bounded identifiers instead of copying high-cardinality data into metrics.
7
8	See /observability/rules for the rule or workflow that puts this decision into practice.

Skills

review-prometheus-metric/rootReview a metric family for semantics, type, units, cardinality, lifecycle, aggregation, cost, and privacy before rollout.

1	---
2	name: review-prometheus-metric
3	description: Review a new or changed Prometheus metric and its Grafana use.
4	---
5
6	# Review Prometheus Metric
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	- [ ] State the operational question, metric type, unit, event or observation point, and which component owns emission.
11	- [ ] Enumerate every label and its maximum values, churn, attacker control, privacy class, and aggregation meaning.
12	- [ ] Calculate projected series and sample volume across replicas, environments, status values, buckets, and deployment overlap.
13	- [ ] Test counter reset, missing target, no traffic, duplicate scrape, restart, and mixed-version behavior in recording and dashboard queries.
14	- [ ] Deploy to a small scope, inspect actual series and query cost, then remove experimental labels and panels that do not answer a real question.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

validate-prometheus-alert/rootBacktest, fire, route, silence, and recover an alert with real metric edge cases and an executable runbook.

1	---
2	name: validate-prometheus-alert
3	description: Validate a Prometheus alert and Grafana evidence before it pages an owner.
4	---
5
6	# Validate Prometheus Alert
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	1. Evaluate the expression over representative history with incidents, deployments, quiet periods, missing targets, counter resets, and seasonal load.
11	2. Force the condition in a test environment and confirm pending duration, firing labels, annotations, grouping, routing, inhibition, and notification delivery.
12	3. Open the linked dashboard and runbook from the notification and verify they identify scope, user impact, likely causes, and safe first actions.
13	4. Resolve the condition and confirm recovery timing, state retention, and notification behavior without flapping.
14	5. Record expected false-positive and false-negative tradeoffs, owner, review date, and the exact signal that would justify retuning.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

Why this pattern

AI agents often put user IDs or URLs in labels, average precomputed percentiles, alert on one noisy sample, or build dashboards with queries that scan raw high-cardinality series repeatedly.

Built for Platform and service teams operating Prometheus-compatible metrics and Grafana dashboards.

Keeps your assistant from:

Exhausting memory through unbounded label values
Computing invalid latency aggregates from client-side quantiles
Paging on transient noise without user impact
Showing rates, totals, and units that change with dashboard interval

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-08-25