# Pathrule Pattern: Kubernetes + Helm (1.0.0) # ::pathrule:package:kubernetes-helm ### [RULE] Every container declares resource requests and limits (path: /k8s) Requests are what the scheduler reserves; limits are what the kernel enforces. A container with neither is scheduled blind and evicted first when the node is under pressure. - Set a CPU and memory request on every container, including init and sidecar containers, based on observed steady-state usage rather than a guess. - Set the memory limit equal to the request for anything you care about. Memory is not compressible: exceeding the limit is an OOM kill, and a request far below the limit means the pod is in the Burstable class and gets evicted early. - Be deliberate with CPU limits. A limit throttles the container at the boundary, which shows up as latency spikes rather than load. For latency-sensitive services, prefer a solid request and a generous or absent limit. - Give a namespace a ResourceQuota and LimitRange so a missing request is caught by the cluster, not discovered during an incident. - Revisit the numbers with real data (a vertical autoscaler in recommend mode, or your metrics) instead of copying them between services. --- ### [RULE] Use all three probes, each for its own job (path: /k8s) The three probes answer three different questions, and copying one endpoint into all three slots produces restart loops during deploys. - `startupProbe` covers initialisation: while it is failing, liveness and readiness are suspended, so a slow-booting app is not killed before it is up. Give it a generous `failureThreshold`. - `readinessProbe` decides whether the pod receives traffic. It should check that this process can serve a request right now (caches warm, migrations applied, worker pool started). - `livenessProbe` decides whether to restart the process. It must only fail when a restart genuinely fixes things: a deadlock, an unrecoverable state. A liveness probe that checks the database restarts every pod during a database blip and turns a partial outage into a total one. - Keep probe endpoints cheap, dependency-light, and unauthenticated within the cluster. No probe should run a query per call. - Tune `periodSeconds`, `timeoutSeconds`, and thresholds so a transient hiccup does not remove capacity, and make readiness fail fast so shutdown drains quickly. --- ### [RULE] Survive disruption: budgets, spread, and graceful shutdown (path: /k8s) Nodes are drained for upgrades constantly. A workload that cannot tolerate that is a workload that has an outage on every cluster maintenance window. - Run at least two replicas for anything serving traffic, and set a `PodDisruptionBudget` (`minAvailable` or `maxUnavailable`) that keeps enough capacity while still allowing a drain to make progress. A budget that blocks every eviction is worse than none: it stalls upgrades. - Spread replicas with `topologySpreadConstraints` across nodes and, where available, zones, so one node or zone loss is not the whole service. - Set `rollingUpdate` `maxUnavailable` and `maxSurge` intentionally, together with the readiness probe, so a bad rollout stops before it replaces every pod. - Handle SIGTERM: fail readiness immediately, finish in-flight requests, then exit. Add a short `preStop` sleep so the endpoints controller removes the pod from the load balancer before the process stops accepting connections, and set `terminationGracePeriodSeconds` longer than your slowest request. - For batch and queue workers, make shutdown mean "stop taking new work and finish the current item", and make the work idempotent so a kill mid-item is safe. --- ### [RULE] Harden the pod: non-root, read-only, no capabilities, pinned image (path: /k8s) A container defaults to more privilege than it needs, and a mutable tag means you cannot say what is running. - Set `securityContext` with `runAsNonRoot: true`, an explicit `runAsUser`/`runAsGroup`, `allowPrivilegeEscalation: false`, `readOnlyRootFilesystem: true`, and `capabilities.drop: ["ALL"]`. Mount an `emptyDir` for the paths that genuinely need to be writable. - Never `privileged: true`, never host network, PID, or IPC namespaces, and no host path mounts outside a deliberate, reviewed infrastructure workload. - Pin images to an immutable tag or a digest and set `imagePullPolicy: IfNotPresent` with that pin. `:latest` makes rollbacks meaningless and two replicas can silently run different code. - Give each workload its own ServiceAccount with only the RBAC it needs, and disable the token mount (`automountServiceAccountToken: false`) when the pod does not call the API server. - Restrict traffic with NetworkPolicies: default deny in the namespace, then allow the specific flows. Enforce the whole set with Pod Security Admission or a policy engine so the next manifest cannot skip it. See the docker-containers pattern for how the image itself is built, and supply-chain-security for provenance. --- ### [RULE] Keep charts declarative, versioned, and free of secrets (path: /charts) A Helm chart is the deployable contract for a service. Treated casually, it becomes a template nobody can render safely. - No secret values in `values.yaml` or in a committed values file. Reference an existing Secret, or generate it from a secret manager (External Secrets, Secrets Store CSI, sealed secrets). Charts are shared and end up in artifact registries. - Give every value a documented default that produces a working, safe deployment, and validate inputs with `values.schema.json` so a typo fails at install rather than at runtime. - Bump the chart `version` on every change and set `appVersion` to the image tag. An unversioned chart makes rollback a guess. - Deploy with `helm upgrade --install --atomic --timeout`, so a failed rollout rolls back instead of leaving half the replicas on new code. Run `helm template` and a policy check (kubeconform, conftest, or your admission policies) in CI. - Do not hand-edit live resources: `kubectl edit` on a chart-managed object is drift that the next upgrade will either revert or fight. Change the chart. - Add a checksum annotation on the pod template for mounted ConfigMaps and Secrets so a config change actually triggers a rollout. --- ### [MEMORY] Configuration, secrets, and how a change reaches a pod (path: /k8s) Config delivery in Kubernetes is subtle in exactly one way: what happens to a running pod when the config changes. - A Secret or ConfigMap consumed as environment variables is read once at start. Changing it does nothing until the pod restarts, so a rollout must be triggered (the standard trick is a checksum annotation on the pod template). - The same object mounted as a volume is updated in place after a short delay, which only helps if the application re-reads the file. Decide which behaviour you want and implement it deliberately. - Keep secret material out of the manifest tree. Pull it from a manager at deploy time or sync it into the cluster, and let RBAC decide who can read the Secret object. - Separate genuinely environment-specific values (replicas, hostnames, resource sizes) from application configuration, so promoting a release does not require editing application settings. - Keep the whole desired state in git and let a controller reconcile it (GitOps) rather than applying from a laptop, so the cluster and the repo cannot disagree silently. See /charts for the chart rules and the secrets-env-management pattern for storage and rotation. --- ### [MEMORY] Scaling: autoscalers only work with honest requests (path: /k8s) Autoscaling failures are almost always upstream of the autoscaler. - Scale on the metric that reflects saturation for the workload: CPU for compute-bound services, queue depth or requests per second for I/O-bound ones (through custom or external metrics). CPU utilisation is measured against the request, so a wrong request makes the target meaningless. - Set `minReplicas` above one for availability, a `maxReplicas` you can actually afford, and stabilisation windows so the autoscaler does not oscillate. Scale up fast, scale down slowly. - The cluster autoscaler adds nodes when pods are pending. Pods pend because of requests, affinity, or topology constraints, so overly strict constraints look like an autoscaler that does not work. - Do not run a horizontal and a vertical autoscaler on the same resource metric; they fight. Use vertical recommendations to set requests, then scale horizontally. - For bursty batch work, prefer a queue plus workers that scale on depth over a fixed deployment sized for the peak, and use priority classes so batch work yields to interactive traffic. See /k8s for the requests rule that all of this depends on. --- ### [SKILL] k8s-production-readiness (path: /) --- name: k8s-production-readiness description: Production readiness gate for a Kubernetes workload: resources, probes, disruption tolerance, security context, config delivery, and observability. Run before the first production deploy and before merging manifest or chart changes. --- # Kubernetes production readiness ## Scheduling - [ ] CPU and memory requests set on every container, from observed usage. - [ ] Memory limit equals request; CPU limit chosen deliberately. - [ ] Namespace has a ResourceQuota and LimitRange. ## Health - [ ] Startup, readiness, and liveness probes are distinct and cheap. - [ ] Liveness does not check external dependencies. - [ ] Readiness fails immediately on SIGTERM. ## Disruption - [ ] At least two replicas for serving workloads. - [ ] PodDisruptionBudget allows a drain to complete while keeping capacity. - [ ] `topologySpreadConstraints` spread replicas across nodes and zones. - [ ] `preStop` and `terminationGracePeriodSeconds` drain in-flight requests. ## Security - [ ] `runAsNonRoot`, explicit user, `readOnlyRootFilesystem`, `capabilities.drop: [ALL]`, no privilege escalation. - [ ] No privileged mode, host namespaces, or host path mounts. - [ ] Image pinned to an immutable tag or digest. - [ ] Dedicated ServiceAccount, minimal RBAC, token mount off if unused. - [ ] NetworkPolicy default-deny plus explicit allows. ## Configuration - [ ] No secrets in chart values or committed files. - [ ] Config change triggers a rollout (checksum annotation) or is re-read at runtime. - [ ] Chart version bumped; `helm upgrade --atomic` used. ## Operations - [ ] Logs go to stdout as structured lines. - [ ] Metrics and traces are exported, and dashboards and alerts exist. - [ ] A rollback command is written down and has been tested once.