# Production-effect oracles and continuous verification

Research date: 2026-08-24  
Status: primary-source architecture assessment; no candidate product was run

## Executive verdict

No assessed system is a complete, independent production-effect oracle.

The market contains four different product classes that are routinely placed in the same comparison:

1. **Grammar:** OpenSLO defines portable SLO documents.
2. **Accountants:** Pyrra, Sloth, Grafana SLO and Nobl9 calculate or report service-level evidence.
3. **Release judges:** Keptn, Kayenta, Harness Continuous Verification and Datadog Deployment Gates return a verdict in a delivery workflow.
4. **Traffic actuators:** Argo Rollouts and Flagger meter exposure and can restore workload or traffic state.

None independently binds an immutable artifact to the actual exposed population, proves that evidence is fresh and complete, represents user, data, security and business effects, preserves ambiguity as abstention, remains protected from the proposing factory, causes domain-appropriate rollback and then proves whole-system recovery. Product green therefore means only the product's narrow contract passed.

## Same-column hard-edge comparison

Legend: **yes** means source-visible built-in support; **bounded** means possible only with material configuration or a caller; **no** means outside the system's contract.

| System | Actual class | Exact change binding | Population proof | Honest abstention | User/data/business effect | Actuation | Proved recovery | Dangerous green edge |
|---|---|---:|---:|---:|---:|---:|---:|---|
| OpenSLO | grammar | no | no | no | bounded schema | no | no | A valid document has no runtime meaning by itself. |
| Pyrra | accountant | no | no | no | bounded metrics | no | no | Total absence can alert; partial loss or a wrong selector can remain plausible. |
| Sloth | accountant/compiler | no | no | no | bounded metrics | no | no | Missing five-minute samples disappear from sum/count arithmetic. |
| Grafana SLO | managed accountant | no | no | bounded | bounded metrics | no | no | Rules do not backfill, maintenance creates gaps and long-window health is not a release verdict. |
| Nobl9 | managed accountant | no | no | no | bounded metrics | no | no | No-data time-slice minutes count as good. |
| Keptn | lifecycle judge | bounded | no | no | bounded queries | no | no | An objective without failure/warning criteria passes; nonblocking failure can leave deployment successful. |
| Kayenta | statistical judge | bounded | bounded | no | bounded metrics | caller | no | Caught metric errors leave normal scoring; an all-error group can effectively score 100. |
| Argo Rollouts | traffic actuator/judge host | bounded | bounded | yes | bounded providers/jobs | yes | no | A metric without success/failure expressions defaults to success. |
| Flagger | traffic actuator/judge host | bounded | bounded | no | bounded metrics/webhooks | yes | no | Missing values fail, but missing and harmful evidence are not distinct states. |
| Harness CV | managed release judge | yes | bounded | no | bounded providers | yes | no | `No Analysis` passes unless fail-on-no-analysis is enabled. |
| Datadog Deployment Gates | preview release judge | yes | bounded | no | bounded monitors/rules | caller | no | The official action defaults `fail-on-error` to false; dry-run always passes. |

The critical distinction is between **observing a number**, **judging a release**, **moving traffic**, and **proving the system recovered**. Buying one does not confer the others.

## Open grammar and SLO accountants

### OpenSLO

[OpenSLO](https://github.com/OpenSLO/OpenSLO) is a specification, not an evaluator. Its green zone is portable vocabulary for services, indicators, objectives and alerts. It has no runtime, evidence store, release identity, cohort ledger, pass/fail/abstain semantics, rollback or recovery. Treat it as an interchange format only.

### Pyrra

[Pyrra v0.10.1](https://github.com/pyrra-dev/pyrra/tree/v0.10.1) is an Apache-2.0 Prometheus SLO system. Its useful open surface is generated recording rules, burn-rate alerts and a UI. The source includes absent-metric alerts, but these establish total absence only. They cannot prove that every expected label or tenant is present, that a selector points at the changed cohort, or that an exporter did not emit stale values. The v0.10.1 release corrected denominator double-counting and alert scoping, which is also evidence that the accounting implementation—not only the SLO document—is part of the correctness boundary.

### Sloth

[Sloth v0.16.0](https://github.com/slok/sloth/tree/v0.16.0) is an Apache-2.0 compiler from its SLO format and a bounded OpenSLO subset to Prometheus rules. Deterministic generation and reviewable output are its strength. Its arithmetic does not create a first-class missing state: absent samples are omitted from sums and counts. Plugins can replace validation and rule generation. This makes it a good rule compiler behind protected review, not a release authority.

### Grafana SLO

[Grafana Cloud SLO](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/) is a managed SLO product rather than the backend of an OSS Grafana component. It fits an existing Grafana estate, but its seven-day minimum objective window, 28-day default, lack of recording-rule backfill and maintenance gaps make it especially unsuitable as an exact release-effect oracle without a second short-window gate. Published pricing starts with Free, Pro at $19/month plus usage, and enterprise at a $25,000 annual minimum; SLO costs are entangled with active-series and platform usage rather than a stable per-verdict price.

### Nobl9

[Nobl9](https://docs.nobl9.com/) is a closed SaaS SLO platform; [`sloctl`](https://github.com/nobl9/sloctl) is open tooling, not the service backend. The decisive semantic edge is missing data: occurrence windows add neither good nor total, while no-data time-slice minutes count as good. Agent cursor persistence is off by default for agent-mediated sources, and replay re-queries the provider rather than reproducing the original evidence. The published Teams package is $14,950/year for 50 SLO units, ten sources and two-year retention. It centralizes SLO governance but does not bind release causality or own recovery.

## Lifecycle and canary judges

### Keptn

[Keptn Lifecycle Toolkit v2.5.0](https://github.com/keptn/lifecycle-toolkit/tree/keptn-v2.5.0) attaches pre-deployment, post-deployment and evaluation tasks to Kubernetes workloads. Blocking prechecks are useful admission controls. However, a missing failure/warning criterion allows an objective to pass, nonblocking failures can result in warning while deployment succeeds, and post-deployment evaluation explicitly does not revert the deployment. The current project is active; the older [`keptn/keptn`](https://github.com/keptn/keptn) control plane reached end of life in 2023 and should not be conflated with it. Keptn is an open lifecycle hook/evaluation organ, not an independent release-and-recovery system.

### Kayenta

[Kayenta in the pinned Spinnaker monorepo](https://github.com/spinnaker/spinnaker/tree/a50144fcecce2ce505cba7c28ceeedfaad26f866/kayenta) compares human-supplied control and experiment scopes using Mann–Whitney analysis, effect thresholds and weighted scoring. The standalone Kayenta repository is archived; current code lives in the Spinnaker monorepo. `NaN` defaults to removal unless configured otherwise, `mustHaveData` is false by default, and at least 50% no-data scores a group zero. More dangerously, caught per-metric runtime exceptions become noncritical `Error`, which is excluded from normal denominator and no-data handling; an all-error group can therefore effectively score 100. Kayenta emits Pass, Marginal or Fail—not abstain—and depends on Spinnaker or another caller for action.

### Argo Rollouts

[Argo Rollouts v1.9.1](https://github.com/argoproj/argo-rollouts/tree/v1.9.1) is the strongest open actuator in this cohort for an Argo-oriented EKS stack. It owns canary/blue-green steps, pauses, experiments, traffic routing and abort, and it preserves Successful, Failed, Inconclusive and Error states. But a metric without success or failure expressions defaults to success, and provider-specific response parsing can create attractive false greens. Abort can restore workload and traffic state; it cannot reverse data, messages, notifications, identity changes or other external effects.

### Flagger

[Flagger v1.44.0](https://github.com/fluxcd/flagger/tree/v1.44.0) is an open Flux-oriented actuator for canary and blue-green progression. Metric and webhook analysis, automated promotion and rollback are valuable, but their truth is inherited from configured providers. Missing values contribute to failure without becoming an explicit abstain class. As with Argo Rollouts, workload/traffic restoration is not complete recovery.

## Commercial release judges

### Harness Continuous Verification

[Harness Continuous Verification](https://developer.harness.io/docs/continuous-delivery/verify/verify-deployments-with-the-verify-step/) has the strongest built-in execution, service, environment and cohort binding observed in this pass. Canary mode can compare concurrent primary and candidate populations, and supported deployment types can trigger rollback. Its dangerous defaults matter: `No Analysis` passes unless a fail option is enabled, and failure strategies can convert verification failure into workflow success. The verification engine is not present in the Harness OSS repositories. Flex pricing is published at $0.75 per HSU for Essentials and $1.25 per HSU for Enterprise after 1,000 free HSUs, but no stable CV-verdict-to-HSU conversion supports a defensible per-release cost.

### Datadog Deployment Gates

[Datadog Deployment Gates](https://docs.datadoghq.com/deployment_gates/) is a Preview feature that binds checks to service, environment and version and can use monitors or just-in-time rules. No matching monitor or more than 50 matches fails, and monitor `ALERT`/`NO_DATA` states fail. The integration edges are weaker: the [official GitHub Action](https://github.com/DataDog/deployment-gate-github-action) defaults `fail-on-error` to false; dry-run always passes; and the generic script can treat exhausted 5xx retries as success. Faulty Deployment Detection is observational and publishes no precision/recall denominator. There is no public gate-specific price or open backend.

## Opinionated Iron Reins assembly

For a greenfield, EKS-centered enterprise platform, the smallest credible open assembly is:

1. **OpenSLO only as interchange**, not as a runtime.
2. **Choose Pyrra or Sloth, not both**, for reviewable Prometheus SLO rule generation.
3. **Prometheus plus OpenTelemetry** for telemetry, with an explicit expected-series and watermark ledger that fails closed on partial, stale or delayed evidence.
4. **Argo Rollouts** for progressive workload/traffic exposure because the surrounding stack already prefers Argo.
5. **A protected external verifier** that is not writable by the coding factory and binds immutable artifact, cluster, environment, rollout revision, tenant/user cohort and time window.
6. **Source-native probes:** synthetic user journeys, SQL/data invariants, authorization and security checks, queue/notification checks, and business-outcome controls.
7. **Explicit states:** pass, fail, abstain, missing, stale, error and inconclusive. Never collapse these to a Boolean green.
8. **A domain recovery coordinator** that inventories every mutated domain, triggers workload/traffic abort, invokes data and external compensation, and proves the recurrence window is clean.

Do not add Keptn, Kayenta or Flagger merely to increase apparent coverage. Each additional controller creates identity, policy, upgrade, telemetry and failure interactions. Add one only when a sealed benchmark demonstrates a missing function that Argo Rollouts plus the protected verifier cannot supply.

## Enterprise and OSS boundary

| System | Open surface | Closed or paid surface | Price evidence |
|---|---|---|---|
| OpenSLO | specification | evaluator chosen separately | none |
| Pyrra | complete Apache-2.0 project | no canonical enterprise edition | none |
| Sloth | complete Apache-2.0 project | no canonical enterprise edition | none |
| Keptn | complete Apache-2.0 project | vendor services/support separate | no canonical product price |
| Kayenta | source in Apache-2.0 Spinnaker | managed Spinnaker offerings separate | no canonical Kayenta price |
| Argo Rollouts | complete Apache-2.0 controller | managed Argo platforms separate | no Rollouts-specific price |
| Flagger | complete Apache-2.0 controller | managed Flux platforms/support separate | no Flagger-specific price |
| Grafana SLO | documentation/client surface | Grafana Cloud backend | Free; Pro $19/month + usage; enterprise $25k/year minimum |
| Nobl9 | `sloctl` client tooling | SaaS evaluator/backend | Teams $14,950/year published package |
| Harness CV | adjacent Harness OSS repos | CV engine and managed control plane | Flex $0.75/$1.25 per HSU; mapping unresolved |
| Datadog Gates | integration clients/actions | gate and detection backend | no gate-specific public price |

## Benchmark consequence

The companion [`fb-production-effect-oracle-001`](../benchmarks/production-effect-oracle.md) attacks 22 conditions including wrong artifact, wrong cohort, label collapse, partial telemetry loss, stale windows, provider error, correlated judge, Goodhart metrics, harmful data changes, apparently successful rollback and recurrence after recovery. A product is not credited for a state it cannot represent. A benchmark run has not yet been authorized; this pass defines the protocol and source-backed hypotheses only.

## Adoption boundary

Public adoption is generally described through community, customer or usage claims without a common denominator. Keptn's CNCF survey evidence reported 2% production and 5% evaluation in a 2025 sample whose denominator varied by question. No comparable organization-level denominator was found for Pyrra, Sloth, OpenSLO, Kayenta, Grafana SLO, Nobl9, Harness CV, Datadog Gates, Argo Rollouts or Flagger. Repository activity, integrations and customer logos are not production-outcome rates.

## What remains human

Humans still own the acceptance policy, harm and cost trade-offs, irreversible domain recovery, exception approval and final accountability. A factory may propose probes and thresholds, but allowing the same factory to author the change, weaken the judge and declare success destroys independence. The path to more autonomy is not fewer explicit authorities; it is narrower, machine-checkable authority with fail-closed handoffs and evidence that another component can challenge.