# FactoryBench: production-effect oracle independence and freshness

Benchmark ID: `fb-production-effect-oracle-001`  
Status: protocol design; research phase only  
Execution authorization: none

## Question

Can a proposed production-effect oracle decide whether the **exact artifact, target and exposed cohort** produced an acceptable technical, user, data, security and business outcome, while reporting missing or ambiguous evidence honestly and remaining independent from the system that made the change?

This is narrower than the complete GitOps benchmark. It holds delivery constant and attacks the judge. A generated SLO, successful query, green dashboard, canary score, pipeline verification step or rollback trigger receives no outcome credit until the evidence can be bound to the actual changed population and independently reconciled.

## Contestant classes

- declarative SLO specifications and compilers;
- SLO/burn-rate engines and alert generators;
- canary-analysis and progressive-delivery judges;
- commercial continuous-verification services;
- synthetic user-journey and business-invariant probes;
- the selected Iron Reins effect-oracle assembly once one exists.

Open source, hosted, free and enterprise editions are separate contestants. A schema-only standard is not scored as an evaluator. A metric evaluator is not scored as a traffic controller or recovery engine.

## Protected evidence envelope

Every verdict must preserve:

`approved source revision → immutable artifact digest → target environment/resource identity → actual control and candidate cohort membership → query/probe/policy revision → expected and observed populations → observation/event/recorded times → raw evidence references → result state → release decision → rollback/compensation/recovery receipt`

The change producer may propose a check, but cannot alter the protected oracle, its identity, policy, thresholds, cohort, clocks, evidence store or hidden postconditions during the run.

## Fixture

The fixture exposes one signed application revision to a bounded tenant/workflow cohort. It includes:

- primary and candidate instances with measured—not requested—traffic membership;
- Prometheus-compatible metrics, logs and OpenTelemetry traces;
- low-volume and high-volume synthetic user journeys;
- PostgreSQL invariants, CDC positions and one delayed asynchronous side effect;
- security and authorization probes;
- a protected business outcome and a deliberately attractive proxy metric;
- regional, tenant, device and permission segments;
- immutable query, policy, deployment and evidence revisions;
- an independent recovery observer.

Contestants receive the same signals and declared observation budget. They do not receive hidden fault labels or protected postconditions.

## Fault and red-herring population

1. The query observes the right service name in the wrong namespace or region.
2. The deployment uses a mutable tag while the oracle records only the service label.
3. Requested canary weight is 10 percent but the candidate receives zero traffic.
4. Candidate traffic leaks into the control cohort.
5. Prometheus samples are stale while the last value remains green.
6. Missing telemetry is interpreted as zero errors or omitted from the aggregate.
7. One user segment fails but the aggregate remains below threshold.
8. The query times out or exceeds cardinality limits after returning a partial series.
9. A long SLO window masks a sharp acute regression.
10. Several correlated metrics are counted as independent evidence.
11. Deployment markers are absent, late or bound to the wrong revision.
12. Latency improves only because the service sheds difficult requests.
13. Conversion improves while fraud, authorization or data invariants regress.
14. The database invariant fails after the metric window closes.
15. A delayed queue, email or webhook side effect fails after workload rollback.
16. The metric provider or collector is partitioned.
17. Oracle policy, query or sensitivity changes during the rollout.
18. The change producer and oracle share a principal that can rewrite the verdict.
19. Rollback restores traffic but not data, schema or external effects.
20. A harmless noisy-neighbor event produces correlated latency without candidate causation.
21. A low-throughput service yields insufficient candidate/control samples.
22. A human or pipeline failure strategy converts failure or no-analysis into success.

## Required result model

The contestant must distinguish at least:

- `pass` — sufficient protected evidence supports every required postcondition;
- `fail` — sufficient evidence falsifies at least one required postcondition;
- `abstain` — evidence is valid but insufficient to decide;
- `missing` — an expected evidence population did not arrive;
- `stale` — evidence exists outside its freshness contract;
- `error` — collection, parsing, query or evaluation failed;
- `inconclusive` — signals conflict or the comparison lacks statistical/semantic power.

No non-pass state may be silently laundered into pass. An accountable policy may choose pause, abort, bounded continuation or human review, but that later decision must preserve the original oracle state.

## Required behavior

The contestant must:

1. bind evidence to the exact artifact, target, actual control/candidate cohorts and observation interval;
2. report expected and observed population counts plus sampling, loss, partial-response and freshness state;
3. preserve raw source references, query/policy revision, principal and timestamps;
4. expose the handling of missing, malformed, stale, error, timeout and inconclusive results;
5. abstain when the data cannot support the requested comparison;
6. prevent the change producer from rewriting the oracle or its result;
7. represent protected technical, user, data, security and business postconditions without averaging a fatal failure away;
8. distinguish correlation, threshold breach and statistical difference from a causal claim;
9. trigger only an explicitly authorized release action;
10. reconcile whether rollback, compensation or forward repair restored every mutated domain;
11. retain unresolved and delayed effects through the declared recurrence window;
12. preserve every manual override and its accountable identity and rationale.

## Metrics

- false-promote, false-abort and correct-abstention rates by fault class;
- wrong-artifact, wrong-target and wrong-cohort acceptance rate;
- expected/observed population accounting completeness;
- stale-green and missing-as-green rate;
- evidence-to-decision latency and maximum affected population;
- user, data, security and business postcondition coverage;
- calibration and power by traffic/sample regime;
- causal overclaim rate;
- rollback-trigger accuracy versus full-domain recovery completeness;
- manual override, human intervention and ambiguous-state minutes;
- audit-envelope completeness and replay fidelity;
- control-plane, telemetry, storage and residual-labor cost.

## Fatal gates

Any of the following disqualifies independent production-effect authority for the tested class:

- the verdict cannot identify the exact artifact, target, cohort or policy/query revision;
- missing, stale, error, timeout, partial or no-analysis evidence can become pass by default;
- the producer can change oracle policy or result with the same effective identity;
- expected versus observed evidence populations are unavailable;
- a required user, data, security or business failure is averaged into an overall pass;
- a threshold or statistical difference is presented as causal proof without a discriminating test;
- rollback success is declared without reconciling data and external effects;
- a manual override erases the original state, actor or rationale;
- raw evidence and decision provenance cannot be replayed;
- recovery depends exclusively on the failed verifier or telemetry plane.

## Human authority

Humans retain the production postcondition set; economic, legal and risk trade-offs; irreversible-effect policy; protected-oracle administration; disputed business truth; manual override; incident command; and recovery declaration. Repeated qualification can transfer a narrowly named reversible change class to policy-bounded automation, but cannot transfer authority merely because an SLO, score or dashboard was green.

## Interpretation

A passing result qualifies one oracle implementation, edition, evidence path, workload class, traffic regime and policy revision. It does not establish whole-application correctness, causal diagnosis, safe remediation, business value, or complete recovery. The intended output is a composable authority with explicit abstention—not a universal green light.