# FactoryBench: GitOps, progressive delivery and production-effect closure

Benchmark ID: `fb-gitops-effect-closure-001`  
Status: protocol design; research phase only  
Execution authorization: none

## Question

After a software factory proposes and verifies a change, can the delivery assembly preserve the exact approved artifact, reconcile the intended Kubernetes state, expose it to a bounded cohort, detect real harm, stop or reverse safely, and prove the resulting technical and business state?

This benchmark scores the complete handoff. It does not award production credit for a merged manifest, a `Synced` application, a `Ready` Deployment, a completed rollout or a green metric in isolation.

## Contestants

- Argo CD alone;
- Flux alone;
- Argo CD + Kargo + Argo Rollouts;
- Flux + Flagger;
- protected CI + direct Helm/kubectl as a deliberately simpler control;
- the selected Iron Reins assembly once one exists.

Every contestant receives the same signed commits, immutable images, policies, identities, clusters, telemetry, synthetic journeys and hidden business invariants. No coding agent may alter the verifier, signer, admission policy, canary population, stop thresholds or recovery oracle.

## Required chain of custody

`approved commit → protected build → image digest + SBOM/provenance → promotion decision → desired-state revision → applying identity → live object UID/generation → traffic cohort → user/business observations → promote/abort → reconciliation/recovery receipt`

Each arrow becomes an `IntegrationContract`. The run fails chain-of-custody if any controller substitutes a mutable tag, loses the source revision, applies with an unbounded shared identity, silently skips a verifier, or cannot bind observations to the exact cohort and artifact digest.

## Fixture

The target is a two-service EKS application with:

- API and asynchronous worker;
- PostgreSQL schema plus backward-compatible expand/migrate/contract sequence;
- object-storage side effect and one externally visible notification;
- tenant and workflow cohort routing;
- Kubernetes Deployment/Service/Ingress, one custom resource and one hook Job;
- OpenTelemetry traces, Prometheus technical metrics, synthetic journeys and a protected business invariant;
- approved non-production, canary and production stages;
- immutable image digests, signed provenance and an intentionally mutable tag trap.

Git, registry, Kubernetes API, rollout controller, telemetry backend and business oracle retain separate identities and audit streams.

## Outcome layers

| Layer | What may prove it | What does not prove it |
|---|---|---|
| source intent | protected commit, review and policy decision | branch name or agent statement |
| artifact identity | digest, signature, SBOM/provenance and source binding | mutable image tag |
| desired-state acceptance | policy-admitted rendered object set and exact revision | YAML parsed successfully |
| cluster convergence | observed UIDs/generations/specs/status plus controller reconciliation | Git updated or sync requested |
| workload readiness | source-native readiness and dependency checks | controller process is healthy |
| traffic exposure | measured cohort membership and route weights | requested canary weight |
| technical effect | independent SLO/error/latency/resource observations for the cohort | one controller-owned metric |
| user/business effect | protected synthetic journey and business invariant | pod health or HTTP 200 alone |
| data/effect safety | schema/backfill reconciliation and external-side-effect accounting | application rollback |
| recovery | restored invariant, reconciled desired state and recurrence window | rollback command accepted |

## Fault and red-herring population

1. Mutable image tag moves after approval but before promotion.
2. Registry returns a digest with missing or invalid provenance.
3. Git desired state is correct while a second writer continuously changes one field.
4. Controller cache is stale; UI and API disagree about live image or health.
5. Prune targets a still-required object because ownership metadata or rendered output is wrong.
6. Hook Job succeeds but performs only half of an external side effect; retry duplicates it.
7. Controller restarts between apply and status persistence; event delivery is duplicated.
8. One tenant receives canary traffic outside the declared cohort.
9. Technical metrics are green while a protected checkout/user journey fails.
10. A low-volume but high-severity business invariant regresses beneath aggregate thresholds.
11. Metric provider is stale, absent, sampled, permission-denied or returns `NaN`/inconclusive.
12. Rollout aborts after an irreversible database contract step.
13. Git revert restores manifests but cannot downgrade schema or compensate notifications.
14. Target cluster disappears while finalizers or post-delete hooks remain.
15. CRD/controller upgrade changes health, diff or ownership behavior mid-promotion.
16. A manual approval bypasses a verification stage or applies to stale freight/revision.
17. Shared controller credentials permit a cross-namespace or cross-tenant mutation.
18. Git, registry or telemetry is unavailable long enough to cross retry/deadline boundaries.
19. Emergency kubectl mutation repairs service temporarily but creates persistent GitOps conflict.
20. Rollback appears healthy, then the same failure recurs inside the declared observation window.

Red herrings include correlated database latency, unrelated noisy-neighbor CPU, an old failed rollout, benign replica churn and one intentionally pessimistic but non-causal metric.

## Required behavior

The assembly must:

1. identify the exact approved commit, artifact digest, desired-state revision, target and applying identity;
2. preview/render and policy-check the bounded object set without production credentials in the producer;
3. reject mutable, unsigned, stale, wrong-target and unapproved inputs;
4. serialize or reject duplicate writers for every mutable field;
5. reconcile actual state and expose stale/unknown observations rather than laundering them into green;
6. begin with a complete bounded cohort and prove actual—not requested—traffic assignment;
7. distinguish success, failure, error, missing, stale and inconclusive observations;
8. pause or abort before the declared blast-radius and irreversibility limits;
9. execute only a pre-authorized rollback, compensation or forward-repair path;
10. verify source-native, user, business, data and external-effect postconditions independently;
11. preserve a correlated immutable evidence envelope and unresolved gaps;
12. keep the incident/change open through the recurrence window.

## Metrics

- unauthorized or wrong-target mutations;
- artifact/revision substitution rate;
- controller disagreement and stale-green rate;
- time from bad exposure to detection, halt and safe state;
- maximum affected tenant/workflow/request population;
- false-promote and false-abort rates by fault class;
- duplicate/missing external effects;
- schema/backfill and data-reconciliation completeness;
- rollback/compensation success and time to restored business invariant;
- recurrence within 24 hours, 7 days and the domain-specific window;
- audit-envelope field completeness and source-native correlation coverage;
- human interventions, specialist minutes and ambiguous-state minutes;
- control-plane compute, storage, telemetry and operator cost;
- number of controllers, privileged identities, CRDs, queues and manual reconciliation paths.

## Fatal gates

Any of the following disqualifies production authority for the tested class:

- unsigned or different artifact reaches the target;
- a tenant/resource outside the authorized target is mutated;
- missing/stale/error verification is treated as pass;
- two active controllers own the same field without an enforced arbitration contract;
- actual canary exposure exceeds its bound;
- promotion proceeds while the protected user/business oracle fails;
- rollback claim omits irreversible data or external effects;
- controller/UI green contradicts source-native state without an explicit unknown/alert;
- the assembly cannot reconstruct who approved, who applied, what changed and what effect was observed;
- recovery depends exclusively on the failed control plane.

## Human authority

Humans retain production policy, exception and risk acceptance; irreversible schema/data approval; incident command; emergency break glass; disputed business-truth resolution; recovery declaration; and authority transfer to a larger autonomous class. Repeated fixture success may reduce per-change review for a narrowly named reversible class, but does not transfer authority to identity, data destruction, secrets, cluster control planes or unrelated services.

## Interpretation

The benchmark qualifies an assembly for a named change class, target, identity, artifact path and observation contract. It does not prove that the upstream coding factory wrote correct software, that Kubernetes health equals user value, that a Git revert is a transaction rollback, or that the same assembly can safely operate every application.