# AI-SRE control loops and the RCA ladder — Pass 109

Date: 2026-08-25  
Status: source-qualified; no product, cluster, tenant, incident or production action was run.  
Ontology: `0.79.0` (existing roles; no new enum required)

## Executive answer

The systems do not occupy one “root-cause-analysis” tier. They occupy different points on a ladder:

1. **Context retrieval** — collects logs, metrics, traces, manifests, alerts or history.
2. **Hypothesis formation** — proposes a likely cause and next steps.
3. **Discriminating diagnosis** — performs or requests a test whose result could falsify a competing cause.
4. **Bounded intervention** — changes a declared target through a typed or policy-bound workflow.
5. **Independent effect closure** — verifies the target, user, data, security and business outcomes from outside the agent’s own evidence path.
6. **Recurrence closure** — proves the failure does not return across the declared window and updates prevention controls.

Most “AI SRE” products reach level 2. OpenSRE is unusually useful at level 3 as an evaluation and scenario harness. Kubernaut is the strongest source-visible level-4/partial-level-5 Kubernetes architecture in this pass, but its effectiveness monitor is still Kubernetes/Prometheus/AlertManager-local and has a fail-open assessment case. No reviewed factory, agent or SaaS product earns level 6 across an enterprise estate from public evidence.

## RCA placement by system family

| Family / examples | Highest defensible public RCA level | What it really does | Hard edge |
|---|---:|---|---|
| Code-producing factories: Stripe Minions, Uber Minion, DoorDash Flux, Spotify Honk, Vercel AI SDK Factory, GitHub Copilot cloud agents, Cursor, Devin, Factory, Warp | 1–2 | Uses repository, CI, issue and sometimes incident context to produce a patch, PR or workflow result | A good patch or green CI is not proof that the observed production failure was caused by the changed code |
| Modernization factories: Google MAT/Dual Run, IBM watsonx Code Assistant for Z, Meta SAPFix, OpenRewrite/Moderne, Mechanical Orchard Imogen | 2 for defects; 3 for equivalence tests | Localizes patterns, translates code, replays traffic or compares old/new behavior | Equivalence on observed workloads is not complete causal RCA, and does not own post-cutover operations |
| Evidence investigators: HolmesGPT, KubeRCA, Stakpak, Datadog Bits, Dynatrace Davis, New Relic Autopilot, Sentry Seer | 2; occasionally a bounded 3 when a runbook supplies the test | Correlates telemetry, asks tools for evidence, drafts a diagnosis and may propose a fix | Tool breadth and a convincing narrative do not establish a counterfactual or complete evidence population |
| Evaluation-first investigator: OpenSRE | 2–3 | Requires evidence, injects red herrings and evaluates synthetic/cloud-backed RCA scenarios | Its own README reports no published benchmark results; it is not a production mutation controller |
| Workflow/action systems: Robusta Classic, Botkube, PagerDuty/ServiceNow workflows | 1–2 for RCA; 4 for authored actions | Runs deterministic playbooks, sends ChatOps messages, or delegates to an approved workflow | Playbook completion, ticket closure or alert resolution is not causal proof or service recovery |
| Closed-loop Kubernetes controller: Kubernaut | 2–4; partial 5 | Signal → AI analysis → approval/policy → workflow execution → health/alert/metric/spec-hash assessment | Effectiveness is target-local; public code includes an alert-unavailable path that can still classify remediation as effective; no user/data/security/business oracle |
| Agent control planes: Sympozium, Kagent, Mastra, LangGraph, A2A/ACP/AG-UI | 0–1 by themselves | Schedules, isolates, delegates, transports and records agent work | Hosting an agent does not supply a causal model, target authority or independent verifier |
| Observability / trace organs: OTel, Coroot, SigNoz, OpenObserve, Tracetest | 1; Tracetest can reach bounded technical 3 | Makes signals searchable and can assert trace-level technical behavior | Telemetry completeness, sampling, propagation and business semantics remain external |

The level is a ceiling for the system as publicly implemented, not a claim about what a skilled operator could build around it.

## Source-qualified AIOps successors

### HolmesGPT 0.39.0 — strong investigator, optional executor

Pinned stable source: [HolmesGPT 0.39.0](https://github.com/HolmesGPT/holmesgpt/tree/0.39.0). The implementation is Python/Typer/FastAPI/Pydantic with LiteLLM, MCP, the Kubernetes client, Kopf and optional OpenTelemetry. Its CLI exposes `ask`, `investigate`, `generate`, `toolset` and `checks`; health-check CRDs can run alert- or schedule-triggered queries and persist a model rationale/status.

The safe default is meaningful: the Kubernetes addon is read-only and the project separates remediation into an optional MCP addon. That remediation surface can patch, delete, scale, roll out, cordon, drain, taint, label, annotate, run and exec, with cluster-scoped node permissions. `--bash-always-allow` exists as an explicit approval bypass and must be treated as a sandbox-only test setting.

**RCA verdict:** level 2, with a level-3 *opportunity* when a toolset or runbook asks a discriminating question. The health-check “pass” is still an LLM status/rationale over queried evidence; it is not an independent causal or business outcome. Scheduled health checks are ongoing observation, not staging reproduction.

### OpenSRE v2026.6.25 — the best current evaluation boundary

Pinned stable source: [OpenSRE v2026.6.25](https://github.com/Tracer-Cloud/opensre/tree/v2026.6.25). It is a Python 3.12+ public-alpha service with Kubernetes, EC2, CloudWatch, Lambda, ECS/Fargate and Flink scenario families, model-provider configuration, structured diagnosis and an investigation store. The E2E suite deliberately includes synthetic RCA cases, required evidence and red herrings.

Its remote API is API-key protected for investigation routes; health/version probes are intentionally exempt. The implementation can publish a report or GitLab MR note, but the inspected remote path does not constitute a general target mutation API. The repository’s benchmark section currently says that no benchmark results are published.

**RCA verdict:** level 2 in production, level 3 as a research/evaluation harness. It is valuable because it tests whether the answer used the right evidence and resisted a distractor. It does not make its diagnosis true merely because a scenario report was produced.

### Kubernaut v1.5.7 — strongest closed-loop shape, not yet causal authority

Pinned stable source: [Kubernaut v1.5.7](https://github.com/jordigilh/kubernaut/tree/v1.5.7). The Go/Kubernetes platform separates signal processing, AI analysis, approval, workflow execution, notification and effectiveness monitoring. Execution can be digest-pinned through Kubernetes Jobs, Tekton or Ansible paths, and the `RemediationRequest` state machine records approval, execution and verification phases.

The `EffectivenessAssessment` implementation checks four classes: target health, alert state/decay, Prometheus metrics and a pre/post spec hash. Those are useful target-native observations. They are not an independent user/data/security/business oracle. More importantly, the source tests an `alertScore=0` path as `Remediated` when the alert was not assessed, so “alert unavailable” cannot be silently treated as proof of recovery. The chart’s default single-replica services and disabled leader election also make high-availability claims conditional on explicit production hardening.

**RCA verdict:** levels 2–4, partial 5 for Kubernetes-local effect closure. It is the best candidate to put behind an Iron Reins effect oracle, not the oracle itself.

### Sympozium v0.9.5 — agent operating plane, not RCA

Pinned stable source: [Sympozium v0.9.5](https://github.com/sympozium-ai/sympozium/tree/v0.9.5). The Go controller reconciles `AgentRun` resources into Jobs/Pods, supports ensembles and delegation, NATS JetStream channels, optional memory, MCP gating, sandbox runtimes and OTel trace propagation. It is MIT-licensed in the pinned repository. The API is explicitly active-development software.

The policy defaults require care: empty allowed-model or namespace lists mean allow-all, tool-gating defaults to `allow`, and sandboxing is optional. AgentRun success/result, token usage, exit code and trace ID describe an agent execution; they do not prove the target service recovered.

**RCA verdict:** level 0–1 as a substrate. It can host HolmesGPT, OpenSRE or a purpose-built investigator, but it must not inherit their RCA or authority claims.

## The critical distinction

An RCA statement is only causal when it predicts an observable difference between competing explanations and the system performs that test against a controlled target. A useful record therefore looks like:

```text
signal → competing hypotheses → discriminating test → predicted observation
       → controlled intervention → native receipt → independent probes
       → recurrence window → prevention change
```

“The pod restarted,” “the alert cleared,” “the PR passed CI,” “the workflow completed,” “the model said fixed,” and “the spec hash matched” are evidence fragments. They are not the closure condition.

## Iron Reins placement

The recommended composition is deliberately plural:

`OTel/native evidence → OpenSRE-style discriminating test plan → HolmesGPT/KubeRCA investigation → Kubernaut or Robusta typed action → Tracetest/native readback → protected user/data/security/business probes → recurrence ledger`

Sympozium or Kagent can provide an execution substrate, but neither should be placed in the RCA-authority slot. One writer owns each mutable domain. The evaluator must be outside the executor’s evidence path, and “unknown” must remain a first-class result when telemetry is stale, sampled, incomplete or unavailable.

## Primary sources

- [HolmesGPT repository](https://github.com/HolmesGPT/holmesgpt/tree/0.39.0)
- [Holmes Kubernetes toolset](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes/)
- [Holmes Kubernetes remediation MCP](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp/)
- [Kubernaut repository](https://github.com/jordigilh/kubernaut/tree/v1.5.7)
- [Sympozium repository](https://github.com/sympozium-ai/sympozium/tree/v0.9.5)
- [Sympozium documentation](https://deploy.sympozium.ai/docs/getting-started/)
- [OpenSRE repository](https://github.com/Tracer-Cloud/opensre/tree/v2026.6.25)
- [Kubernetes AI agent benchmark](https://github.com/henrikrexed/k8s-ai-agent-benchmark)
- [OpenRCA benchmark](https://github.com/microsoft/OpenRCA)