Engineering Robust
Multi-Agent Coding Pipelines

1. Scope, Definitions, and Assurance Levels

Before examining failure modes and mechanical controls, we establish operational boundaries, define terminology, and outline a graduated maturity model for pipeline assurance.

1.1 Definition of Mechanical Controls

In an agentic assurance context, we define a mechanical control as:

Mechanical controls are not assumed to be infallible or capable of discovering unstated intent; rather, their execution is independent of the agent under evaluation.

1.2 Operational Scope

This architecture addresses repository-level autonomous software modification wherein one or more LLM agents ingest source code, formulate technical plans, modify designated implementation files, execute approved tools, and submit candidate patches. The architecture operates under explicit boundary conditions:

• In Scope: Guardrails against semantic intent drift, test modification / evaluation gaming, stale-state workspace clobbering, secret extraction into guest environments, unauthorized network egress, and unreviewed production merges.
• Out of Scope: Formal verification of arbitrary algorithmic correctness, real-time protection against fully privileged bare-metal host compromise, or automated discovery of unstated human business requirements.

1.3 Actor and Trust Zone Model

To eliminate ambiguity between agent capability and system authority, the architecture defines six operational entities across distinct trust boundaries:

1.4 Agentic Assurance Maturity Levels

Organizations adopt mechanical controls progressively. We define six assurance levels to structure deployment milestones:

2. Why Prompt Guidance Is Not Enforcement

Natural-language instructions improve agent behavior, but they cannot establish an independently enforceable boundary. When an agent receives an instruction in its prompt, that same agent is responsible for interpreting the rule, evaluating edge cases, and asserting whether it complied. In safety- or integrity-sensitive software engineering pipelines, this conflates guidance with enforcement.

2.1 Statistical and Contextual Limits of Prompts

Empirical research identifies fundamental constraints in prompt-only governance:

• Context Degradation and Retrieval Attenuation: In evaluations of long-context language models, performance can be position-sensitive, with relevant information located in the middle of long contexts receiving less effective use in the study's evaluated tasks (Lost in the Middle, Liu et al., 2024). Placing critical architectural constraints or security rules solely in prompt headers or dynamic memory files does not guarantee attention during long, multi-turn tool-call trajectories.
• Inconsistent Instruction Hierarchies: Multi-agent systems frequently rely on system/user prompt separation to enforce privileged constraints. However, empirical evaluations have found that several tested models do not consistently prioritize privileged instructions when formatting conflicts or counter-framing are introduced in evaluated settings (Control Illusion, Geng et al., arXiv:2502.15851).
• Context Pollution and Communication Topologies: Accumulating past post-mortems and negative instructions in dynamic memory files (e.g., MEMORIES.md) inflates token costs and increases latency. In one controlled study, naïve full-broadcast synchronization increased hallucination in a multi-agent travel-planning setting, while the same effect did not replicate in the study's software-planning tasks (Rodrigues, arXiv:2606.21666). The result supports treating communication topology and shared-state protocol design as context-dependent engineering variables rather than assuming broadcast coordination is uniformly beneficial. Structured protocols like SEMAP (Mao et al., arXiv:2510.12120) organize state and report failure reductions in evaluated software engineering tasks.

2.2 The Three Orthogonal Pipeline Properties

To avoid conflating mechanisms, the architecture distinguishes three related but analytically separate properties:

• Safety & Security (Containment): Preventing unauthorized data exfiltration, credential leakage, arbitrary network access, or lateral host movement (enforced via microVM sandboxes, host credential proxies, and egress filters).
• Integrity & Correctness (Verification): Ensuring proposed changes conform to interface contracts, satisfy independent specifications, resist mutation, and pass type and build gates executed in a pinned, reproducible environment (enforced via schemas, compilers, and immutable test suites).
• Coordination Consistency (State Management): Preventing concurrent workspace overwrites, stale interface assumptions, and unmanaged merge conflicts (enforced via isolated worktrees, dependency-aware write checks, and merge locks).

3. Documented Empirical Failure Modes

Engineering a robust control plane requires cataloging the specific failure modes observed in autonomous coding pipelines and mapping each to an empirical evidence base.

• Intent Divergence & Semantic Drift: The preprint Semantic Consensus (Acharya, arXiv:2604.16339) identifies Semantic Intent Divergence as a failure mode in which cooperating agents develop inconsistent interpretations of shared objectives under siloed context and absent process models, proposing process-aware conflict detection and governance mechanisms. Natural language agreements lack machine-checkable boundary definitions (CodeCtx, Baker, 2026).
• Evaluation Gaming and Test-Oracle Uncertainty: Automated test execution (exit 0) is bounded evidence, not absolute proof of correctness. Empirical audits reveal dual-directional oracle failures:
  • False Acceptance (Weak Oracles): In an empirical audit of SWE-bench Verified tasks, Rajan (2026) found that 28.5% of evaluated tasks accepted Docker-verified patches that were functionally defective due to incomplete assertion coverage in the evaluation harness.
  • False Rejection & Ambiguity: Conversely, in an audit of 138 difficult failures in SWE-bench Verified, OpenAI (2026) reported that 59.4% of evaluated failures suffered from flawed reference tests or underspecified descriptions that rejected valid candidate implementations.
• Same-Agent TDD Pathologies: When an executing agent generates both test fixtures and implementation code, it can produce tautological or weak assertions; practitioner reporting has documented examples of this failure mode (Autonoma, 2025). In a practitioner experiment, Böckeler (2024) reported that single-agent TDD loops yielded negligible mutation testing improvements because agents frequently skipped or simulated the initial failing (red) state or tailored assertions to preexisting implementation code.
• Stale-State Coordination Failures: While Git worktrees provide separate working directories, they do not prevent agents from committing changes based on obsolete interface assumptions. As evaluated in STORM (Liu et al., 2026), state-aware coordination can mediate writes made from stale observed state. In a production case study on multi-agent pipelines, Bhaumik (2024) reported that an uncoordinated 500ms cache read triggered cascade errors across collaborating agents.
• Boundary Escape & Credential Disclosure: Coding agents granted unconstrained shell access can leak environment variables, query cloud metadata endpoints (e.g., 169.254.169.254), or invoke billable third-party APIs during test execution instead of using local mocks.

4. Reference Architecture: The Three-Plane Model

To systematically address these failure modes, we present a vendor-neutral reference architecture structured into three decoupled planes governed by a Trusted Computing Base (TCB).

┌─────────────────────────────────────────────────────────────────────────────┐
│                              HUMAN AUTHORITY                                │
│          • Contract Approval                • Production Merge Sign-Off     │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │
┌──────────────────────────────────────▼──────────────────────────────────────┐
│                            CONTROL PLANE (TCB)                              │
│  • Task Manifest Registry            • Worktree / Runner Provisioner        │
│  • Filepath & Tool Policy Engine     • Host-Side Credential Proxy           │
│  • Egress Firewall Controller        • Immutable Attestation & Audit Log    │
└───────────────────┬─────────────────────────────────────┬───────────────────┘
                    │                                     │
         Frozen Spec & Test Bundle             Immutable Evaluation Attestation
                    │                                     │
┌───────────────────▼───────────────┐     ┌───────────────▼───────────────────┐
│          PLANNING PLANE           │     │         EVALUATION PLANE          │
│  • OpenAPI / JSON Schema Contract │     │  • Read-Only Test & Fixture Suite │
│  • Independent TDD Authoring      │     │  • Pinned Compiler / Linter AST   │
│  • Risk & Scope Manifest          │     │  • Mutation Testing Engine        │
└───────────────────┬───────────────┘     └───────────────▲───────────────────┘
                    │                                     │
           Signed Task Manifest                    Candidate Patch
                    │                                     │
┌───────────────────▼─────────────────────────────────────┴───────────────────┐
│                           EXECUTION PLANE (GUEST)                           │
│  • Isolated Git Worktree / MicroVM Sandbox (Least Privilege)                │
│  • Write-Allowlisted Implementation Surface (Evaluation Surface Immutable)  │
│  • Default-Deny Outbound Egress (Host Proxy Mediated)                       │
└─────────────────────────────────────────────────────────────────────────────┘
          

4.1 Control-Objective Matrix

The architecture maps explicit control objectives to required mechanical primitives, generated evidence artifacts, and acknowledged residual risks:

4.2 Control-Plane Specifications

To make the architecture actionable for platform engineering teams, we define the concrete data structures and policies enforced by the control plane.

Specification 1: Signed Task Manifest Schema

Before an execution agent is spawned, the control plane generates an immutable, signed manifest binding the agent to an approved scope:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentTaskManifest",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "manifest_schema_version", "policy_engine_version", "task_id",
    "repository_url", "base_commit_sha", "contract_hash", "evaluation",
    "writable_path_patterns", "denied_path_patterns", "egress_policy",
    "budget_limits", "approver", "signature"
  ],
  "properties": {
    "manifest_schema_version": { "type": "string", "const": "1.0" },
    "policy_engine_version": { "type": "string", "minLength": 1 },
    "task_id": { "type": "string", "format": "uuid" },
    "repository_url": { "type": "string", "format": "uri" },
    "base_commit_sha": { "type": "string", "pattern": "^[a-f0-9]{40}$" },
    "contract_hash": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
    "evaluation": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "bundle_digest",
        "runner_image_digest",
        "acceptance_command",
        "acceptance_command_digest",
        "environment_allowlist"
      ],
      "properties": {
        "bundle_digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
        "runner_image_digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
        "acceptance_command": {
          "type": "array",
          "minItems": 1,
          "items": { "type": "string", "minLength": 1 }
        },
        "acceptance_command_digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
        "environment_allowlist": {
          "type": "array",
          "items": { "type": "string", "pattern": "^[A-Z_][A-Z0-9_]*$" },
          "uniqueItems": true
        }
      }
    },
    "writable_path_patterns": {
      "type": "array",
      "minItems": 1,
      "items": { "type": "string", "minLength": 1 }
    },
    "denied_path_patterns": {
      "type": "array",
      "minItems": 1,
      "items": { "type": "string", "minLength": 1 }
    },
    "egress_policy": {
      "type": "object",
      "additionalProperties": false,
      "required": ["policy_id", "policy_digest"],
      "properties": {
        "policy_id": { "type": "string", "minLength": 1 },
        "policy_digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" }
      }
    },
    "budget_limits": {
      "type": "object",
      "additionalProperties": false,
      "required": ["max_retries", "max_wallclock_seconds", "max_tokens"],
      "properties": {
        "max_retries": { "type": "integer", "minimum": 0, "maximum": 5 },
        "max_wallclock_seconds": { "type": "integer", "minimum": 1, "maximum": 1800 },
        "max_tokens": { "type": "integer", "minimum": 1, "maximum": 500000 }
      }
    },
    "approver": {
      "type": "object",
      "additionalProperties": false,
      "required": ["identity", "timestamp"],
      "properties": {
        "identity": { "type": "string" },
        "timestamp": { "type": "string", "format": "date-time" }
      }
    },
    "signature": {
      "type": "object",
      "additionalProperties": false,
      "required": ["algorithm", "key_id", "signed_at", "payload_digest", "value"],
      "properties": {
        "algorithm": { "type": "string", "enum": ["Ed25519", "ECDSA_P256", "RSASSA_PSS_SHA_256"] },
        "key_id": { "type": "string" },
        "signed_at": { "type": "string", "format": "date-time" },
        "payload_digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
        "value": { "type": "string" }
      }
    }
  }
}

Signer, Verifier & Time Trust Boundary: The execution agent must not possess a manifest-signing key or the ability to register a trusted signer. Manifest signatures are generated by a planning authority or control-plane service identity. Admission control and CI evaluate signature and token validity against a control-plane trusted time source; the execution guest's system clock is not authoritative for manifest expiry or attestation validity. Admission control verifies the manifest signature, payload digest, expiration timestamp, contract hash, and evaluation-bundle hash before provisioning the execution environment. CI repeats those checks before producing an acceptance attestation.

Evaluator Canonicalization & Environment Allowlisting: acceptance_command_digest is computed over the RFC 8785/JCS-canonicalized evaluator descriptor, including the argument array, working directory, declared environment allowlist, and runner-image digest; the execution agent cannot modify any component of that descriptor. The evaluator inherits no ambient environment variables except those explicitly allowlisted and recorded in the evaluator descriptor.

Path Authorization & Normalization Semantics: Path authorization is evaluated on canonicalized, repository-relative paths after symlink resolution. Deny rules strictly override allow rules. The control plane rejects path traversal (..), symlink escape, alternate data streams, case-folding ambiguities, and any write whose resolved destination falls outside the approved workspace root.

Specification 2: Immutable Evaluation Surface

The evaluation surface comprises all files and configurations that determine whether candidate code is accepted. Independently owned means that the executor lacks write authority over the test bundle, evaluator configuration, test command, dependency-resolution inputs, and acceptance criteria for its assigned task. The full evaluation surface must be cryptographically bound to the task manifest and immutable to the execution agent. Where evaluator artifacts are exposed within the guest, they must be mounted read-only; CI workflow definitions, acceptance commands, runner configuration, and protected-branch policies must be enforced outside the executor's write authority:

• Test Suites & Fixtures: The approved test suite and fixtures for the task, including relevant pre-existing regression tests, independently authored task tests, integration fixtures, and end-to-end scenarios selected by the planning/QA authority.
• Dependency Resolution: Lockfiles (e.g., package-lock.json, poetry.lock) and package configuration manifests.
• Build & CI Scripts: Pipeline definitions (e.g., .github/workflows/), linter configs, compiler flags, and test runner arguments.
• Evaluation Runner Image: Pinned OCI digest of the container running the evaluation suite.

Specification 3: Sandbox Egress & Credential Containment Policy

Execution environments operate with default-deny network access. Necessary external communication is mediated through a host-side proxy. Raw credentials are not mounted in the guest; the proxy injects or exchanges scoped credentials only on the host side after policy evaluation. Docker Sandboxes documents a microVM-based execution model and configurable credential/network controls (Docker Inc., 2026); organizations should validate effective policy posture at deployment time and record the applied policy digest:

version: "1.0"
metadata:
  policy_id: "pol-sbx-strict-v2"
  policy_digest: "sha256:7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069"
  enforcement_tier: "L3_CAPABILITY_CONTAINED"
  issued_at: "2026-09-01T00:00:00Z"
  expires_at: "2026-09-01T01:00:00Z"

filesystem:
  writable_paths:
    - "src/modules/billing/**"
    - "src/utils/math/**"
  read_only_mounts:
    - host_path: "/var/bundles/eval-task-4819"
      guest_path: "/workspace/tests"
  denied_paths:
    - ".git/**"
    - ".github/**"
    - "package.json"
    - "package-lock.json"

network:
  default_egress: "deny"
  dns:
    resolver: "host-controlled"
    allow_only_listed_hosts: true
    block_private_ranges: true
    block_link_local_ranges: true
    block_metadata_endpoints: true  # Blocks 169.254.169.254 / Cloud IMDS
  ingress: "deny"
  allowlist:
    - host: "registry.npmjs.org"
      port: 443
      protocol: "https"
      allowed_methods: ["GET", "HEAD"]
    - host: "pypi.org"
      port: 443
      protocol: "https"
      allowed_methods: ["GET", "HEAD"]

resources:
  cpu_cores: 2
  memory_mb: 4096
  disk_mb: 10240
  pids_max: 256
  max_wallclock_seconds: 1800
  max_processes: 128

output_controls:
  max_stdout_bytes: 1048576
  max_stderr_bytes: 1048576
  redact_secret_patterns: true
  permitted_artifact_exports:
    - "patch.diff"
    - "test-results.junit.xml"
    - "attestation.json"

logging:
  network_decisions: true
  file_write_attempts: true
  tool_invocations: true
  export_to_control_plane: true

security:
  allow_privilege_escalation: false
  raw_sockets: false
  host_pid_namespace: false
  host_network_namespace: false
  host_ipc_namespace: false
  host_mount_namespace: false
  credential_mediation:
    mode: "host_proxy"
    raw_credentials_mounted_in_guest: false
    secret_references:
      - proxy_token_id: "npm-read-token"
        scope: "read:packages"
        rate_limit_per_minute: 60

Egress Filtering, Destination Validation & IPv6 Handling: Destination validation occurs at the host-side proxy or network-policy enforcement point, after DNS resolution and before connection establishment. For HTTP(S), the proxy must re-evaluate each redirect target, enforce the allowlist against the resolved destination and SNI/Host metadata, deny IP-literal destinations unless explicitly authorized, and prevent credentials from being forwarded to a changed origin. The egress controller must evaluate the final resolved destination on every connection, normalize IPv4-mapped IPv6 addresses (e.g., ::ffff:169.254.169.254), block loopback (127.0.0.0/8, ::1/128), link-local (fe80::/10), cloud metadata (169.254.169.254), and prohibited private address ranges, prevent DNS rebinding, and deny direct IP connections unless explicitly authorized.

Build-Time vs. Runtime Dependency Caching: For high-assurance profiles, dependency resolution should occur before execution from a verified internal artifact mirror or content-addressed cache. Runtime registry access is an exception path and must be constrained by lockfile digests, package integrity metadata, provenance checks, and policy-logged retrieval.

Verifiable Attestation Content Model: To ensure evaluation evidence is machine-auditable and tamper-evident, the CI engine emits a structured attestation binding: (1) source repository URL and base commit SHA, (2) candidate patch commit SHA, (3) task-manifest digest, (4) control-policy digest, (5) evaluation-bundle digest, (6) runner image OCI digest, (7) exact test execution command, (8) test, static analysis, and mutation results, (9) builder identity and timestamp, (10) cryptographic signature, and (11) exception/waiver ID if applicable.

Specification 4: Dependency-Change Exception Policy

While standard execution tasks deny writes to package manifests (e.g., package.json, poetry.lock), real-world development periodically requires adding or upgrading libraries. To maintain evaluation integrity, dependency changes are never permitted inside standard implementation tasks. Instead, they follow a dedicated exception path:

5. Operational Pipeline and Assurance State Machine

The reference architecture translates governance rules into an automated, multi-stage state machine with formal transition criteria and failure handling.

5.1 The Four Mechanical Assurance Rules

The state machine enforces four core operational rules:

1. 1. Treat Automated Success as Bounded Evidence A successful build, type check, or test pass (exit 0) confirms conformance only to the specific assertions executed. It does not establish full behavioral correctness. Pipeline gates must require defense-in-depth verification (static analysis, schema conformance, mutation testing, and review) rather than relying on binary test results as a comprehensive oracle.
2. 2. Preserve Total Evaluator Independence The full evaluation surface must be cryptographically bound to the task manifest and immutable to the execution agent. Where evaluator artifacts are exposed within the guest, they must be mounted read-only; CI workflow definitions, acceptance commands, runner configuration, and protected-branch policies must be enforced outside the executor's write authority.
3. 3. Enforce Layered Adequacy Checks In safety-critical or core architectural modules, the CI evaluator must inject automated mutation testing (e.g., Mutmut, Stryker) to verify that the test suite detects intentional semantic mutations. For general codebases, mutation testing is applied on a calibrated risk-tier basis during PR evaluation or nightly regression runs.
4. 4. Enforce Bounded Autonomy with Evidence-Preserving Quarantine When an agent exceeds its retry budget, triggers a policy violation, or fails evaluation gates, the control plane immediately revokes execution capabilities and quarantines the branch. Crucially, the system does not discard execution artifacts; it archives the task manifest, tool-call audit logs, patch diffs, test output, and container digests for post-mortem analysis and policy refinement.

6. Deployment Profiles and Residual Risk Analysis

Real-world adoption balances security assurance with operational complexity. We outline two implementation profiles and document unmitigated residual risks.

6.1 Residual Risk and Defense-in-Depth Mitigations

A rigorous reference architecture must explicitly identify residual risks that fall outside automated control-plane guarantees:

7. Primary Sources and Evidence Bibliography

Appendix: Tool and Vendor Mapping to Architectural Roles

To maintain architectural portability, the three-plane model is vendor-neutral. Commercial and open-source tools function as execution hosts configured by pipeline policy, rather than standalone governance systems: