Chaos Testing

Deterministic failure-mode testing framework for the AgenC runtime's replay and storage subsystems.

Source: runtime/src/eval/chaos-matrix.ts

Overview

The Chaos Testing framework provides a structured approach to validating how the runtime behaves under adverse conditions. Rather than relying on random fault injection, it defines a Chaos Scenario Matrix -- a catalog of named failure scenarios, each with a known trigger, an expected anomaly, and a classification that determines whether the outcome is fully reproducible or statistically bounded.

Every scenario targets one of the runtime's critical subsystems: replay comparison, persistent storage, state transitions, or network connectivity. Running the full matrix produces a pass/fail report that maps directly to the anomaly codes used by the replay engine and alerting pipeline.

Scenario Structure

Each entry in the chaos matrix implements the ChaosScenario interface:

interface ChaosScenario {
  id: string;                     // Namespaced identifier, e.g. "chaos.store_write_failure"
  category: ChaosScenarioCategory;
  trigger: string;                // Human-readable description of what causes this scenario
  expectedAnomaly: {
    code?: ReplayAnomalyCode;     // Expected anomaly code (replay scenarios)
    alertCode?: string;           // Expected alert code (non-replay scenarios)
    severity: 'error' | 'warning' | 'info';
    kind: ReplayAlertKind;
  };
  classification: 'deterministic' | 'probabilistic';
  fixture: string;                // Path to fixture data used by the test
}

Field Reference

Classification Model

Scenario Categories

`replay` -- Replay Comparison Anomalies

Tests that validate the replay comparator's ability to detect divergences between on-chain event projections and locally recorded traces.

`chaos.comparator.hash_mismatch` -- The most common replay failure. The fixture contains a local trace whose content hash diverges from the projected on-chain hash. The comparator must detect this and emit an error-severity anomaly with the corresponding ReplayAnomalyCode.

`chaos.comparator.missing_event` -- Simulates a gap in the local trace. An event recorded on-chain has no corresponding entry in the local replay log. This typically indicates a dropped write or a subscriber lag.

`chaos.comparator.unexpected_event` -- The inverse of missing_event. The local trace contains an event that does not appear in the on-chain projection. Classified as warning because the extra event may be an optimistic pre-commit that was later rolled back.

`chaos.comparator.type_mismatch` -- The local trace and on-chain projection both contain an event at the same index, but the event types differ. This is a corruption indicator and is treated as an error.

`store` -- Storage Failures

Tests that validate the runtime's behavior when the underlying storage layer fails or returns corrupt data.

`chaos.store.write_failure` -- The fixture configures a storage backend that rejects writes after a specific byte offset. The runtime must detect the incomplete write, avoid committing partial state, and surface the failure through the alerting pipeline.

`chaos.store.read_corruption` -- The fixture provides a storage backend that returns altered bytes on read. The runtime must detect the integrity violation (typically via checksum mismatch) and refuse to use the corrupted data.

`transition` -- State Machine Violations

Tests that validate the runtime's state machine enforcement layer.

`chaos.transition.impossible_state` -- The fixture forces a state transition that violates the protocol's state machine (e.g., moving a task from Completed back to Claimed). The runtime must reject the transition and emit an error-level alert.

`chaos.transition.timing_violation` -- The fixture delivers events with timestamps that violate the expected ordering constraint (e.g., a completion event timestamped before the corresponding claim event). Classified as warning because clock skew can cause benign ordering anomalies in production.

`network` -- Network Connectivity Failures

Tests that validate the runtime's behavior when network operations fail.

Network scenarios are classified as probabilistic because they depend on timing and I/O scheduling.

`partial_write` -- Incomplete Write Operations

Tests that validate recovery from writes that succeed partially before failing.

Partial-write scenarios verify that the runtime can detect and recover from incomplete persistence operations without data loss or silent corruption.

Running Chaos Tests

Command Line

# Run all chaos tests
npm run test:chaos

# Run chaos tests via grep filter
npm run test -- --grep chaos

# Run a specific scenario category
npm run test -- --grep "chaos.comparator"
npm run test -- --grep "chaos.store"
npm run test -- --grep "chaos.transition"

# Run with verbose output
npm run test:chaos -- --reporter verbose

Programmatic Execution

import { chaosMatrix } from '@agenc/runtime/eval/chaos-matrix';

// Iterate all scenarios
for (const scenario of chaosMatrix) {
  console.log(`${scenario.id} [${scenario.classification}] -- ${scenario.trigger}`);
}

// Filter by category
const replayScenarios = chaosMatrix.filter(s => s.category === 'replay');
const deterministicOnly = chaosMatrix.filter(s => s.classification === 'deterministic');

Writing a Chaos Test

Each chaos test loads a fixture, executes the trigger condition, and asserts that the runtime produces the expected anomaly:

import { describe, it, expect } from 'vitest';
import { chaosMatrix } from '../eval/chaos-matrix.js';
import { ReplayComparator } from '../eval/replay-comparator.js';

describe('chaos.comparator', () => {
  const scenarios = chaosMatrix.filter(s => s.category === 'replay');

  for (const scenario of scenarios) {
    it(scenario.id, async () => {
      // Load fixture
      const fixture = await loadFixture(scenario.fixture);

      // Execute trigger
      const comparator = new ReplayComparator(fixture.config);
      const result = await comparator.compare(fixture.localTrace, fixture.onChainProjection);

      // Assert expected anomaly
      expect(result.anomalies).toContainEqual(
        expect.objectContaining({
          code: scenario.expectedAnomaly.code,
          severity: scenario.expectedAnomaly.severity,
        })
      );
    });
  }
});

CI Integration

Chaos tests run as a dedicated step in the CI pipeline, separate from unit and integration tests.

Pipeline Configuration

chaos_tests:
  runs-on: ubuntu-latest
  needs: [unit_tests]
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: '20'
    - name: Install dependencies
      run: npm ci
    - name: Run chaos tests
      run: npm run test:chaos
      env:
        CHAOS_FAIL_THRESHOLD: 0.05

Key CI Behaviors

Environment Variables

Adding a New Scenario

1. Define the fixture. Create a JSON file in fixtures/chaos/ that contains the data needed to trigger the failure condition.

2. Add the scenario to the matrix. Append a new entry to the chaosMatrix array in runtime/src/eval/chaos-matrix.ts:

{
  id: 'chaos.store.index_corruption',
  category: 'store',
  trigger: 'Storage index points to wrong offset after compaction',
  expectedAnomaly: {
    alertCode: 'STORE_INDEX_CORRUPT',
    severity: 'error',
    kind: ReplayAlertKind.StorageIntegrity,
  },
  classification: 'deterministic',
  fixture: './fixtures/chaos/store-index-corruption.json',
}

3. Write the test. If the new scenario falls under an existing category, the category-level test loop picks it up automatically. If it introduces a new category, create a new test file following the pattern in runtime/src/eval/__tests__/.

4. Validate locally. Run the full chaos suite and confirm the new scenario passes:

npm run test:chaos -- --reporter verbose

5. Update the threshold (if probabilistic). If the new scenario is probabilistic, verify that the CI fail-rate threshold still holds with the expanded matrix.

Relationship to Other Test Layers

The chaos testing framework occupies a specific position in the overall test strategy:

Chaos tests complement fuzz tests: fuzz tests explore the input space of on-chain instructions to find invariant violations, while chaos tests exercise known failure modes in the off-chain runtime to verify that detection and recovery mechanisms work correctly.

Troubleshooting

Deterministic scenario fails intermittently

A deterministic scenario should never fail intermittently. If it does:

Probabilistic scenario always fails

Fixture not found

Error: ENOENT: no such file or directory, open './fixtures/chaos/...'

CI threshold exceeded

If the CI step fails with a threshold violation:

1. Review the failure report artifact to identify which probabilistic scenarios failed.
2. Determine whether the failures indicate a real regression or environmental noise (e.g., CI runner I/O latency).
3. If the failures are environmental, consider increasing CHAOS_FAIL_THRESHOLD temporarily and filing a follow-up issue to stabilize the scenario.
4. If the failures indicate a regression, fix the underlying issue before merging.