Incident Investigation
The incident investigation subsystem provides deterministic reconstruction and analysis of on-chain task lifecycle events. It models incidents as structured cas
Incident Investigation
The incident investigation subsystem provides deterministic reconstruction and analysis of on-chain task lifecycle events. It models incidents as structured cases with canonical ordering, sealed evidence exports, and a typed query DSL for analyst workflows.
Source modules:
runtime/src/eval/incident-case.ts -- Case model and transition orderingruntime/src/eval/evidence-pack.ts -- Evidence export with integrity hashesruntime/src/eval/query-dsl.ts -- Analyst query language and canonical hashingIncident Case Model
An IncidentCase is a self-contained record that captures every state transition within a trace window. Schema version 1 guarantees deterministic reconstruction: given the same on-chain events, the same case structure is always produced.
IncidentCase
| Field | Type | Description |
|---|---|---|
schemaVersion | 1 | Fixed schema version for deterministic reconstruction |
caseId | string | Unique identifier for this incident case |
createdAtMs | number | Unix timestamp (milliseconds) when the case was created |
traceWindow | TraceWindow | Slot and timestamp boundaries for the investigation window |
transitions | IncidentTransition[] | Ordered list of state transitions observed in the window |
anomalies | IncidentAnomalyRef[] | References to detected anomalies |
actorMap | IncidentActor[] | All actors involved in the incident |
evidenceHashes | IncidentEvidenceHash[] | SHA-256 integrity hashes for each piece of evidence |
caseStatus | CaseStatus | Current status of the investigation |
taskIds | string[] | On-chain task PDAs associated with this case |
disputeIds | string[] | On-chain dispute PDAs associated with this case |
TraceWindow
The trace window defines the on-chain boundaries of the investigation.
interface TraceWindow {
fromSlot: number;
toSlot: number;
fromTimestampMs: number;
toTimestampMs: number;
}Both slot-based and timestamp-based boundaries are recorded. Slot range is authoritative for on-chain ordering; timestamps provide human-readable context.
CaseStatus
| Status | Description |
|---|---|
open | Case created, not yet assigned to an analyst |
investigating | Analyst actively reviewing transitions and evidence |
resolved | Root cause identified, resolution applied or documented |
archived | Case closed and moved to long-term storage |
type CaseStatus = 'open' | 'investigating' | 'resolved' | 'archived';IncidentTransition
Each transition records a single state change observed on-chain.
interface IncidentTransition {
seq: number; // Monotonic sequence number within the case
slot: number; // Solana slot where the transition occurred
timestampMs: number; // Unix timestamp in milliseconds
signature: string; // Transaction signature (base58)
eventName: string; // Protocol event name (e.g. "TaskCreated")
type: string; // Transition classification
pda: string; // Program-derived address of the affected account
fromState: string; // State before the transition
toState: string; // State after the transition
actorPubkey: string; // Public key of the actor who triggered the transition
metadata?: Recordstring, unknown>; // Optional event-specific data
}Canonical Transition Ordering
Transitions within a case are sorted using a deterministic multi-key comparator. This guarantees that two independent reconstructions of the same on-chain events produce an identical transition sequence.
Sort priority (highest to lowest):
| Priority | Field | Direction | Rationale |
|---|---|---|---|
| 1 | seq | ascending | Primary ordering from event ingestion |
| 2 | slot | ascending | On-chain slot provides causal ordering |
| 3 | timestampMs | ascending | Sub-slot ordering when available |
| 4 | signature | lexicographic | Deterministic tiebreaker for same-slot events |
| 5 | eventName | lexicographic | Secondary tiebreaker |
| 6 | type | lexicographic | Tertiary tiebreaker |
| 7 | pda | lexicographic | Final tiebreaker ensures total ordering |
This ordering is applied automatically when a case is constructed or when transitions are added.
Task State Machine
The investigation model tracks tasks through a defined state machine. Valid transitions:
discovered --> claimed --> completed
\-> failed
\-> disputed --> completed
\-> failed| From State | To State | Trigger |
|---|---|---|
discovered | claimed | Worker claims the task |
claimed | completed | Worker submits successful result |
claimed | failed | Execution fails or times out |
claimed | disputed | Any party initiates a dispute |
disputed | completed | Dispute resolved in favor of the worker |
disputed | failed | Dispute resolved against the worker |
Transitions that fall outside this state machine are flagged as anomalies.
IncidentActor
interface IncidentActor {
pubkey: string; // Base58-encoded public key
role: ActorRole; // Role within the incident
label?: string; // Optional human-readable label
}
type ActorRole = 'creator' | 'worker' | 'arbiter' | 'authority' | 'unknown';| Role | Description |
|---|---|
creator | Entity that created the task |
worker | Agent that claimed and executed the task |
arbiter | Dispute resolution voter |
authority | Protocol authority (e.g. program upgrade authority) |
unknown | Actor whose role could not be determined from on-chain data |
IncidentAnomalyRef
interface IncidentAnomalyRef {
anomalyId: string; // Unique anomaly identifier
code: string; // Machine-readable anomaly code
severity: string; // "low" | "medium" | "high" | "critical"
description: string; // Human-readable summary
transitionSeqs: number[]; // Sequence numbers of related transitions
}IncidentEvidenceHash
interface IncidentEvidenceHash {
label: string; // Descriptive label (e.g. "transition-log", "actor-map")
algorithm: 'sha256'; // Hash algorithm used
hash: string; // Hex-encoded hash digest
}Evidence Pack Export
The evidence pack system produces portable, verifiable exports of incident data. Each export is described by an EvidencePackManifest that binds the data to its query context and runtime environment.
EvidencePackManifest
| Field | Type | Description |
|---|---|---|
schemaVersion | number | Manifest schema version |
seed | string | Random seed used for reproducible export ordering |
queryHash | string | SHA-256 hash of the canonical query that produced this pack |
cursorRange | { from: string; to: string } | Cursor boundaries of the exported data |
runtimeVersion | string | Version of @agenc/runtime that generated the export |
schemaHash | string | Hash of the schema definition used during export |
toolFingerprint | string | Fingerprint of the export tool (for reproducibility audits) |
sealed | boolean | Whether the pack is sealed (immutable) |
createdAtMs | number | Unix timestamp (milliseconds) when the pack was created |
evidenceHashes | IncidentEvidenceHash[] | Integrity hashes for all evidence artifacts in the pack |
interface EvidencePackManifest {
schemaVersion: number;
seed: string;
queryHash: string;
cursorRange: { from: string; to: string };
runtimeVersion: string;
schemaHash: string;
toolFingerprint: string;
sealed: boolean;
createdAtMs: number;
evidenceHashes: IncidentEvidenceHash[];
}Sealed Mode
When an evidence pack is sealed, its contents are frozen and protected by SHA-256 integrity hashes. Sealed packs cannot be modified after creation -- any tampering invalidates the evidence hashes stored in the manifest.
Use the --sealed flag on the CLI to produce a sealed export:
agenc-runtime replay incident \
--task-pda <TASK_PDA> \
--from-slot <FROM_SLOT> \
--to-slot <TO_SLOT> \
--sealed \
--store-type sqlite \
--sqlite-path .agenc/replay-events.sqliteWhen sealed is true:
evidenceHashesWhen sealed is false:
RedactionPolicy
Before exporting, sensitive data can be stripped using a redaction policy.
interface RedactionPolicy {
stripFields: string[]; // Field paths to remove entirely (e.g. "metadata.apiKey")
redactPatterns: RegExp[]; // Regex patterns to replace with "[REDACTED]"
redactActors: string[]; // Base58 public keys to anonymize in the export
}| Field | Behavior |
|---|---|
stripFields | Named fields are deleted from all transitions and metadata before export |
redactPatterns | String values matching any pattern are replaced with [REDACTED] |
redactActors | Public keys are replaced with deterministic pseudonyms (e.g. actor-1, actor-2) while preserving referential integrity across the export |
Example:
const policy: RedactionPolicy = {
stripFields: ['metadata.rawPayload', 'metadata.apiKey'],
redactPatterns: [/Bearer\s+[A-Za-z0-9._-]+/g],
redactActors: [
'7xKXtg2CW87d97TXJSDpbD5jBkheTqA83TZRuJosgAsU',
],
};Analyst Query DSL
The query DSL provides a structured, typed interface for filtering incident data. Queries are canonicalized before execution to ensure that logically identical queries always produce the same hash -- enabling deterministic cache lookups and reproducible evidence packs.
QueryDSL
interface QueryDSL {
taskPda?: string; // Filter by task PDA (base58)
disputePda?: string; // Filter by dispute PDA (base58)
actorPubkey?: string; // Filter by actor public key (base58)
eventType?: string; // Filter by event type name
severity?: string; // Filter by anomaly severity
slotRange?: { // Filter by slot range
from: number;
to: number;
};
walletSet?: string[]; // Filter by set of wallet public keys (base58)
anomalyCodes?: string[]; // Filter by anomaly codes
}Actor Fields
When filtering by actorPubkey, the query matches against all actor roles recorded in the incident. The following actor field names are recognized in the on-chain event data:
| Actor Field | Description |
|---|---|
creator | Task or dispute creator |
worker | Task executor |
authority | Protocol authority |
voter | Dispute voter |
initiator | Dispute initiator |
defendant | Party being disputed |
recipient | Reward or transfer recipient |
updater | Entity that updated account state |
agent | General agent reference |
Wallet Validation
All public key fields (taskPda, disputePda, actorPubkey, and entries in walletSet) are validated as base58-encoded Solana public keys before query execution. Invalid keys cause the query to be rejected with a validation error.
// Valid: standard Solana base58 public key
const query: QueryDSL = {
actorPubkey: '7xKXtg2CW87d97TXJSDpbD5jBkheTqA83TZRuJosgAsU',
slotRange: { from: 280_000_000, to: 280_100_000 },
};
// Invalid: not a valid base58 public key -- will be rejected
const bad: QueryDSL = {
actorPubkey: 'not-a-valid-key',
};Canonical Query
Every QueryDSL is normalized into a CanonicalQuery before execution. Normalization ensures that two logically identical queries (e.g. differing only in field order or whitespace) produce the same hash.
interface CanonicalQuery {
canonical: string; // Canonical JSON string (sorted keys, no whitespace)
hash: string; // SHA-256 hex digest of the canonical string
dsl: QueryDSL; // The normalized DSL object
}Normalization rules:
- All fields are sorted alphabetically by key name
- Undefined/null fields are omitted
- Array values (e.g.
walletSet,anomalyCodes) are sorted lexicographically - The result is serialized as compact JSON (no whitespace)
- The SHA-256 hash is computed over the canonical JSON bytes
import { canonicalize } from '@agenc/runtime/eval';
const query: QueryDSL = {
walletSet: ['Bbb...', 'Aaa...'],
eventType: 'TaskCompleted',
slotRange: { from: 280_000_000, to: 280_100_000 },
};
const canonical = canonicalize(query);
// canonical.canonical = '{"eventType":"TaskCompleted","slotRange":{"from":280000000,"to":280100000},"walletSet":["Aaa...","Bbb..."]}'
// canonical.hash = "a3f2...c891" (SHA-256 of the above string)Two analysts running the same logical query on the same data will always get the same canonical.hash, which in turn produces the same queryHash in the evidence pack manifest.
Combining Filters
All filters in a QueryDSL are combined with logical AND. To retrieve transitions matching any of multiple criteria, issue separate queries and merge the results.
// All transitions for a specific task within a slot range
const query: QueryDSL = {
taskPda: '4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU',
slotRange: { from: 280_000_000, to: 280_100_000 },
};
// All high-severity anomalies involving a specific actor
const query2: QueryDSL = {
actorPubkey: '7xKXtg2CW87d97TXJSDpbD5jBkheTqA83TZRuJosgAsU',
severity: 'high',
};
// All transitions matching specific anomaly codes across a wallet set
const query3: QueryDSL = {
walletSet: [
'7xKXtg2CW87d97TXJSDpbD5jBkheTqA83TZRuJosgAsU',
'4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU',
],
anomalyCodes: ['MISSING_TRANSITION', 'UNEXPECTED_STATE'],
};End-to-End Workflow
A typical incident investigation follows these steps:
- Backfill -- Ingest on-chain events for the target slot window into a local store.
- Compare -- Run replay comparison to detect anomalies between expected and observed state.
- Build Case -- Construct an
IncidentCasewith transitions, actors, and anomaly references. - Query -- Use the query DSL to filter and examine specific transitions.
- Export -- Produce a sealed evidence pack for audit or dispute proceedings.
# 1. Backfill events
agenc-runtime replay backfill \
--to-slot 280100000 \
--rpc https://api.mainnet-beta.solana.com \
--store-type sqlite \
--sqlite-path .agenc/replay-events.sqlite
# 2. Compare against local trace
agenc-runtime replay compare \
--local-trace-path ./trace.json \
--task-pda 4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU \
--store-type sqlite \
--sqlite-path .agenc/replay-events.sqlite
# 3. Reconstruct incident (sealed export)
agenc-runtime replay incident \
--task-pda 4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU \
--from-slot 280000000 \
--to-slot 280100000 \
--sealed \
--store-type sqlite \
--sqlite-path .agenc/replay-events.sqlite