Failure Modes and Effects Analysis (FMEA) & Risk Assessment

1. FMEA Table

Scoring Criteria

RPN (Risk Priority Number) = Severity × Probability × Detection

1.1 DependencyGraph Failure Modes

1.2 CommitmentLedger Failure Modes

1.3 ProofDeferralManager Failure Modes

1.4 RollbackController Failure Modes

1.5 SpeculativeScheduler Failure Modes

1.6 On-Chain Failure Modes

Summary Statistics

2. Risk Matrix

Visual Risk Matrix (Severity vs Probability)

Risk levels: CRITICAL (Sev x Prob > 40) — immediate action. HIGH (20-40) — action within 1 sprint. MEDIUM (10-20) — monitor and plan. LOW (< 10) — accept or defer.

Risk Category Distribution

CRITICAL ZONE (Red)        HIGH ZONE (Orange)         MEDIUM ZONE (Yellow)
─────────────────────      ──────────────────         ────────────────────
• DG-002 (Stale data)      • DG-001 (Cycles)          • SS-001 (Wrong decision)
• RC-003 (Missed task)     • DG-004 (Race)            • SS-006 (Expiring claims)
• PDM-003 (Deadlock)       • CL-001 (Lost commit)     • PDM-004 (Timeout)
• RC-004 (Mid-exec)        • CL-004 (State viol.)     • CL-002 (Wrong stake)
• RC-001 (Incomplete)      • PDM-001 (Lost proof)     • SS-005 (Starvation)
                           • OC-002 (Reorg)           • RC-007 (Cleanup)
                           • SS-002 (OOM)             • OC-001 (Tx failure)

3. Critical Risks (Top 5)

3.1 RISK-001: Stale Dependency Data (DG-002)

RPN: 240 | Severity: 8 | Probability: 5 | Detection: 6

Description

The DependencyGraph contains outdated information about task dependencies or completion status. This causes downstream tasks to execute against an assumed state that doesn't match the actual committed state.

Impact

Likelihood Assessment

Mitigation Strategy

Residual Risk

After mitigations: RPN ~60 (Sev: 8, Prob: 2, Det: 4)

Owner

Team: Runtime Core

DRI: TBD (assign senior engineer)

Review Cadence: Weekly during implementation; Monthly post-launch

3.2 RISK-002: Missed Task in Rollback Scope (RC-003)

RPN: 216 | Severity: 9 | Probability: 4 | Detection: 6

Description

When a rollback is triggered, the RollbackController fails to identify all affected downstream tasks. Some speculative tasks continue execution or remain in a pending state with invalid ancestry.

Impact

Likelihood Assessment

Mitigation Strategy

Residual Risk

After mitigations: RPN ~36 (Sev: 9, Prob: 2, Det: 2)

Owner

Team: Rollback/Recovery

DRI: TBD

Review Cadence: Weekly

3.3 RISK-003: Concurrent Modification Race in DependencyGraph (DG-004)

RPN: 224 | Severity: 8 | Probability: 4 | Detection: 7

Description

Multiple threads/processes simultaneously modify the dependency graph, leading to data races that corrupt the graph structure. This can result in lost edges, phantom edges, or inconsistent traversal results.

Impact

Likelihood Assessment

Mitigation Strategy

Residual Risk

After mitigations: RPN ~48 (Sev: 8, Prob: 2, Det: 3)

Owner

Team: Runtime Core

DRI: TBD

Review Cadence: During code review; weekly testing

3.4 RISK-004: Lost Commitment Record (CL-001)

RPN: 162 | Severity: 9 | Probability: 3 | Detection: 6

Description

A commitment record is created in memory but fails to persist to durable storage. On restart or crash, the commitment is lost, making it appear the task was never executed.

Impact

Likelihood Assessment

Mitigation Strategy

Residual Risk

After mitigations: RPN ~27 (Sev: 9, Prob: 1, Det: 3)

Owner

Team: Storage/Persistence

DRI: TBD

Review Cadence: Monthly reliability review

3.5 RISK-005: Deadlock in Proof Pipeline (PDM-003)

RPN: 160 | Severity: 8 | Probability: 4 | Detection: 5

Description

The proof generation pipeline enters a deadlock state where workers are waiting on each other or on resources that will never become available. All proof generation stops, causing backpressure that freezes the entire system.

Impact

Likelihood Assessment

Mitigation Strategy

Residual Risk

After mitigations: RPN ~40 (Sev: 8, Prob: 2, Det: 2.5)

Owner

Team: Proof Generation

DRI: TBD

Review Cadence: Weekly performance review

4. Security Considerations

4.1 Economic Attacks

4.1.1 Griefing via Speculation

Attack Vector: Malicious agent intentionally triggers rollbacks to waste honest agents' compute resources.

Mechanism:

1. Attacker claims parent task, creates speculative commitment
2. Honest agents speculate on children, investing compute
3. Attacker abandons parent or submits invalid proof
4. All downstream work is wasted; honest agents bear compute cost

Impact: High (DoS on honest agents; economic drain)

Mitigations:

Residual Risk: Medium — Economic incentives reduce but don't eliminate griefing

4.1.2 Stake Manipulation

Attack Vector: Agent manipulates stake calculations to under-bond, reducing slashing penalty.

Mechanism:

1. Exploit bug in stake calculation logic
2. Report incorrect speculation depth
3. Get slashed less than should be

Impact: Medium (economic leakage; unfair advantage)

Mitigations:

Residual Risk: Low — On-chain validation is authoritative

4.1.3 Front-Running Speculation

Attack Vector: Observer sees high-value speculative commitment and front-runs to claim task first.

Mechanism:

1. Monitor mempool for speculative commitments
2. Race to claim same task before original agent
3. Profit from information asymmetry

Impact: Medium (unfair competition; MEV extraction)

Mitigations:

Residual Risk: Medium — MEV is inherent to public blockchains

4.2 DoS Vectors

4.2.1 Proof Queue Flooding

Attack Vector: Submit massive number of proof requests to overwhelm pipeline.

Mechanism:

1. Create many speculative tasks with dependencies
2. Each generates proof requests
3. Pipeline overloaded; legitimate proofs delayed

Impact: High (service degradation for all users)

Mitigations:

Residual Risk: Low — Multiple layers of defense

4.2.2 Graph Explosion Attack

Attack Vector: Create complex dependency graphs designed to slow traversal.

Mechanism:

1. Create dense graph with many edges
2. Trigger rollback that requires full traversal
3. System slows to crawl during traversal

Impact: Medium (temporary performance degradation)

Mitigations:

Residual Risk: Low — Limits bound worst-case

4.2.3 Storage Exhaustion

Attack Vector: Create many speculative commitments that never resolve.

Mechanism:

1. Create speculative tasks at maximum allowed rate
2. Never submit proofs
3. Storage fills with pending commitments

Impact: Medium (disk space exhaustion)

Mitigations:

Residual Risk: Low — TTL and limits prevent unbounded growth

4.3 Information Leakage

4.3.1 Speculative State Exposure

Attack Vector: Probe speculative state to learn about pending computations.

Mechanism:

1. Query dependency graph or commitment ledger
2. Infer what tasks are being speculated on
3. Gain competitive advantage

Impact: Low-Medium (information asymmetry)

Mitigations:

Residual Risk: Low — Access control is straightforward

4.3.2 Timing Side Channels

Attack Vector: Observe timing of operations to infer system state.

Mechanism:

1. Measure response times for various queries
2. Infer cache state, speculation depth, queue lengths
3. Time attacks or gain unfair advantage

Impact: Low (difficult to exploit meaningfully)

Mitigations:

Residual Risk: Low — Timing attacks rarely critical here

4.4 Race Conditions

4.4.1 Rollback During Proof Submission

Attack Vector: Race between rollback signal and proof submission.

Mechanism:

1. Task A fails, triggering rollback of dependent task B
2. Simultaneously, B's proof is being submitted on-chain
3. Proof succeeds before rollback completes
4. Inconsistent state: B confirmed but A failed

Impact: High (invalid state committed on-chain)

Mitigations:

Residual Risk: Low — On-chain validation is the final authority

4.4.2 Concurrent Commitment Updates

Attack Vector: Two processes update same commitment concurrently.

Mechanism:

1. Process A reads commitment state = SPECULATIVE
2. Process B reads commitment state = SPECULATIVE
3. A updates to PROVING
4. B updates to ROLLED_BACK
5. Final state depends on write order

Impact: Medium (inconsistent commitment state)

Mitigations:

Residual Risk: Low — OCC is well-understood pattern

5. Correctness Proof Sketches

5.1 Invariant: "Proof Never Submitted Before Ancestors Confirmed"

Formal Statement: For any task T with proof submitted at time t_s, all ancestors A ∈ ancestors(T) have confirmation time t_c(A) < t_s.

Proof Sketch:

1. Base Case: Task T with no ancestors (root task)

- ancestors(T) = ∅

- Invariant trivially holds (vacuously true)

2. Inductive Case: Task T with ancestors A₁, A₂, ..., Aₙ

Precondition (enforced by ProofDeferralManager):

- ProofDeferralManager maintains a blocking wait on each ancestor

- awaitAncestorConfirmation(T) blocks until ∀Aᵢ: status(Aᵢ) = CONFIRMED

Proof:

- Let T be any task with pending proof P

- Before submitProof(P) is called:

- checkSubmissionAllowed(T) queries CommitmentLedger

- For each Aᵢ ∈ ancestors(T):

- If status(Aᵢ) ≠ CONFIRMED: checkSubmissionAllowed returns FALSE

- Submission blocked until all ancestors confirmed

- Once all ancestors confirmed at times t_c(Aᵢ):

- checkSubmissionAllowed(T) returns TRUE at time t_check

- t_check > max(t_c(Aᵢ)) for all i

- submitProof(P) called at time t_s ≥ t_check

- Therefore: t_s > t_c(Aᵢ) for all ancestors Aᵢ ∎

3. Edge Cases:

- Reorg: If ancestor Aᵢ is re-orged after t_s but before T's confirmation

- Detection: FinalityTracker monitors for reorgs

- Response: T's proof is also invalidated and must be resubmitted

- Concurrent ancestor confirmation: Multiple ancestors confirm simultaneously

- Handled: awaitAncestorConfirmation uses barrier synchronization

- All must confirm before barrier releases

Implementation Requirement: The checkSubmissionAllowed() function MUST be atomic with submitProof() to prevent TOCTOU race.

5.2 Invariant: "Rollback Cascade is Complete" (No Orphaned Tasks)

Formal Statement: After rollback(T) completes, for all tasks D ∈ descendants(T): status(D) ∈ {ROLLED_BACK, NEVER_STARTED}.

Proof Sketch:

1. Algorithm: rollbackCascade(T) operates as follows:

function rollbackCascade(T):
  affected = computeAffectedSet(T)
  for D in reverse_topological_order(affected):
    markRolledBack(D)
  validateNoOrphans(affected)

2. Proof of Completeness:

Claim: computeAffectedSet(T) returns all descendants of T.

Proof:

- computeAffectedSet performs BFS from T following dependency edges

- DependencyGraph maintains bidirectional edges: parent→children, child→parents

- BFS visits every reachable node from T via child edges

- By definition, descendants(T) = reachable nodes via child edges

- Therefore, affected ⊇ descendants(T)

Claim: Every D ∈ affected is marked ROLLED_BACK.

Proof:

- Loop iterates over all elements in affected set

- Each iteration calls markRolledBack(D)

- markRolledBack is idempotent (marking twice is safe)

- After loop: ∀D ∈ affected: status(D) = ROLLED_BACK

Claim: No orphans exist after rollback.

Proof (by contradiction):

- Assume orphan O exists: O ∈ descendants(T) but status(O) ≠ ROLLED_BACK

- Since O ∈ descendants(T), O is reachable from T via child edges

- Therefore O ∈ affected (by completeness of BFS)

- But we proved ∀D ∈ affected: status(D) = ROLLED_BACK

- Contradiction. Therefore no orphan exists. ∎

3. Validation Step: validateNoOrphans(affected):

function validateNoOrphans(affected):
  for D in affected:
    assert status(D) == ROLLED_BACK
    for child in children(D):
      assert child in affected OR status(child) == NEVER_STARTED

4. Edge Cases:

- Concurrent task creation: New descendant created during rollback

- Prevention: Acquire write lock on affected subgraph before rollback

- New tasks blocked until rollback completes

- Already confirmed descendant: D confirmed on-chain before rollback

- This should be impossible if ancestor hasn't confirmed (Invariant 5.1)

- If occurs due to bug: Critical alert; manual reconciliation required

5.3 Invariant: "Depth Limit Enforced" (Bounded Speculation)

Formal Statement: For all tasks T in speculative state, depth(T) ≤ max_depth where depth(T) = length of longest path from any confirmed ancestor to T.

Proof Sketch:

1. Definition:

- depth(T) = 0 if T has no speculative ancestors (all ancestors confirmed)

- depth(T) = max(depth(parent) for parent in speculative_parents(T)) + 1

2. Enforcement Points:

Point A: Task scheduling in SpeculativeScheduler

function scheduleSpeculative(T):
  currentDepth = computeSpeculativeDepth(T)
  if currentDepth >= config.max_depth:
    return REJECT_DEPTH_EXCEEDED
  // ... proceed with scheduling

Point B: Commitment creation in CommitmentLedger

function createCommitment(T, depth):
  if depth > config.max_depth:
    throw DepthExceededException
  // ... create commitment

Point C: On-chain validation (Solana program)

pub fn create_speculative_commitment(ctx: Context, depth: u8) -> Result<()> {
  require!(depth <= ctx.accounts.config.max_depth, ErrorCode::DepthExceeded);
  // ... proceed
}

3. Proof of Enforcement:

Claim: No task T can enter speculative execution with depth > max_depth.

Proof:

- Path to speculative execution: scheduleSpeculative → executeTask → createCommitment

- At scheduleSpeculative: depth checked, rejected if exceeds

- At createCommitment: depth checked again (defense in depth)

- On-chain: final validation before commitment recorded

- All three checks must pass for depth > max_depth task

- If any check fails, task is not speculatively executed

- Therefore invariant holds ∎

4. Depth Calculation Correctness:

Claim: computeSpeculativeDepth(T) correctly computes depth(T).

Proof:

function computeSpeculativeDepth(T):
  if all ancestors confirmed:
    return 0
  maxParentDepth = max(computeSpeculativeDepth(P) for P in parents(T) if P.speculative)
  return maxParentDepth + 1

- Base case: No speculative ancestors → depth = 0 ✓

- Inductive case: depth = max parent depth + 1

- This matches the definition exactly ∎

5. Edge Cases:

- Ancestor confirms after depth calculation: Depth decreases; still within limit

- Config change lowering max_depth: Existing tasks grandfathered; new tasks use new limit

- Concurrent depth calculation: Each calculation independent; no race condition

6. Assumptions & Dependencies

6.1 System Assumptions

6.2 External Dependencies

6.3 Dependency Failure Modes

Solana RPC Failure

Database Failure

7. Monitoring & Detection

7.1 Critical Metrics

7.2 Detection Strategies by Failure Mode

DG-002: Stale Dependency Data

Response: Trigger full graph refresh; investigate source of staleness.

RC-003: Missed Task in Rollback

Response: Immediate page; run manual reconciliation; root cause analysis.

PDM-003: Deadlock in Proof Pipeline

Response: Restart proof workers; if persists, full service restart.

CL-001: Lost Commitment Record

Response: Check storage health; recover from WAL or on-chain state.

OC-002: Chain Reorganization

Response: Re-validate all recently confirmed commitments; rollback if needed.

7.3 Alerting Tiers

7.4 Dashboard Panels

Speculation Health Dashboard

The dashboard displays six key metric cards and a time-series chart:

A time-series panel tracks Speculation Rate (tasks/min) over 24 hours for trend analysis.

7.5 Runbook Integration

Each high-risk failure mode maps to an operational runbook procedure:

Appendix A: Risk Register Summary

Appendix B: Review History