Production Hardening

This document describes the production hardening system for the AgenC runtime. Production hardening applies a secure-by-default configuration layer on top of an operator's existing runtime config, enforcing strict safety baselines for mainnet deployments.

Production hardening is not a separate binary or runtime mode. It is a decorator that merges hardened settings into whatever config the operator already has. The merge is strict: any field that fails schema validation causes an immediate error rather than falling back to an unsafe default.

Overview

The hardening profile covers four categories:

Each category is applied as a set of non-negotiable defaults. Operators may override specific fields only by passing explicit overrides at merge time. Fields not overridden inherit the hardened baseline.

Usage

Applying the hardening profile

import { applyProductionHardening } from '@agenc/runtime';
import { loadConfig } from '@agenc/runtime/config';

const baseConfig = loadConfig('./agenc.config.ts');

const hardened = applyProductionHardening(baseConfig, {
  overrides: {
    rpcUrl: 'https://api.mainnet-beta.solana.com',
  },
});

The returned hardened object is a fully resolved runtime config with all hardening defaults applied. It can be passed directly to the runtime entrypoint.

With fallback RPC endpoints

const hardened = applyProductionHardening(baseConfig, {
  overrides: {
    rpcUrl: 'https://api.mainnet-beta.solana.com',
    rpcFallbacks: [
      'https://solana-mainnet.g.alchemy.com/v2/YOUR_KEY',
      'https://mainnet.helius-rpc.com/?api-key=YOUR_KEY',
    ],
  },
});

With custom policy limits

const hardened = applyProductionHardening(baseConfig, {
  overrides: {
    rpcUrl: 'https://api.mainnet-beta.solana.com',
    policy: {
      maxConcurrentTasks: 10,
      taskBudgetLamports: 500_000_000,
      executionTimeoutMs: 120_000,
      circuitBreakerThreshold: 5,
      rateLimitPerMinute: 30,
    },
  },
});

Hardening categories

Policy

The policy layer enforces operational safety bounds on agent execution.

Circuit breaker behavior. When consecutive task failures reach the threshold, the runtime halts new task scheduling for the cooldown period. This prevents cascading failures from propagating across dependent tasks or draining funds through repeated failing executions.

// Circuit breaker opens after 3 consecutive failures
// and blocks new tasks for 60 seconds
{
  circuitBreakerThreshold: 3,
  circuitBreakerCooldownMs: 60_000,
}

Endpoints

The endpoints layer ensures the runtime connects to production-grade RPC infrastructure with health monitoring and automatic failover.

Failover behavior. The runtime monitors the primary RPC endpoint at the configured interval. If the health check fails three consecutive times, the runtime automatically fails over to the next endpoint in the fallback chain. If all endpoints are unhealthy, the runtime enters a degraded state and emits an alert.

Evidence

The evidence layer ensures all incident data is sealed, retained, and protected from accidental or malicious deletion.

Auto-sealing. When an incident is created, the evidence store immediately computes a cryptographic seal over the incident payload. This seal is stored alongside the evidence and verified on any subsequent read. Tampering with sealed evidence causes a verification failure.

Deletion protection

In production, evidence deletion is disabled by default. Any attempt to delete evidence through the runtime API is rejected with an EVIDENCE_DELETION_DISABLED error.

If an operator requires deletion capability (for example, to comply with a data retention regulation), the override must be explicit:

const hardened = applyProductionHardening(baseConfig, {
  overrides: {
    rpcUrl: 'https://api.mainnet-beta.solana.com',
    evidence: {
      deletionEnabled: true,
      deletionRequiresApproval: true,
    },
  },
});

When deletionRequiresApproval is true, every deletion attempt is logged to the audit trail and requires a second confirmation step before execution.

Readiness checks

Before the hardening profile is applied, the system runs a series of readiness checks. If any check fails, applyProductionHardening throws a HardeningReadinessError with a structured list of failures.

Checks performed

Running readiness checks independently

Readiness checks can be run without applying the full hardening profile. This is useful in CI pipelines or pre-deployment validation steps.

import { checkProductionReadiness } from '@agenc/runtime';

const result = await checkProductionReadiness({
  programId: 'YOUR_PROGRAM_ID',
  rpcUrl: 'https://api.mainnet-beta.solana.com',
  walletPath: '~/.config/solana/mainnet-deployer.json',
  evidenceStorePath: '.agenc/evidence.sqlite',
});

if (!result.ready) {
  console.error('Readiness check failed:');
  for (const failure of result.failures) {
    console.error(`  ${failure.code}: ${failure.message}`);
  }
  process.exit(1);
}

CLI readiness check

agenc health --network mainnet --config ./agenc.config.ts

Expected output when all checks pass:

[ok] Program ID matches deployment manifest
[ok] Wallet balance: 4.23 SOL (minimum: 0.5 SOL)
[ok] RPC endpoint reachable: https://api.mainnet-beta.solana.com (latency: 42ms)
[ok] Evidence store writable: .agenc/evidence.sqlite

All readiness checks passed.

Expected output when a check fails:

[ok] Program ID matches deployment manifest
[FAIL] Wallet balance: 0.001 SOL (minimum: 0.5 SOL)
[ok] RPC endpoint reachable: https://api.mainnet-beta.solana.com (latency: 42ms)
[ok] Evidence store writable: .agenc/evidence.sqlite

1 readiness check failed. Resolve before applying production hardening.

Merge behavior

The hardening profile uses strict merge semantics. This section describes how conflicts between the operator's base config and the hardened defaults are resolved.

Merge rules

1. Hardened defaults win. If the operator's config does not specify a value for a hardened field, the hardened default is used.
2. Explicit overrides win. If the operator passes a value in the overrides object, that value takes precedence over both the base config and the hardened default.
3. No fallback to unsafe defaults. If a required field is missing from all three sources (base config, hardened defaults, overrides), the merge fails with a schema validation error.
4. Fail-fast on schema errors. Any value that does not match the expected type or range causes an immediate HardeningSchemaError.
5. Evidence export sealed by default. The exportSealedByDefault flag is always true in the hardened profile. It can only be set to false via an explicit override.
6. Relaxation requires explicit override. Any hardened setting that is more restrictive than the operator's base config cannot be relaxed unless the operator provides an explicit override. The merge does not silently adopt the less restrictive value from the base config.

Merge precedence diagram

Lowest priority                              Highest priority
     |                                            |
     v                                            v

  Base config  -->  Hardened defaults  -->  Explicit overrides
  (operator)       (safety baseline)       (overrides object)

Error handling

import { applyProductionHardening, HardeningSchemaError } from '@agenc/runtime';

try {
  const hardened = applyProductionHardening(baseConfig, {
    overrides: {
      rpcUrl: 'https://api.mainnet-beta.solana.com',
      policy: {
        executionTimeoutMs: -1, // invalid: must be positive
      },
    },
  });
} catch (err) {
  if (err instanceof HardeningSchemaError) {
    console.error('Schema validation failed:');
    for (const issue of err.issues) {
      console.error(`  ${issue.path}: ${issue.message}`);
    }
    // Output:
    //   policy.executionTimeoutMs: must be a positive integer
  }
}

Recommended production checklist

Follow these steps in order when preparing a production deployment.

1. Verify environment health

agenc health --network mainnet --config ./agenc.config.ts

Confirm all readiness checks pass before proceeding.

2. Apply the hardening profile

import { applyProductionHardening } from '@agenc/runtime';
import { loadConfig } from '@agenc/runtime/config';

const baseConfig = loadConfig('./agenc.config.ts');
const hardened = applyProductionHardening(baseConfig, {
  overrides: {
    rpcUrl: 'https://api.mainnet-beta.solana.com',
  },
});

3. Validate the merged config

agenc doctor --config ./agenc.config.ts --hardened

Expected output:

Merged config validation:
  [ok] Policy: all limits within safe bounds
  [ok] Endpoints: primary + 0 fallbacks configured
  [ok] Evidence: auto-seal enabled, deletion disabled
  [ok] Schema: all fields valid

No issues found.

4. Deploy with monitoring enabled

agenc start --config ./agenc.config.ts --hardened --monitor

The --monitor flag enables real-time alerting for circuit breaker trips, RPC failovers, and evidence store errors.

Troubleshooting

References