Document ID: GOS-80-CHAIN-NO-REWORK-CHECKLIST Version: 0.1.0 Status: DRAFT Owner Role: CHAIN_OFFICER Last Review Date: 2026-03-04 Next Review Date: 2026-04-04

Goal

Lock critical decisions now so migration from local/stage to AWS production does not require redesign.

Execution Board

Status legend: ⚪ TODO → 🟡 IN_PROGRESS → 🔴 BLOCKED / 🟢 DONE

| # | Workstream | Owner | Target Date | Status | |---|---|---|---|---| | 1 | Event Contract Freeze | CHAIN_OFFICER | 2026-03-04 | 🟢 DONE | | 2 | Partitioning and Ordering | LEDGER_ENGINEER | 2026-03-04 | 🟢 DONE | | 3 | Idempotency and Exactly-Once Boundaries | ORCHESTRATION_ENGINEER | 2026-03-09 | ⚪ TODO | | 4 | DLQ and Replay Discipline | SRE_LEAD | 2026-03-04 | 🟢 DONE | | 5 | Retry/Timeout Policy | ORCHESTRATION_ENGINEER | 2026-03-10 | ⚪ TODO | | 6 | Security and Secrets | SECURITY_OFFICER | 2026-03-10 | ⚪ TODO | | 7 | Observability Baseline | OBSERVABILITY_OWNER | 2026-03-11 | ⚪ TODO | | 8 | Deployment Safety | PLATFORM_ENGINEER | 2026-03-11 | ⚪ TODO | | 9 | Environment Parity | RELEASE_MANAGER | 2026-03-12 | ⚪ TODO | | 10 | Infrastructure as Code Lock-In | PLATFORM_ENGINEER | 2026-03-12 | 🔴 BLOCKED | | 11 | Governance and Change Control | GOVERNANCE_OFFICER | 2026-03-13 | ⚪ TODO | | 12 | Exit Criteria to Production | PROGRAM_OWNER | 2026-03-14 | ⚪ TODO |

Weekly update rule:

• Update Status and Target Date every Friday.
• If a row moves to BLOCKED, add blocker note in the relevant section below.

Current sprint notes:

• #1 is 🟢 DONE: schema compatibility sign-off is published in docs/80_chain/schemas/README.md for v1 baseline contracts.
• #2 is 🟢 DONE: producer-path audit confirms orchestration records are emitted through intent-key enforcement helpers.
• #4 is 🟢 DONE: controlled failure DLQ + replay evidence captured in docs/80_chain/99_audit_logs/SINGLE_OPERATOR_DLQ_REPLAY_DEV_2026-03-04.md.
• #4 and #7 are linked to KES smoke validation path via npm run kafka:kes-smoke and npm run kafka:kes-consumer.
• #10 is 🔴 BLOCKED: backend plan/config files are ready, but stage validation is blocked because Terraform CLI is not installed on the current machine.
  • Next action: Install Terraform CLI, then run terraform init -reconfigure -backend-config=backend.stage.hcl in docs/80_chain/templates/terraform/ecs-kafka-orchestrator.

Progress update (#10):

• Minimal backend plan added in docs/80_chain/templates/terraform/ecs-kafka-orchestrator/BACKEND_SETUP.md.
• Backend config files added: backend.dev.hcl, backend.stage.hcl, backend.prod.hcl.
• Validation attempt result (2026-03-04): terraform command not found (CommandNotFoundException).

1) Event Contract Freeze

• [x] Freeze JSON schemas for milestone.approved, mint.requested, mint.confirmed, anchor.requested, anchor.confirmed, orchestration.dlq.
• [x] Enforce backward-compatible schema evolution (additive only unless major version bump).
• [x] Keep schemaVersion mandatory on all events.

2) Partitioning and Ordering

• [x] Keep message key policy fixed: key = intentId.
• [x] Prohibit producer paths that allow arbitrary key override.
• [x] Validate ordering-sensitive handlers assume per-key ordering only.

Producer-path audit sign-off (2026-03-04):

• Enforced keying path verified in packages/financial-layer/chain/orchestration/partition-key.ts via resolveIntentPartitionKey and buildKafkaRecordInput.
• KafkaJS producer path verified in packages/financial-layer/chain/orchestration/kafka-kafkajs.ts (publish -> helper -> producer send).
• Stub producer path verified in packages/financial-layer/chain/orchestration/kafka-stubs.ts (send -> helper).
• Runtime demo path verified in scripts/kafkajs-producer-consumer-demo.js (bus publish without arbitrary key override).
• Single-operator path verified in scripts/single-operator-e2e.js (stub producer send with event intent continuity).

3) Idempotency and Exactly-Once Boundaries

• [ ] Keep idempotency key mandatory for mint/anchor write paths.
• [ ] Ensure consumers are safe for at-least-once delivery (dedupe on intent + stage).
• [ ] Persist handler checkpoint/outcome before side-effect re-entry.

4) DLQ and Replay Discipline

• [x] Standardize DLQ envelope fields (eventId, intentId, reason, failedAt, sourceTopic).
• [x] Define replay criteria and operator approval gate.
• [x] Keep replay script deterministic and audited.

KES smoke linkage (acceptance criteria):

• Run npm run kafka:kes-smoke with API trigger path enabled (KES_ORCHESTRATOR_KAFKA_ENABLED=true).
• If downstream handler fails during controlled test, verify DLQ envelope preserves eventId, intentId, and source topic context.
• Replay test must re-emit from DLQ without changing intentId key semantics.

DLQ/replay sign-off (2026-03-04):

• Controlled failure injection validated with SINGLE_OPERATOR_FAIL_ON_TOPIC=mint.requested.
• DLQ envelope verified in tmp/outputs/single_operator_dlq.jsonl with required fields.
• Replay verified by scripts/single-operator-replay-dlq.js with intentKeyMismatches = 0.
• Audit artifact recorded: docs/80_chain/99_audit_logs/SINGLE_OPERATOR_DLQ_REPLAY_DEV_2026-03-04.md.

5) Retry/Timeout Policy

• [ ] Define per-step timeout budgets (mint submit, anchor submit, confirmations).
• [ ] Use bounded retries with backoff and explicit terminal state.
• [ ] Avoid infinite retries in service runtime.

6) Security and Secrets

• [ ] Keep credentials only in SSM/Secrets Manager (never in plain env files).
• [ ] Scope IAM policies to minimal Kafka/ECS/Logs permissions.
• [ ] Enforce private subnets + restricted SG ingress/egress.

7) Observability Baseline

• [ ] Log intentId, eventId, topic, partition, offset on every consume/publish boundary.
• [ ] Add failure counters per step and DLQ volume metrics.
• [ ] Add alert thresholds for retry exhaustion and consumer crash loops.

KES smoke linkage (acceptance criteria):

• During npm run kafka:kes-smoke, capture publish/consume traces containing intentId, eventId, topic, partition, and offset.
• Validate smoke summary output contains receivedCount and event-level fields from kes.orchestrator.events.
• Record one baseline run artifact per environment (dev, stage) for regression comparison.

8) Deployment Safety

• [ ] Use immutable image tags per deploy (not only latest).
• [ ] Roll forward with ECS force-new-deployment and revision tracking.
• [ ] Keep rollback command/runbook verified per environment.

9) Environment Parity

• [ ] Keep same topic names and schema versions in dev/stage/prod.
• [ ] Keep same keying/idempotency logic in all environments.
• [ ] Restrict environment differences to scale/timeouts/credentials only.

10) Infrastructure as Code Lock-In

• [ ] Manage ECS task/service through Terraform only after bootstrap.
• [ ] Configure Terraform remote state + locking (S3 + DynamoDB).
• [ ] Require plan review before apply in stage/prod.

11) Governance and Change Control

• [ ] Require ADR entry for any change in schema/keying/retry semantics.
• [ ] Require compatibility note in PR for event-related changes.
• [ ] Keep a versioned migration ledger in docs for auditability.

12) Exit Criteria to Production

• [ ] 7-day stage soak without unresolved DLQ growth.
• [ ] Replay drill executed successfully on non-prod data.
• [ ] Incident runbook tested by on-call operator.
• [ ] Security review sign-off complete.

Stage replay drill package (prepared 2026-03-04):

• Execution-ready evidence sheet: docs/80_chain/99_audit_logs/SINGLE_OPERATOR_DLQ_REPLAY_STAGE_2026-03-04.md.
• Dev proof reference: docs/80_chain/99_audit_logs/SINGLE_OPERATOR_DLQ_REPLAY_DEV_2026-03-04.md.
• Stage execution remains pending until stage MSK reachability + credentials gate is opened.

References

• docs/80_chain/SINGLE_OPERATOR_MODE.md
• docs/80_chain/AWS_MSK_ENV_MATRIX.md
• docs/80_chain/AWS_ECS_TASK_ENV_BLOCKS.md
• docs/80_chain/templates/terraform/ecs-kafka-orchestrator/
• docs/80_chain/SMOKE_EVIDENCE_TEMPLATE.md