AWS MSK Environment Matrix (Dev / Stage / Prod)

Document ID: GOS-80-CHAIN-AWS-MSK-MATRIX Version: 0.1.0 Status: DRAFT Owner Role: CHAIN_OFFICER Last Review Date: 2026-03-04 Next Review Date: 2026-04-04

Purpose

Provide a copy/paste-ready baseline for Kafka-related environment variables when running orchestration on AWS with MSK.

Related:

• ECS task definition env blocks: docs/80_chain/AWS_ECS_TASK_ENV_BLOCKS.md
• ECS task definition templates: docs/80_chain/templates/ecs-task-kafka-orchestrator.dev.json, docs/80_chain/templates/ecs-task-kafka-orchestrator.stage.json, docs/80_chain/templates/ecs-task-kafka-orchestrator.prod.json
• Terraform templates: docs/80_chain/templates/terraform/ecs-kafka-orchestrator/
• No-rework migration checklist: docs/80_chain/NO_REWORK_MIGRATION_CHECKLIST.md

Scope

• Applies to KafkaJS orchestration paths (kafka:demo and services using KafkaJsOrchestrationBus).
• kafka:smoke remains a local Docker lifecycle helper and is not intended for AWS runtime usage.

Environment Matrix

| Variable | Dev (AWS) | Stage (AWS) | Prod (AWS) | |---|---|---|---| | KAFKA_BROKERS | b-1.kvary-dev-msk.<region>.amazonaws.com:9098,b-2.kvary-dev-msk.<region>.amazonaws.com:9098 | b-1.kvary-stage-msk.<region>.amazonaws.com:9098,b-2.kvary-stage-msk.<region>.amazonaws.com:9098 | b-1.kvary-prod-msk.<region>.amazonaws.com:9098,b-2.kvary-prod-msk.<region>.amazonaws.com:9098,b-3.kvary-prod-msk.<region>.amazonaws.com:9098 | | KAFKA_DEMO_CLIENT_ID | kvary-kafka-demo-dev | kvary-kafka-demo-stage | kvary-kafka-demo-prod | | KAFKA_DEMO_TOPIC | mint.requested | mint.requested | mint.requested | | KAFKA_DEMO_TIMEOUT_MS | 30000 | 30000 | 45000 | | KAFKA_DEMO_WARMUP_MS | 1500 | 1500 | 2000 | | KAFKA_DEMO_GROUP_ID | kvary-demo-dev | kvary-demo-stage | kvary-demo-prod |

Security / Networking Notes

• Prefer private subnets and VPC-internal broker access.
• Restrict security groups to application subnets only.
• Use AWS IAM auth or TLS/SASL per MSK policy.
• Do not expose broker endpoints publicly.

Suggested Runtime Profiles

Dev

• Fast feedback, lower throughput.
• Keep retries moderate and log verbosity higher.

Stage

• Production-like settings with controlled load tests.
• Validate consumer group rebalance behavior and retry windows.

Prod

• Longer timeouts, stricter ACLs, audited credentials.
• Immutable topic/retention policy changes through change control.

Local vs AWS Command Guidance

• Local: npm run kafka:smoke
• AWS/MSK connectivity check: npm run kafka:demo

Example (PowerShell)

$env:KAFKA_BROKERS='b-1.kvary-stage-msk.eu-central-1.amazonaws.com:9098,b-2.kvary-stage-msk.eu-central-1.amazonaws.com:9098'
$env:KAFKA_DEMO_CLIENT_ID='kvary-kafka-demo-stage'
$env:KAFKA_DEMO_GROUP_ID='kvary-demo-stage'
$env:KAFKA_DEMO_TOPIC='mint.requested'
$env:KAFKA_DEMO_TIMEOUT_MS='30000'
$env:KAFKA_DEMO_WARMUP_MS='1500'
npm run kafka:demo