Reality Verdict

• VERIFIED: Kafka is real and operational in this repo.
• VERIFIED: Kafka is currently most important in KES + outbox + projection flows.
• VERIFIED: Kafka is not yet equally central across all product domains.
• VERIFIED: Broad claims that the entire platform is already event-driven would be misleading.

Messaging Infrastructure

Configuration

Primary runtime configuration lives in:

• services/svc-tenders/src/config.ts
• services/api/src/routes/kes-orchestrator.ts

Verified environment variables:

• KES_ORCHESTRATOR_KAFKA_ENABLED
• KES_ORCHESTRATOR_KAFKA_BROKERS
• KES_ORCHESTRATOR_KAFKA_TOPIC
• KES_ORCHESTRATOR_KAFKA_CLIENT_ID
• KES_ORCHESTRATOR_KAFKA_GROUP_ID
• KES_ORCHESTRATOR_KAFKA_FROM_BEGINNING
• KES_ORCHESTRATOR_KAFKA_DLQ_ENABLED
• KES_ORCHESTRATOR_KAFKA_DLQ_TOPIC
• KES_ORCHESTRATOR_KAFKA_MAX_RETRIES
• KES_ORCHESTRATOR_KAFKA_RETRY_BASE_MS
• KES_ORCHESTRATOR_KAFKA_RETRY_MAX_MS
• KES_OUTBOX_ENABLED
• KES_OUTBOX_KAFKA_TOPIC
• KES_OUTBOX_POLL_MS
• KES_OUTBOX_BATCH_SIZE
• KES_OUTBOX_MAX_ATTEMPTS
• KES_OUTBOX_MONITORING_ENABLED
• KES_OUTBOX_MONITORING_HOST
• KES_OUTBOX_MONITORING_PORT
• KES_DOMAIN_EVENTS_DLQ_ENABLED
• KES_DOMAIN_EVENTS_DLQ_TOPIC

Kafka Client Libraries

• VERIFIED: kafkajs is used in:
  • services/api/src/kafka/kesEventProducer.ts
  • services/svc-tenders/src/kafka/kesEventConsumer.ts
  • services/svc-tenders/src/kafka/kesDomainEventConsumer.ts
  • services/svc-tenders/src/kafka/kesOutboxRelay.ts

Topic Inventory

| Topic | Where Defined | Producer | Consumer | Purpose | State | | --- | --- | --- | --- | --- | --- | | kes.orchestrator.events | services/svc-tenders/src/config.ts | services/api/src/routes/kes-orchestrator.ts | services/svc-tenders/src/kafka/kesEventConsumer.ts | proxy mutation envelope stream for KES actions | REAL | | kes.orchestrator.events.dlq | same | KesEventConsumer invalid/final-failure DLQ publish | services/svc-tenders/src/kafka/kesDlqReplay.ts | operator replay/inspection for failed KES proxy events | REAL | | kes.orchestrator.domain-events | services/svc-tenders/src/config.ts | services/svc-tenders/src/kafka/kesOutboxRelay.ts | services/svc-tenders/src/kafka/kesDomainEventConsumer.ts | durable domain-event stream from Postgres outbox | REAL | | <KES_OUTBOX_KAFKA_TOPIC>.dlq default kes.orchestrator.domain-events.dlq | services/svc-tenders/src/kafka/kesDomainEventConsumer.ts | kesDomainEventConsumer | replay tooling is INFERRED only; explicit consumer not found | invalid domain-event payload DLQ | PARTIAL | | milestone.approved | packages/financial-layer/chain/orchestration/kafka-dto.ts | demo/scripts | demo/scripts | multichain orchestration demo | MIXED | | mint.requested | same | demo/scripts | demo/scripts | multichain mint workflow demo | MIXED | | mint.confirmed | same | demo/scripts | demo/scripts | multichain mint confirmation demo | MIXED | | anchor.requested | same | demo/scripts | demo/scripts | anchoring demo | MIXED | | anchor.confirmed | same | demo/scripts | demo/scripts | anchoring demo | MIXED | | orchestration.dlq | same | demo/scripts | demo/scripts | demo DLQ topic | MIXED |

Producer Inventory

1. KES proxy mutation producer

• Source: services/api/src/routes/kes-orchestrator.ts
• Trigger: every non-GET KES proxy call after upstream response is received.
• Event type:
  • KES_ORCHESTRATOR_MUTATION_RECORDED
  • envelope built by services/api/src/kafka/kesEventSchema.ts
• Safety verdict:
  • VERIFIED: direct producer, not outbox-driven.
  • VERIFIED: not transactionally tied to upstream service DB commit.
  • VERIFIED: producer failure is swallowed after comment-only handling.
  • Verdict: useful operational event stream, but not canonical durability.

2. Outbox relay producer

• Source: services/svc-tenders/src/kafka/kesOutboxRelay.ts
• Trigger: polling undispatched rows from kes_outbox_events.
• Query evidence:
  • uses FOR UPDATE SKIP LOCKED
  • marks dispatched_at or increments attempts
• Safety verdict:
  • VERIFIED: this is the durable producer path.
  • VERIFIED: DB write and outbox enqueue are transactionally linked via triggers.
  • VERIFIED: Kafka publication is async and at-least-once.

Consumer Inventory

1. KesEventConsumer

• Source: services/svc-tenders/src/kafka/kesEventConsumer.ts
• Consumes: kes.orchestrator.events
• Failure behavior:
  • strict JSON/envelope validation
  • retry with exponential backoff
  • DLQ publish for parse failures and handler failures
  • idempotency via kafka_consumer_idempotency
• Critical truth:
  • VERIFIED: current default main() handler logs consumption only.
  • Business-side effects beyond logging are UNVERIFIED in the current wired runtime.

2. kesDomainEventConsumer

• Source: services/svc-tenders/src/kafka/kesDomainEventConsumer.ts
• Consumes: kes.orchestrator.domain-events
• Side effects:
  • writes kes_domain_event_projection_events
  • writes kes_domain_event_projection_counters
  • writes kes_domain_event_projection_aggregates
• Failure behavior:
  • strict contract validation
  • optional DLQ for invalid domain payloads
  • idempotency keyed by eventId
• Verdict:
  • VERIFIED: this is the most meaningful downstream event-driven processor in the repo.

Outbox Pattern

Tables and Migrations

• services/svc-tenders/migrations/0027_kes_outbox.sql
  • creates kes_outbox_events
  • creates unique index on dedupe_key
  • creates trigger from kes_orchestrator_events
• services/svc-tenders/migrations/0028_tenders_events_outbox.sql
  • enqueues tender events to kes_outbox_events
• services/svc-tenders/migrations/0031_legacy_tender_events_outbox.sql
  • enqueues legacy tender events to kes_outbox_events
• services/svc-tenders/migrations/0030_event_contract_versioning.sql
  • normalizes stored outbox payload contract with eventType and eventVersion

Canonicality

• VERIFIED: KES/tender outbox is real and canonical for domain-event publication.
• VERIFIED: outbox is partial platform-wide. It is not used consistently by all services/domains.

Idempotency / Duplicate Protection

Consumer idempotency store

• Table: kafka_consumer_idempotency
• Migration: services/svc-tenders/migrations/0026_kafka_consumer_idempotency.sql
• Code: services/svc-tenders/src/kafka/idempotencyStore.ts

Actual behavior

• VERIFIED: at-least-once delivery with consumer-side duplicate protection.
• VERIFIED: not true exactly-once.
• VERIFIED: kesEventConsumer uses payload.requestId fallback eventId.
• VERIFIED: kesDomainEventConsumer uses eventId.

DLQ and Replay

Implemented

• kes.orchestrator.events.dlq
  • producer: KesEventConsumer
  • replay tool: services/svc-tenders/src/kafka/kesDlqReplay.ts
• poison outbox replay:
  • tool: services/svc-tenders/src/kafka/kesOutboxPoisonReplay.ts
• relay monitoring:
  • endpoints in services/svc-tenders/src/kafka/kesOutboxRelay.ts
  • alert docs referenced by services/svc-tenders/README.md

Incomplete

• INFERRED: domain-events DLQ replay story is weaker/documented less clearly than orchestrator-event DLQ replay.

Event Contract Discipline

• VERIFIED: KES orchestrator proxy events have explicit envelope schema and parser.
• VERIFIED: domain events enforce schemaVersion, eventType, eventVersion.
• VERIFIED: event naming in KES area is much more disciplined than the rest of the repo.
• UNVERIFIED: broader cross-service event version compatibility governance beyond these flows.

KES Event Flow

Command / mutation entry

1. Web or operator action calls /api/v1/kes/orchestrator/* via services/api.
2. Gateway proxies to svc-tenders /kes-orchestrator/*.
3. svc-tenders persists KES state + writes kes_orchestrator_events.
4. Postgres trigger enqueues row into kes_outbox_events.
5. Optional relay publishes to kes.orchestrator.domain-events.
6. Domain consumer updates projection tables.
7. Read routes expose case, projection, control-plane, and suggestions.

Supporting evidence

• Gateway proxy: services/api/src/routes/kes-orchestrator.ts
• Durable write routes: services/svc-tenders/src/server.ts
• Repository writes: services/svc-tenders/src/repository.ts
• Outbox migrations: services/svc-tenders/migrations/0027_kes_outbox.sql
• Relay: services/svc-tenders/src/kafka/kesOutboxRelay.ts
• Projection consumer: services/svc-tenders/src/kafka/kesDomainEventConsumer.ts

Strongest Event-Driven Areas

• VERIFIED: KES + tender outbox + domain-event projection pipeline
• VERIFIED: idempotency + DLQ tooling in svc-tenders
• VERIFIED: replay/operator tooling is real enough to be operationally meaningful

Misleading Claims To Avoid

• “Kafka is the universal backbone of every product flow.”
  • False.
• “All KES Kafka consumers perform business mutations.”
  • False; the kes.orchestrator.events consumer is currently mostly transport/logging oriented.
• “Financial-layer Kafka orchestration is already a core production runtime.”
  • False; current evidence points to demo/script-heavy usage.