Messaging / Payments Hardening Runbook

Purpose

This is the narrow operator/developer runbook for the messaging, call, and payments stack after the hardening pass.

It covers:

• health endpoints
• websocket reconnect expectations
• real Algorand signer readiness
• the small smoke command

Health Endpoints

• API: http://127.0.0.1:4001/api/v1/health
• Messaging: http://127.0.0.1:4026/health
• Payments: http://127.0.0.1:4027/health
• Dev signer: http://127.0.0.1:4591/health

Messaging Health

The messaging health payload now reports:

• database readiness
• websocket path
• connected account count
• total open sockets

Payments Health

The payments health payload now reports:

• database readiness
• messaging dependency reachability
• signer dependency reachability
• Algorand mode / chain / asset configuration
• ready for real transfer execution

ok reflects local service health.

ready reflects whether the configured real-transfer path is usable.

Recommended Local Env

Set these in local development when testing real Algorand transfers:

• MESSAGING_SERVICE_URL=http://localhost:4026
• PAYMENTS_SERVICE_URL=http://localhost:4027
• PAYMENTS_ALGORAND_MODE=TESTNET_SIGNER
• ALGOD_TESTNET_URL=https://testnet-api.algonode.cloud
• ALGORAND_SIGNER_URL=http://127.0.0.1:4591/sign/algorand/payment
• ALGORAND_SIGNER_MNEMONIC=...

Never commit a real mnemonic.

Dev Signer Notes

The dev signer is still a deliberate trust boundary for local/test usage.

For payment signing it now:

• validates sender ownership against the configured mnemonic
• fetches live suggested params from algod
• builds a real Algorand payment transaction
• signs it
• returns signed bytes to svc-payments

Websocket Recovery Expectations

The web live connection now:

• reconnects with exponential backoff
• retries on browser online
• retries on focus
• retries when the tab becomes visible again

The call shell now revalidates the active call when the websocket transport reopens.

Smoke Command

Run:

npm run smoke:messaging-payments

Optional real-transfer readiness mode:

MESSAGING_PAYMENTS_SMOKE_EXPECT_SIGNER=1 npm run smoke:messaging-payments

The smoke script validates:

• API health
• messaging health + realtime stats
• payments health + database readiness
• signer health when explicitly expected

Payments Reconciliation Worker

When reconciliation is enabled, run:

PAYMENTS_RECONCILIATION_ENABLED=true npm --prefix services/svc-payments run work:payments-reconciliation

Each scan now synchronizes persisted reconciliation findings:

• active stuck/finality issues are opened or refreshed
• findings that no longer appear in the scan are marked RESOLVED
• worker logs include flagged and resolved counts for the scan

Detected conditions:

• STUCK_PENDING: a payment projection remains PENDING beyond PAYMENTS_PENDING_STUCK_THRESHOLD_SECONDS
• CONFIRMED_WITHOUT_FINALITY: execution is confirmed while finality has not been recorded after PAYMENTS_RECONCILIATION_FINALITY_GRACE_SECONDS
• FAILED_WITHOUT_FINALITY: execution failed while finality has not been recorded after the grace window

Lifecycle semantics:

• OPEN means the finding is present in the latest full reconciliation scan.
• RESOLVED means the finding disappeared from a later full scan; the row is retained for audit history.
• If the same (intent_id, finding_code) appears after resolution, the existing row reopens instead of creating a duplicate.
• Repeated identical scans are idempotent and should not produce duplicate findings or noisy writes.

Operator visibility:

• API gateway: GET /api/v1/payments/admin/payments/reconciliation?limit=50
• Payments service: GET /admin/payments/reconciliation?limit=50
• Admin UI: /admin/payments
• Review findingId, intentId, optional groupId, findingCode, findingStatus, first/last/resolved timestamps, and current projection status before taking any manual action.

Reconciliation is detect-only. It must not create custody, debit accounts, submit automatic retries, execute fallback rails, or mutate settlement state. Retry/fallback decisions remain manual operator actions outside this worker and outside the read-only admin view.

Keepz Bank Settlement

svc-payments uses Keepz as the primary BANK settlement provider when BANK rails are enabled:

PAYMENTS_BANK_LIVE_ENABLED=true \
PAYMENTS_LIVE_ENABLED_RAILS=BANK,CRYPTO \
PAYMENTS_BANK_PROVIDER_TYPE=KEEPZ \
KEEPZ_EXECUTION_MODE=DRY_RUN \
npm --prefix services/svc-payments run dev

For live Keepz execution, set KEEPZ_EXECUTION_MODE=LIVE, KEEPZ_API_BASE_URL, and KEEPZ_API_KEY. For sandbox execution, set KEEPZ_EXECUTION_MODE=SANDBOX, KEEPZ_CLIENT_ID, KEEPZ_CLIENT_SECRET, and optionally KEEPZ_SANDBOX_BASE_URL. Keepz beneficiary settlement profiles must provide bankAccountRef as either a Georgian IBAN or Keepz receiver UUID.

Buy Now Keepz Split Order

Buy Now settlement uses one Keepz eCommerce order with splitDetails: the buyer pays the full order amount once, Keepz settles the seller net leg and the Kvary platform commission leg. Kvary does not receive the full buyer amount, does not hold escrow, and does not create a fallback direct transfer when split order readiness fails.

Enable this path only after Keepz confirms eCommerce split order support for the merchant account:

PAYMENTS_BANK_PROVIDER_TYPE=KEEPZ \
KEEPZ_SPLIT_ORDER_ENABLED=true

Create the draft through the payments API or API gateway:

• Payments service: POST /admin/payments/buy-now-split-order
• Buyer-facing payments service: POST /payments/buy-now-split-order
• API gateway: POST /api/v1/payments/admin/payments/buy-now-split-order
• Buyer-facing API gateway: POST /api/v1/payments/buy-now-split-order
• Web auction bridge: POST /api/auctions/:id/buy-now-payment

The web auction bridge resolves the seller from auction detail and the Kvary commission recipient from server-side configuration before calling the gateway. Set one of BUY_NOW_PLATFORM_ACCOUNT_ID, KVARY_PLATFORM_ACCOUNT_ID, or PAYMENTS_PLATFORM_ACCOUNT_ID; set BUY_NOW_PLATFORM_COMMISSION_BPS when the default 300 bps commission should be overridden.

Server-side environment for this buyer-facing bridge:

• Web route: API_GATEWAY_BASE_URL, AUCTION_DECLARATION_SERVICE_URL or TENDERS_SERVICE_URL, BUY_NOW_PLATFORM_ACCOUNT_ID or KVARY_PLATFORM_ACCOUNT_ID or PAYMENTS_PLATFORM_ACCOUNT_ID, and optional BUY_NOW_PLATFORM_COMMISSION_BPS.
• API gateway: PAYMENTS_SERVICE_URL.
• Payments service: PAYMENTS_BANK_PROVIDER_TYPE=KEEPZ, KEEPZ_SPLIT_ORDER_ENABLED=true, PAYMENTS_BANK_LIVE_ENABLED, PAYMENTS_LIVE_ENABLED_RAILS, KEEPZ_EXECUTION_MODE, KEEPZ_FORCE_DRY_RUN, KEEPZ_API_BASE_URL, KEEPZ_API_KEY, KEEPZ_CLIENT_ID, KEEPZ_CLIENT_SECRET, KEEPZ_SANDBOX_BASE_URL, KEEPZ_LIVE_BASE_URL, KEEPZ_AUTH_TIMEOUT_MS, KEEPZ_TRANSACTION_TIMEOUT_MS, and KEEPZ_TIMEOUT_MS.

Do not expose Keepz credentials or Kvary platform recipient ids through NEXT_PUBLIC_ variables. Receiver kind is not a separate browser input: settlement profile bankAccountRef encodes it as keepz:user:<receiver-id>, keepz:branch:<branch-id>, keepz:iban:<iban>, a Georgian IBAN, or a Keepz receiver UUID.

Required profiles:

• Seller profile must be VERIFIED, support BANK, and expose a Keepz-recognized bankAccountRef.
• Kvary platform commission profile must be VERIFIED, support BANK, and expose a Keepz-recognized bankAccountRef.
• Supported bankAccountRef formats are keepz:user:<receiver-id>, keepz:branch:<branch-id>, keepz:iban:<iban>, a Georgian IBAN, or a Keepz receiver UUID.

Readiness failures return 409 and do not create a payment intent. Important reason codes are KEEPZ_SPLIT_ORDER_NOT_CONFIGURED, SELLER_PAYOUT_PROFILE_MISSING, PLATFORM_COMMISSION_RECIPIENT_MISSING, and SPLIT_LEGS_UNBALANCED. Commission BPS calculations use minor units and round down; the seller leg receives the remainder so the two splitDetails entries always sum to the buyer payable total.

Buy Now Payload Smoke

Run this smoke after building svc-payments; it prints the canonical Keepz create-order payload and does not call live or sandbox rails:

npm --prefix services/svc-payments run build
npm run smoke:keepz-buy-now-split

Expected smoke scenario: buyer total 10000.00 GEL, commission 300.00 GEL, seller net 9700.00 GEL. The printed Keepz payload must contain gross amount: 10000, two splitDetails legs, the seller receiver, the Kvary commission receiver, a deterministic uniqueId, and no commissionType, toIban, or direct receiverId fields in split-order mode.

Example split payload shape:

{
   "amount": 10000,
   "currency": "GEL",
   "description": "Buy Now smoke auction-smoke-10000-gel x1",
   "uniqueId": "<deterministic-uuid>",
   "splitDetails": [
      {
         "receiverType": "USER",
         "receiverIdentifier": "seller-smoke-receiver",
         "amount": 9700
      },
      {
         "receiverType": "BRANCH",
         "receiverIdentifier": "kvary-platform-commission",
         "amount": 300
      }
   ]
}

Seller Payout Source

The browser calls POST /api/auctions/:id/buy-now-payment with buyer account and quantity only. The web route fetches auction detail from the auction declaration service, reads auction.ownerStakeholderId as the seller account, reads the Kvary platform account from server env, and forwards both to POST /api/v1/payments/buy-now-split-order. svc-payments then loads seller and platform settlement profiles from party_settlement_profiles; each profile must be VERIFIED, support BANK, and expose a Keepz-recognized bankAccountRef.

Missing seller account blocks in the web bridge with SELLER_PAYOUT_PROFILE_MISSING. Missing or invalid seller settlement profile blocks in svc-payments with SELLER_PAYOUT_PROFILE_MISSING. Both paths return 409 and do not create a payment intent.

Callback And Finality Limits

There is currently no dedicated Keepz webhook/callback endpoint. Keepz finality is observed through the BANK rail reconciliation path: execution calls KeepzProviderClient.getInstructionStatus, BankRailAdapter.reconcile maps provider status to PENDING, CONFIRMED, or FAILED, and settlement handlers update the settlement execution plus payment finality. Reconciliation findings then detect stuck pending or finality gaps from payment projection state.

Provider limitation: split-recipient finality is not represented as separate seller/platform leg finality from a callback. The current model stores one Buy Now payment intent with split metadata and can confirm or fail the overall Keepz order; seller net and Kvary commission legs remain metadata unless Keepz exposes per-split-recipient settlement status and a webhook/status adapter is added.

Manual Browser Smoke

1. Start web, API gateway, auction service, and svc-payments with the server-side env above; keep KEEPZ_EXECUTION_MODE=DRY_RUN or sandbox unless live rails are explicitly intended.
2. Open an auction detail page with Buy Now enabled.
3. Choose a whole-number quantity.
4. Click Buy Now.
5. Confirm one Buy Now split-order intent is created through POST /api/auctions/:id/buy-now-payment and POST /api/v1/payments/buy-now-split-order.
6. Confirm the buyer sees one payment action, not separate seller and platform charges.
7. In admin/payment detail or logs, confirm gross buyer amount, seller net, Kvary commission, settlementFlow=KEEPZ_ECOMMERCE_SPLIT_ORDER, and custodyModel=NO_ESCROW_KEEPZ_SPLIT_ORDER.
8. Remove or invalidate the seller payout profile and repeat; Buy Now must return 409, show the seller payout readiness failure, and create no payment intent.

Real Transfer Checklist

1. Start the dev signer:

npm run signer:dev

2. Start the platform services.
3. Confirm svc-payments /health reports ready: true.
4. Bind the sender wallet address to the same address derived from the signer mnemonic.
5. Run a small DM transfer and confirm:
   • thread shows payment_intent -> payment_pending -> payment_confirmed
   • payments table has a real transaction id
   • sender/recipient notifications are emitted