From diagram to demo: a runnable workflow runtime

It’s easy to nod along to an architecture diagram. The test I actually care about is what happens when the worker dies right after a side effect, or a review sits waiting for a human, or a run has to replay from the middle without corrupting the outside world.

So I built the companion repo: poc-to-scale-demo. It turns the workflow runtime from the previous post into something you can clone, run, and deploy.

The repo has one workflow definition, two runtime variants, two framework adapters, six executable scenarios, and an Azure deployment path, all arranged so you can watch the runtime misbehave and recover from a CLI.

What the repo proves

It’s built around claims you can check by running it:

• The workflow definition can stay stable while the runtime changes beneath it.
• Retry, replay, human review, crash recovery, and compensation should all be observable from a CLI.
• The production runtime should fail differently from the POC runtime on the same input.
• Framework choice should not own the entire durability story; the common runtime contract should be visible.
• Local runs, Docker-backed dependencies, and Azure should exercise the same runtime contract.

One workflow definition

The central artifact is workflow.yaml. It defines the same case-triage graph as the previous post: classify branches into enrich and fetch_refs, both feed decide, and decide either hands off directly or routes through review. Retries are declared per node, and handoff declares its compensating action alongside the forward effect.

That file is byte-identical across every combination the repo supports:

• POC runtime, on LangGraph
• POC runtime, on Microsoft Agent Framework
• Production runtime, on LangGraph
• Production runtime, on Microsoft Agent Framework

Four implementations share one definition; all the hardening happens in the runtime around it.

Six executable scenarios

The primary interface is the scenarios/ directory and the demo CLI that drives it. Each scenario is a YAML file defining an input and a fault to inject, and each maps to a runtime behavior you can inspect from the CLI.

git clone https://github.com/sachinkahawala/poc-to-scale-demo
cd poc-to-scale-demo

uv sync --extra langgraph --extra msaf
cp .env.example .env

uv run demo run happy
uv run demo run transient-failure
uv run demo run crash-mid-effect
uv run demo worker --drain
uv run demo run human-review
uv run demo approve <review_token>
uv run demo run compensation
uv run demo replay <run_id> --from decide

Scenario	What to look for
happy	Completed trace and one delivered outbox row
transient-failure	enrich fails twice, then attempt 3 succeeds
crash-mid-effect	First run exits non-zero; worker --drain resumes without a duplicate CRM case
human-review	Run parks with a review token; approval resumes it
compensation	Forward effect lands, then crm.cancel_case compensates it
replay	Upstream outcomes are reused; downstream outcomes are re-executed

They’re listed in the order I’d run them.

happy is the baseline. The graph walks classify, then enrich and fetch_refs, then decide, then handoff. The trace tree (demo trace <run_id>) matches the DAG above and gives the other scenarios a known-good reference point.

transient-failure trips enrich twice. The node has retry: { max: 4, backoff: jitter } in workflow.yaml, so the runtime backs off, retries, and the run completes. The trace records the failed attempts, with attempt numbers monotonically increasing. Every retry is visible; none are collapsed into a single “succeeded” line.

crash-mid-effect is the highest-signal failure case. The worker process dies (os._exit) after the handoff effect lands in the local outbox but before the outcome is written to state. The first demo run crash-mid-effect exits non-zero because the worker is gone. Then demo worker --drain picks up the redelivered task: a new attempt finds the dangling intent, queries the outbox by the recorded request id, and adopts the existing outcome. No second case is created.

human-review forces decide to come back at confidence 0.6, which is below the 0.8 threshold on the decide to handoff edge, so the run takes the decide to review edge instead and parks. demo inspect <run_id> shows a pending review with a token; demo approve <token> resumes the run and it completes through handoff. While the run is parked, no process is blocked. State sits in storage until an event resumes execution.

compensation lands handoff successfully, then trips a later node so that the workflow has to roll back. The runtime calls handoff’s declared compensate action (crm.cancel_case against the local outbox), and the trace shows both the forward and the reverse on the same node. The outside world is left consistent without a manual cleanup script.

replay re-executes from a recorded point in the run. demo replay <run_id> --from decide voids the outcomes of decide and everything downstream, then re-enqueues the run. The re-walk short-circuits every node whose recorded outcome is still valid (same input hash, same idempotency key, recorded outcome returned) and re-executes only the voided ones. The full history stays in the trace.

POC and production runtimes

Every scenario takes --variant poc|prod. The two variants share the graph, the node implementations, the model gateway, and the scenarios. They differ in the runtime boundary:

	--variant poc	--variant prod (default)
Process model	one process owns everything	scheduler + stateless workers
State	in memory	durable store; intent before every effect, outcome after
Tasks	in-process function calls	at-least-once queue with redelivery and a DLQ
Crash mid-run	the run is gone	redelivery resumes; recorded intent makes the retry idempotent
Human review	blocks the process	parks the run; an event wakes it

The most useful comparison is crash-mid-effect. Under --variant poc the run is gone with the process, because state lived inside the process. Under --variant prod, queue redelivery and the intent/outcome ledger recover the run without duplicating the external effect. You can watch the two variants fail differently on the same input.

Two frameworks, one runtime contract

The repo implements every variant twice - once on LangGraph, once on Microsoft Agent Framework - with both calling Azure OpenAI through the same model gateway and running the same execute loop in src/common/executor.py. Every scenario passes under both with --framework langgraph|msaf.

The detailed framework comparison lives in docs/LANGGRAPH_VS_MSAF.md. What stood out building both is how much of the production durability sits outside the graph frameworks, in src/common/:

• the intent/outcome ledger
• the effect-recovery arm (query the world by request id after a crash)
• compensations for non-idempotent effects
• the model gateway (rate limit, per-run budget, deployment mapping)
• the task queue between scheduler and workers
• replay as a first-class operation over recorded inputs

Most of what makes the system durable has nothing to do with the graph framework - it’s the ledger, the recovery arm, the compensations, the gateway, and the queue. Running the same definition on two frameworks with one shared recovery contract is how the repo makes that visible.

Local, Docker, and Azure execution

By default the prod variant runs on SQLite (state) and a durable local queue. That requires no Azure account, no Docker, and no cloud spend. Those defaults are enough to demonstrate every scenario.

For storage and queue semantics closer to the Azure deployment, running docker compose up -d brings up Postgres and the Azure Service Bus emulator locally:

docker compose up -d
DEMO_STATE_BACKEND=postgres DEMO_BROKER_BACKEND=servicebus \
SERVICEBUS_CONNECTION_STRING="Endpoint=sb://localhost;...UseDevelopmentEmulator=true;" \
uv run demo run happy

The Azure deployment is the same production runtime mapped onto managed services.

For the actual Azure deployment, azd up provisions this topology and azd down tears it back out. The walkthrough, cost notes, and teardown checks live in docs/AZURE_DEPLOY.md.

The Azure path creates standing paid resources, mainly Postgres and Service Bus. Treat it as disposable infrastructure and run azd down --purge after testing.

On Azure, OTel spans export to Application Insights with span names equal to node ids, so the trace tree in the portal is the graph from workflow.yaml. The workflow definition is also the schema for the trace.

What is intentionally out of scope

Several production concerns are deliberately left out because they wouldn’t change the contracts being demonstrated:

• Auth. The CLI, scheduler /review endpoint, and demo credentials are local-dev-only. The repo demonstrates runtime patterns rather than service hardening.
• A web UI animating the graph. The CLI trace and App Insights cover it; a UI is its own product.
• APIM in front of Azure OpenAI. The natural next step for multi-tenant rate isolation, but it roughly doubles the infra for one demo. It is discussed in the framework comparison doc and left out of the deployment.
• Multi-region / HA Postgres. A single Flexible Server with backups is enough to demonstrate durability.
• An actual CRM. The handoff effect writes to a local outbox table that exposes a queryable request id, the same shape as the post’s idempotent-effect example, none of the third-party plumbing.

Each of these would be a real decision in a production deployment, but none of them changes the runtime contract the repo demonstrates.

Where to start

The fastest way to evaluate the repo is:

1. Run crash-mid-effect under both variants. This gives the smallest demo that makes the POC-to-prod runtime difference visible.
2. Open src/common/executor.py and read the loop. The loop is short, readable, and sits at the center of every scenario.
3. Write a seventh scenario. Pick a failure mode outside the current six (a model timeout that turns into budget exhaustion, a review that times out past its SLA, an effect whose compensation itself fails) and add a YAML for it. If the scenario passes under --variant prod and exposes the expected POC limitation under --variant poc, the runtime is doing what it claims.

The previous post ended with a promise to cover end-to-end workflow evaluation, and that post now exists: Evals for workflows, not just nodes, with its own companion repo that pins and evaluates this one. The runtime needed to be executable and observable before evaluation could mean much, which is what this repo is for.

I hope it gives you a practical place to test these ideas in your own systems. If you find a failure mode the six scenarios miss, I would be glad to hear about it.