Your first stress test
You will write a sourcing policy as a typed rule with a mandate. Then you will run it against the current policy on the same 300 futures, and read what the numbers say about it. Everything here runs on your own machine.
Where things stand. PolicyForge's first screen, the Network Studio, builds and checks networks; the screens for writing and testing policies come in P7. In this tutorial you write the rule yourself, in Python, exactly as the compiler produces it from plain language. To see the compiler at work, run python -m examples.compile_intent: it compiles this tutorial's intent offline from a recorded answer, or any intent live with --live and an Anthropic API key.
1 · Set up
You need Python 3.12 and the code from the repository. uv installs Python 3.12 for you if you don't have it.
git clone https://github.com/kathuman/claude-projects.git
cd claude-projects/policyforge/backend
uv venv --python 3.12 .venv
uv pip install --python .venv -e ".[dev]"
.venv/bin/python -m pytest -q # Windows: .venv\Scripts\python
You should see 179 tests pass in about five minutes. They include the checks that the simulator conserves stock and matches theory, so a green run means the engine you are about to use is sound.
2 · Meet FoodNet
FoodNet is a fictional food manufacturer. Material flows from suppliers to plants, then to distribution centres, then to markets. One period is a week.
| Supplier | Character | Cost | Lead time | Fill rate | Capacity |
|---|---|---|---|---|---|
| A | cheap, slow, reliable | 0.90× | 21–24 days | 99 % | 6,000 /wk |
| B | mid-priced, variable | 1.00× | 12–15 days, CV 0.35 | 93 % ± 6 | 6,000 /wk |
| C | expensive, fast, small | 1.30× | 3–4 days | 98 % | 2,000 /wk |
Cost is a multiplier on each SKU's unit cost. Fill rate is the share of each order actually shipped; the rest is never delivered.
Two plants make 10 SKUs. Two of them are chilled, with a shelf life of 21 and 28 days. Total demand is about 8,350 units a week. Normally the plants buy from A and B. C has no regular volume: it is the backup. The FoodNet README explains every field.
To see it, open the Network Studio, which starts on FoodNet. The diagram shows how material flows, the map shows where everything is, and the tables hold every number. Click any node or lane to see its fields with their units and bounds.
# in a Python shell, from backend/ from core.domain.network import load_foodnet net = load_foodnet() # reads and validates data/foodnet/network.yaml net.demand_mean("MKT_N1", "S04") # 440.0 units/week of dry pasta in North 1
3 · Write the policy
This is the intent, as a planner would say it:
"If Supplier B misses more than 10 % of orders for three weeks running, shift ambient volume to Supplier C, don't switch back for a month, and the sourcing agent can do this alone up to 30 % of volume."
As a rule in the DSL, from examples/first_stress_test.py:
fill_b = VarRef(var=Variable.supplier_fill_rate, supplier="SUP_B")
Policy(
id="b-to-c-switch",
scope=Scope(categories=["ambient"]), # ambient SKUs only
rule=SourcingSwitch(
primary="SUP_B", secondary="SUP_C",
trigger=Condition(var=fill_b, op="lt", threshold=0.90, for_periods=3),
switch_back=Condition(var=fill_b, op="ge", threshold=0.90, for_periods=2),
shift_fraction=0.30, # move 30 % of B's volume
min_dwell_periods=4, # "don't switch back for a month"
),
mandate=Mandate(
owner="Planning manager", approver="Head of supply chain",
agent_id="sourcing-agent-01",
autonomy=[AuthorityLimit(kind=LimitKind.volume_shift_pct, value=30)],
... # expires_at: 90 days by default
),
...
)
Four things to notice:
- Every threshold is explicit. "Misses more than 10 %" became
fill rate < 0.90. "Three weeks running" becamefor_periods=3. Nothing is left to the agent's interpretation. - The rule reads only observable state. Try
VarRef(var="competitor_price"): it is rejected, because that variable is not in the vocabulary. - Bounds are enforced.
threshold=1.5fails, because a fill rate lies between 0 and 1. So doesmin_dwell_periods=500. - The mandate is separate from the rule. The rule says what to do. The mandate says how much of it the agent may do without asking: here, at most 30 % of volume.
The incumbent, the rule running today, is plain base stock: order up to 10 days of cover at each DC (12 for chilled SKUs) and 35 days at each plant. It meets the constitution in normal times. The candidate is the incumbent plus the switch. Comparing the two isolates exactly what the switch adds.
4 · Pick the futures
A policy reacts to whatever happens, so you test it against futures that differ. This tutorial uses four of the eight scenarios in the default suite:
| Scenario | What happens |
|---|---|
baseline | Seasonal, random demand; normal lead times and fill rates. |
supplier-b-outage | Supplier B loses 80 % of its capacity from week 20 for 8 weeks. |
correlated-shock | In half the futures, A and B both lose 60 % of capacity, their lead times rise 50 %, and North and Central demand jumps 20 %, all at once, for 8 weeks. |
lead-time-inflation | Supplier B's lead times double, creeping up over 10 weeks from week 20. |
Each scenario is a stack of components, so you can build your own: an outage and a promotion, for example.
5 · Run it
python -m examples.first_stress_test # 300 futures × 104 weeks per scenario, about 10 s
The heart of it is three lines per scenario:
draws = generate(lay, scenario, R=300, T=104, seed=42) # every random draw, made up front
incumbent = simulate(lay, draws, compile_program(lay, incumbent_policies))
candidate = simulate(lay, draws, compile_program(lay, candidate_policies))
Both policies run on the same draws: the same demand, the same lead times, the same supplier failures, week by week. This is called common random numbers. Any difference in the results comes from the policy, not from luck. Run it twice with the same seed and you get identical numbers.
6 · Read the result
The output, with seed 42:
| Scenario | Policy | Fill | Fill P5 | Chilled fill | Cost €/week | Switches |
|---|---|---|---|---|---|---|
| baseline | incumbent | 98.2 % | 97.8 % | 98.5 % | 30,672 | 0.0 |
| candidate | 98.2 % | 97.8 % | 98.5 % | 30,763 | 4.1 | |
| supplier-b-outage | incumbent | 98.0 % | 97.6 % | 98.4 % | 30,576 | 0.0 |
| candidate | 98.2 % | 97.8 % | 98.5 % | 30,764 | 5.4 | |
| correlated-shock | incumbent | 94.1 % | 88.1 % | 96.9 % | 31,020 | 0.0 |
| candidate | 96.6 % | 92.8 % | 97.4 % | 30,977 | 4.6 | |
| lead-time-inflation | incumbent | 96.7 % | 95.2 % | 98.4 % | 30,179 | 0.0 |
| candidate | 96.9 % | 95.6 % | 98.4 % | 30,261 | 4.1 |
How to read it: Fill is the share of demand met from stock in the week it arose, averaged over the 300 futures; weeks 0–7 are left out while the system settles. Fill P5 is the 5th percentile: the fill rate of the worst 5 % of futures. Cost covers purchase, production, transport, holding, ordering, expediting, penalties and spoilage. Switches is the number of sourcing switches per future over two years.
What the numbers say
- Under a correlated shock, the switch earns its keep. Fill holds at 96.6 % against 94.1 %, and the worst 5 % of futures improve by almost 5 points (92.8 % against 88.1 %). Cost goes down, because fewer penalties are paid.
- In a Supplier B outage alone, the incumbent copes. Plants hold 35 days of cover, enough to absorb eight weeks of a weak B. The switch adds 0.2 points of fill for €188 a week more, because Supplier C is 30 % dearer.
- In normal times it changes little, and it is not free. B's normal variability (93 % ± 6) trips the trigger about once a year: 4.1 switches over two years count both switching over and switching back. On average that adds €91 a week, for no gain in service. That is a cue to tune the trigger: try 0.85, or four weeks running.
- It is blind to lead-time creep. When B gets slow but still ships in full, the trigger, which watches fill rate, barely fires (96.7 % → 96.9 %). A second trigger on
observed_lead_timewould catch this. - Chilled SKUs meet the constitution's 97 % in normal times (98.5 %), thanks to the incumbent's 12 days of chilled cover. Under the shock they slip below it, a little less with the switch (97.4 % against 96.9 %).
This is the point of evaluating a policy rather than a plan: the same rule looks almost free in one future, essential in another, and blind in a third. You see all three before it goes live.
7 · Get the verdict
The quick table above is your own reading. PolicyForge also evaluates a candidate formally, against the constitution and three baselines, on all eight scenarios:
python -m examples.evaluate_report # 500 futures per scenario, about a minute
It writes docs/example-report.md (for people) and docs/example-report.json (every distribution, sample and fan chart series, for the interface to come). The report holds:
- A verdict: approve, approve with conditions, or reject, with every reason generated from the numbers.
- A hard constraint breached in more than 5 % of futures under normal conditions means reject.
- Under stress, a breach becomes a condition only when the candidate makes it worse than the incumbent.
- The candidate is then compared with the incumbent on each objective in the constitution's priority order (service, then resilience, cost, carbon), and the first one where they differ decides.
- The constitution table: for each scenario and policy, the share of futures breaching each hard constraint.
- Three baselines on the same futures:
- the incumbent;
- a naive rule: mean demand × lead time, no safety stock, no forecast;
- rolling optimisation: a linear program solved with HiGHS every four weeks over an eight-week horizon, on the expected forecast with safety-stock targets. This is how a planning system works.
- Distributions of every metric: mean, P5, P50, P95, and CVaR (the mean of the worst 5 %). Also regret, measured against the best policy in each future.
For this switch, the verdict is approve.
- It beats the incumbent on service, the first priority: a worst-case fill rate of 95.3 % against 94.6 %, averaged over the eight scenarios.
- The constraints it breaches under stress, the incumbent breaches as well.
- Under the correlated shock, it cuts class A service breaches from 48.8 % of futures to 7.0 %.
Rolling optimisation is the instructive comparator. It runs leaner than the incumbent, and the worst futures expose it: it misses the chilled target in 94 % of futures even in normal times. A plan optimised for the expected future has no answer to the unexpected one.
8 · Try it yourself
- Exceed the mandate. Set
shift_fraction=0.6but leave the mandate at 30 %. The simulator blocks the extra volume and counts an escalation each time the agent wants more than it may do alone. PassSimConfig(on_limit="flag")tosimulateand the extra volume moves anyway, flagged for review. - Catch the slow supplier. Wrap the switch in a
Compositewith a secondSourcingSwitchtriggered byobserved_lead_timeabove 21 days. Does lead-time creep improve? - Make the incumbent leaner. Replace it with
background_policies(lay, dc_days=7, plant_days=21). Fill falls to about 69 % even in normal times. 21 days at a plant does not even cover Supplier A's 21–24-day lead time plus a week of review, so the plant keeps running out and a backlog builds. Backlog is served first, so new demand waits. The order-up-to level has to exceed demand over the lead time, with room to spare. - Add an expedite trigger on the DCs:
ExpediteTrigger(condition=CoverageBelow(days=3), mode="express_road", cost_cap_per_period=4000). How much of the shock's damage does expediting buy back, and at what cost? Run the report and check whether the €20,000-a-month expedite budget holds. - Break FoodNet in the Studio. Start the API (
.venv/bin/python -m uvicorn api.app:appinbackend/) and openhttp://127.0.0.1:8000/app/. Run the quick test, then set Supplier B's capacity to 2,000 and run it again, with the diagram coloured by fill rate. Which markets suffer first, and why those? The Health tab shows the supplier now overloaded. Undo (Ctrl+Z) restores FoodNet exactly. - Watch it in production. Run
.venv/bin/python -m examples.monitor_demo. It deploys the §10 demo policy, starts a live stream, and quietly takes 60 % of Supplier C's capacity from week 20. The assumption "C has spare capacity of at least 1,100 units a week" fires that same week and tops the inbox; the re-test runs on C as observed. Now setSEVERITY = 0.4at the top of the file and run it again: C keeps 1,200 units, the assumption still holds, and nothing fires. The monitor watches the assumption, not the capacity. - Delay the information. The
bullwhipscenario delays what the agents see by two weeks and adds noise. Then trySimConfig(own_orders_known=False), where an agent forgets even its own recent orders. That is the classic beer-game instability, with order swings many times those of demand.