Govern the rule, not the plan
As AI agents take over replenishment, allocation, expediting and supplier switching, the supply chain manager's job shifts from making plans to governing policies: decision rules, plus a mandate that says what the agent may decide alone. PolicyForge is a workbench for that job.
About PolicyForge
A planning system optimises one plan for one expected future. A policy is different: it reacts to whatever future happens. So it has to be judged across many futures, and its robustness, stability and governance count as much as its average cost.
Governing a policy means five things:
- stating intent in plain language;
- turning it into an explicit rule, with every threshold visible and bounded;
- setting the agent's mandate: what it may decide alone and when it must escalate;
- testing the rule against uncertainty before it goes live, against the rule it replaces;
- monitoring the assumptions behind it, so you know when it no longer holds.
PolicyForge follows that sequence:
The core flow. Box outlines show progress: green is built, amber is partly built or next, blue-grey is planned.
Design principles
- The LLM never produces code. It will compile plain language into a typed, bounded policy language, and nothing else. Every rule is a validated object; execution is deterministic Python.
- A closed vocabulary. Rules may read only eight observable state variables (inventory position, on-hand, backorders, pipeline, forecast, observed lead time, supplier fill rate, capacity utilisation). Anything else fails validation.
- Every number has bounds and a unit. Cover and lead times are in days, persistence in weeks, money in EUR. A test checks that every number in every schema is bounded.
- Mandates expire. Ninety days by default, never more than 366 days without review.
- Reproducible, fair comparisons. Every run is seeded. All randomness is drawn before the run, so every policy faces exactly the same futures (common random numbers).
- Honest about limits. Every design decision is logged with its rationale, and every simplification is listed.
What works today (v0.7.0)
PolicyForge is in early development. Its first screen, the Network Studio, is live. The rest is a Python library and an API you run from the command line; the tutorial walks you through it.
The Network Studio built
Policies are tested on a model of the supply network, so that model has to be easy to build and keep up to date. The Studio does it visually:
- Diagram: suppliers → plants → DCs → markets in four columns. Click to add, drag between nodes to draw a lane, drag to reorder, and colour by utilisation, cost, lead time or quick-test fill rate.
- Map: the same network on a world map, with lanes along great circles. Nodes are placed and moved by dragging, and a lane's distance is suggested from the map.
- Tables: SKUs, nodes, lanes and the market × SKU demand grid as spreadsheets. Paste straight from Excel; a bad cell is reported by row and column.
- Inspector: a form generated from the model's JSON Schema, with every field's bounds, units and description. Lead times are shown as a distribution, with their mean and 95th percentile.
- Live checks: field errors at once in the browser. Then the server's checks of how the parts fit together, and a health panel (utilisation against demand, shelf life against lead time), each shown on the element concerned.
- Quick test (a second of simulation with the incumbent), versions with field-level diffs and re-test flags on the policies that depend on them, and a scenario builder with a timeline and a preview of what each scenario changes.
Open the Studio. On this site it edits, checks fields, and opens and saves YAML or JSON files without any server. For reference checks, health, the quick test, CSV files, versions and scenarios, run the API locally and open http://127.0.0.1:8000/app/.
A browser test rebuilds FoodNet from scratch in the Studio, through the tables, the map and the diagram, and checks that the engine loads exactly the shipped network.
Monitoring built
A deployed policy keeps being checked against the world it runs in. A live stream runs the network week by week with the deployed policies. Its true scenario is hidden from the monitors and can be changed during a run: for example, Supplier C quietly losing capacity from week 20. Four kinds of monitor watch what an operations team would see:
- Assumption drift: every assumption of every deployed policy becomes a monitor. "Supplier C has spare capacity of at least 1,100 units a week" fires in the week the capacity drops.
- Input drift: demand per market and lead time per supplier, against what the stress test assumed (CUSUM, with PSI and the KS test as evidence). Lead times are seen as shipments arrive, so drift shows only once the slower shipments land.
- Performance drift: fill rate and cost against the P5–P95 envelope the stress test predicted.
- Mandate escalations: an agent reaching the limits of its mandate.
Thresholds are calibrated on the stress test's own futures, so a quiet world raises a false alarm in at most 5 % of runs.
Alerts go to an escalation inbox, ranked by severity × 1/reversibility × financial exposure, and each alert explains its own score. One call re-runs the stress test on the network as observed (C's capacity as it is now) and returns a new verdict, without touching the approval record.
python -m examples.monitor_demo runs it end to end: the alert comes in week 20, tops the inbox, and the re-test still approves. Even a smaller Supplier C is a better backup than none.
The policy language built
Six rule types: base stock (in units or days of cover), (s, S), allocation (fair share, priority or proportional, with reservations), expedite triggers (projected stockout or low coverage, with a cost cap), sourcing switches (with persistence, a minimum dwell and switch-back), and composites of these. Each rule comes with its mandate, assumptions, the constitution it is tested against, and its provenance. Schemas are exported, and the DSL reference is generated from them.
The registry and API built
Networks, constitutions and policies are stored as versions, and any two versions can be compared field by field.
A policy moves from draft through tested, approved and deployed to retired. Approval is refused unless the latest evaluation of that exact version, against the current constitution and network, says approve, and the refusal says why. Deploying gives the agent one active mandate.
Agents read it at GET /agents/{agent_id}/mandate: limits, escalation rules, prohibitions, the rule, and its expiry. An expired mandate is reported as expired.
Every change goes into a hash-chained audit log: the database refuses edits, and GET /audit/verify finds any made around it. When a network or constitution changes, the policies tested on the old one are flagged for re-test.
The compiler built
Plain language in, a validated policy out. Claude (Opus 5.5 by default) answers only in JSON constrained to the policy schema, and never writes code.
Its answer then goes through three checks:
- the schema's types and bounds;
- the closed vocabulary;
- a deterministic validator: every id must exist in the network, the scope must cover something, and the policy must be tested against the active constitution.
Errors go back to the model for at most two retries. Warnings, such as cover shorter than the lead time or an expedite cap that could exceed the monthly budget, go to the person signing off.
Each compiled policy comes with:
- its assumptions, marked as stated by you or inferred;
- clarifying questions where a missing number would change the rule;
- a plain-language back-translation to compare with your intent;
- its provenance: model and prompt hash.
20 recorded intents run offline, and python -m examples.compile_intent shows the result.
The simulator built
It runs weekly over a supplier → plant → DC → market network, vectorised over replications. It models stochastic lead times per order, supplier capacity and fill rates, finite plant capacity, shelf life and spoilage, backorders or lost sales, information delay and noisy signals, and the mandate's limits (blocked or flagged). 1,000 replications × 104 weeks take about 6 seconds. It conserves stock exactly, and it matches the textbook fill rate of a base-stock system to within 0.03 percentage points.
FoodNet and the scenario library built
A fictional food manufacturer: 3 suppliers, 2 plants, 3 DCs, 6 markets and 10 SKUs (2 chilled). Eight stress scenarios: a quiet baseline, a demand surge, a promotion spike, a supplier outage, lead-time creep, a plant capacity loss, a correlated shock, and noisy, delayed signals.
Evaluation and verdict built
A candidate is compared on the same futures with three baselines: the incumbent, a naive rule, and rolling optimisation: a linear program solved with HiGHS every four weeks, the way a planning system works.
Every metric gets a distribution (mean, P5, P50, P95 and CVaR) and regret against the best policy in each future: service by class, category and market; cost in eight categories; resilience after a shock; bullwhip and order nervousness; escalations; and CO₂.
A constitution table gives the share of futures breaching each hard limit. The verdict (approve, approve with conditions, or reject) states every reason from the numbers. An example report evaluates the tutorial's sourcing switch.
Roadmap
The build goes phase by phase, and each phase starts only when the previous one passes its acceptance tests.
| Phase | What it delivers | Accepted when | Status |
|---|---|---|---|
| P1 Domain and data | The DSL, mandate, assumptions, constitution; FoodNet; schemas | Models round-trip; bad bounds rejected; schema docs generated | done |
| P2 Simulator | Vectorised engine, an executor per rule, 8 scenarios, common random numbers | Conservation; exact reproducibility; 1,000 × 104 under 30 s; analytical fill rate within ±1 pp | done |
| P3 Baselines and evaluation | Incumbent, naive and rolling-optimisation baselines (Pyomo with HiGHS); full metrics; robustness verdict; report | Bullwhip ratio above 1 under delay; the optimiser solves; reports for the default suite | done |
| P4 LLM compiler | Provider interface, offline mock, compiler with retries, validator, back-translation | 20 golden intents compile; out-of-vocabulary references rejected | done |
| P5 Registry and API | Lifecycle, approval gate, hash-chained audit log, agent mandate endpoint, diffs | No approval without a passing evaluation; tampering detected | done |
| P5b Network Studio | Build and maintain networks visually: a diagram, a world map, tables, an inspector generated from the schema, live checks, a quick test, YAML/CSV import and export, versions with diffs, and a scenario builder | FoodNet rebuilt from scratch in the Studio equals the shipped one; errors shown on the right element; versions flag policies for re-test | done |
| P6 Monitoring | Live stream, input, assumption and performance drift monitors, escalation inbox | Assumption drift alerts within 4 weeks of injected drift | done |
| P7 Frontend | Constitution, Author, Stress test, Registry and Monitor screens, around the Network Studio | The full demo runs in the browser | next |
| P8 Hardening | Docker, Makefile, user guide, architecture, limitations | Runs on a clean machine; coverage ≥ 80 % | planned |
Proposed additions
- A leaner incumbent for FoodNet done in P3. 35 days at plants; at DCs 10 days, 12 for chilled SKUs. It meets the constitution in normal times and is fragile enough under shocks for stress tests to tell policies apart.
- A public demo on this site (after P7). GitHub Pages cannot run the Python backend. Two options: precomputed stress-test reports shown in the browser, or the simulator itself in the browser through Pyodide, which ships NumPy.
- Seasonal forecasts done in P3, as an option. On FoodNet they change little (+0.02 points of fill): its seasonal swings are mild, and smoothing tracks them.
- An LLM-written report narrative done in P4. A summary that may cite only numbers present in the report, checked automatically; one retry, then no summary.
Read more
- Tutorial: write a policy, stress-test it, read the result and get the verdict, in about 20 minutes.
- Example evaluation report: the tutorial's sourcing switch against three baselines on eight scenarios.
- Design decisions: 44 so far, each with its context, options and rationale.
- Status: what works, what is stubbed, and the known limitations, phase by phase.
- FoodNet: every field of the synthetic network, with its unit.
All data is synthetic. No real company is described.