Apache-2.0 · one binary · v0.1.2
forge fans a giant JSONL of prompts across N OpenAI-compatible engines — vLLM, SGLang, llama.cpp, Ollama, or a hosted API — on your own spot GPUs. The GPU work stays in the engines. forge is only the orchestration shell: work queue, sharding, retries, spot-interruption handling, backpressure, result aggregation.
§01 — the claim, tested
A 300-item batch at concurrency 8, killed with kill -9 the way
a spot instance dies. No output file to join back against the input on
keys, no bespoke checkpoint recipe. The lease queue is the resume
state, so it answers the question directly.
at the kill
8 live leases
The in-flight items are still leased to a process that no longer exists. 288 done, 4 never started.
after the lease TTL
8 orphaned
Those leases lapse and the items become interrupted: true — the fingerprint of a machine that died.
reclaimable
12
4 pending plus 8 orphaned. Exactly the set resume re-dispatches, join-free.
after resume
300/300
forge verify: every input custom_id terminal. 0 missing, 0 duplicates.
Why that is safe rather than lucky. A lease is fenced by
(custom_id, attempt), so a result posted for a stale attempt
by a resurrected worker is rejected — an item cannot commit twice.
An item is only done once its result is durably stored, so a
crash between store and acknowledge re-runs an idempotent store rather
than losing the work. resume re-dispatches exactly
pending + orphaned; done rows are never
re-leased.
§02 — heterogeneous by default
Dispatch used to be wave-based: lease a batch, dispatch it, wait for all of it, lease the next. That puts a barrier at every batch boundary — so one slow worker gates the whole fleet.
400-item batch, concurrency 8 — two fast boxes plus one weak one
Read the top two rows together: under wave dispatch, adding a third machine made the fleet slower than one box on its own. That is the head-of-line block, measured. Dispatch is now a sliding window with per-worker free-slot accounting — every completion immediately frees a slot topped up with a fresh lease, and a saturated worker is skipped rather than queued behind.
| fleet | wave dispatch | sliding window | gain |
|---|---|---|---|
| 1 fast box | 10.86 items/s | 12.99 items/s | +20% |
| 2 fast boxes | 21.08 items/s | 25.76 items/s | +22% |
| 2 fast + 1 weak | 9.15 items/s | 25.59 items/s | 2.8× |
In the mixed run the weak box chews its own 49 items at its own pace while the fast boxes stay saturated — zero 429s, engine-side peak queue depth of 4. Throughput is the sum of what each engine can do rather than a multiple of the slowest. A naive sequential client against the same single box manages 1.58 items/s.
§03 — when you get the numbers wrong
Misdeclare a worker's concurrency eight times over the engine's real limit. A naive concurrent client permanently loses work; forge converges on the actual cap.
| client — 200 items, declared concurrency 64 | completed | permanently lost | engine 429s |
|---|---|---|---|
| naive 64-thread, fixed 1 s retry, 10 attempts | 158/200 | 42 | 1,107 |
| forge — AIMD + header cooldown | 200/200 | 0 | 72 |
AIMD halves the in-flight window on each 429 burst, and a shared per-worker
cooldown honours Retry-After so the whole fleet backs off in
lockstep rather than each thread discovering the limit alone. That is
94% less rejected traffic, and nothing dead-lettered.
The coordinator is nearly free
Saturating three remote worker nodes over real VPC HTTP at 26.89 items/s, forge itself peaked at 9.7 MB RSS and 0.9% of one core (2% peak). It adds scheduling, durability and accounting without meaningfully taxing the box it runs on.
Against a sequential client
16.1× on that remote fleet, and about 90% of the fleet's theoretical 30/s ceiling — with no tuning. Every run: all items done, zero dead-letters.
examples/engine_sim.py — a mock
that models a hard concurrency cap, queueing beyond it, 429 with
Retry-After, and per-request jitter — not against real
GPUs. That is deliberate: the harness is the variable under test, not GPU
throughput, which lives in vLLM and SGLang and is out of forge's scope.
These figures tell you what the coordinator does with a fleet. They tell you
nothing about how fast your model runs.
§04 — the arbitrage, priced by you
Offline batch inference is roughly five to ten times cheaper per token than
online serving, and spot GPUs cut another 60–90% off on-demand.
forge cost turns the real usage forge
captured into the invoiceable difference — and it never guesses a
price.
Both sides of that sum are yours. forge does not shop for GPUs and
does not know what spot costs today — it multiplies the rate you give
it by the hours the run took, and prices the same tokens against the online
rate you also give it. Pass --gpu-cost instead if you already
have one figure for the run. What forge contributes is the token
side: those totals are summed from the real usage the
engines reported, not estimated from your input file.
It is honest in the other direction too: on a job too small to amortise the GPU bill the saved figure goes negative, which tells you the batch was not worth a dedicated fleet. Cached-prompt tokens price at the full input rate unless you pass a discounted rate for them — conservative by default, so the saving is never overstated.
Against the alternative
Cloud Batch APIs hide the operational pain but give a flat ~50% discount, with no choice of model weights, data locality or GPU economics. forge is the other trade: you keep all three, and you own the procurement decision — forge runs the shell, it does not buy your GPUs.
Keep the prefix cache hot
--prefix-bucket reorders the input by shared (model, system-prompt) so the engine's automatic prefix cache actually stays warm and those cached_tokens materialise instead of being theoretical.
§05 — the boundary
forge is a single homogeneous fan-out of independent items across interchangeable workers. The boundary is enforced by the data model rather than by discipline — there is nowhere in the core to express what a workflow engine exists to express.
Not a workflow engine
No task graph, no depends_on or fan-in, no cron or triggers, no YAML topology. Arbitrary task graphs with inter-task dependencies are dagron's job. forge can hand a flat plan to dagron; the arrow points outward only.
Not a provisioner
“Get me 400 spot GPUs across 12 regions” is out of scope. forge consumes interruption signals and re-queues at item granularity. It never launches, tears down or autoscales an instance.
Not an inference engine
The GPU heavy-lifting stays in vLLM, SGLang, llama.cpp or Ollama, where it belongs. forge never batches on the wire either — the engine's continuous batching packs concurrent requests.
Getting it
A signed multi-arch image, or build the static binary from source. The v0.1.2 GitHub release carries no attached binaries yet.
Results can land in object storage (S3, GCS, Azure) rather than a local file, and forge can serve the OpenAI Batch REST API surface so existing clients point at it unchanged.
Apache-2.0
The kill test, the fleet comparison and the overload run all ship as scripts and mocks you can run in three terminals. The regression that caused the head-of-line block is pinned by a test.