Developer documentation
Build against the specification.
A key, the Python SDK, and the loop that turns one executable task definition into a training signal, a sealed evaluation, and a release decision you can hand to someone who was not in the room.
01
Get an API key
One workspace and one key per email. Issued immediately during the beta.
Store it now. We keep only a hash, so this value cannot be shown again — if it is lost we can revoke it and issue a new one, nothing more.
curl -X POST https://formalengines.com/api/signups -H 'content-type: application/json' -d '{"email":"you@example.com"}'Same endpoint, same response: 201 with the key, 409 if the email already has a workspace.
02
Install the SDK
Python 3.12 or newer. The base client is httpx, Pydantic, and the shared contracts.
pip install formal-engines --extra-index-url https://formalengines.com/pypi/simple/Wheels are served from this domain. formal-engines is not on public PyPI yet, so the extra index is required.
pip install 'formal-engines[authoring]' --extra-index-url https://formalengines.com/pypi/simple/The authoring extra adds environment packaging and local validation. It is imported only when you callspecifications.publish(path=…), as the quickstart below does.
03
Quickstart
Publish a specification, train against it, and export the release evidence.
import os
from formal_engines import FormalEngines
from formal_engines.types import Gate, ModelRef, TrainingPlan
# The client reads both of these from the environment; export them in your shell instead if you prefer.
os.environ["FORMAL_ENGINES_API_KEY"] = "fe_live_your_key"
os.environ["FORMAL_ENGINES_BASE_URL"] = "https://formalengines.com/api"
fe = FormalEngines()
# Package a local verifiers environment and publish it as an immutable specification version.
spec = fe.specifications.publish(
path="./environments/support_agent",
train_split="train",
eval_split="sealed_eval",
)
# Baseline: score the untrained model on the sealed split.
baseline = fe.evaluations.run(
specification=spec,
model=ModelRef(name="Qwen/Qwen3.5-4B"),
split="sealed_eval",
).wait()
# GRPO against the same specification that defines the reward.
training = fe.training.start(
TrainingPlan(
specification=spec,
base_model="Qwen/Qwen3.5-4B",
algorithm={"type": "grpo"},
max_steps=100,
batch_size=128,
rollouts_per_example=8,
)
)
checkpoint = training.wait().best_checkpoint
# Candidate: the trained checkpoint, scored on the split training never saw.
candidate = fe.evaluations.run(
specification=spec, model=checkpoint, split="sealed_eval"
).wait()
# Gates turn the two runs into a recorded pass or fail.
decision = fe.releases.decide(
candidate=candidate,
baseline=baseline,
gates=[
Gate.metric("task_success", minimum=0.82),
Gate.regression("policy_violation", maximum_delta=0.0),
],
)
# Signed bundle: specification version, run ids, metrics, gate results.
decision.export("release-evidence.json")publish packages the directory, creates the version, validates it, and publishes it in one call. Both run handles expose .wait(), .events(), .metrics(), and.cancel(); polling, cursors, retries, and idempotency stay inside the SDK.
04
How it works
Versions, runs, sealed splits, gates, evidence.
A specification is an executable environment — task state, tools, permitted actions, constraints, and the verifiers that decide what counts as success — published as an immutable version. Training and evaluation both address that version by id, so a reward and a score always come from the same definition of correct. Candidate checkpoints are scored on a sealed split that training never sees, which is the only thing that makes the resulting number worth quoting. Gates state the release condition as data: an absolute floor on a metric, or a no-regression bound measured against the baseline run. A release decision applies those gates to a candidate and baseline pair and freezes the outcome into an evidence bundle — specification version, run ids, metrics, gate results, and an HMAC-SHA256 signature over the canonical payload — so the record survives the argument about whether the model got better.
The product story05
API at a glance
Seven resource groups under one base URL.
| Resource | Purpose |
|---|---|
/v1/specifications | Create a specification, add versions, validate one, publish it. A published version is immutable and is what every run refers to. |
/v1/evaluation-runs | Score a model or checkpoint on one split of a version. Metrics and traces hang off the run. |
/v1/training-runs | Start post-training against a specification, then read metrics, logs, and checkpoints. |
/v1/runs/{run_id}/events | One server-sent event feed for both run types. Events carry a monotonic sequence, so reconnecting with the last cursor loses nothing and repeats nothing. |
/v1/release-decisions | Apply gates to a candidate and baseline pair, record the verdict, and export the evidence file. |
/v1/evidence-bundles | Retrieve the signed bundle behind a decision. |
/v1/artifacts | Content-addressed transfer of packaged environments. specifications.publish() uses it for you. |
- Base URL
https://formalengines.com/api- Auth
Authorization: Bearer fe_live_…on every request. Keys are workspace-scoped.- Idempotency
- Every mutating request accepts an
Idempotency-Key. Replaying a key returns the original run instead of starting a second one.
06
Private beta
What you can expect from the shared compute pool right now.
Formal Engines is in invite-scale private beta. A signup gives you a workspace and a key straight away, but every run is scheduled on a small shared beta compute pool: training jobs queue, and large ones may not be admitted at all. If you need dedicated capacity, a particular accelerator, or a model size beyond a few billion parameters, write to us with the task and the shape of the workload before you start. Your specifications, traces, and checkpoints stay in your workspace.
hello@formalengines.com