Policy & quotas

layered resource caps + a reservation ledger — what a run is allowed, decided before it runs

What the policy engine does

Every submit passes one admission gate: it clamps the run’s resources to the effective ceilings and rejects it if the org/user is already at its concurrency cap. Caps come from four layers that fold in precedence:

platform
env-derived deployment seed (lowest precedence)
org
per-tenant policy
project
per-project policy
user
per-user policy (highest precedence — wins)

For each axis, the highest-precedence layer that sets it wins, and the result records provenance (which layer produced each cap). An axis no layer sets stays permissive. With no policy doc, only the platform seed applies — a deployment that never sets policy behaves exactly as before.

An explicit POLYGON_MAX_* env var is the operator’s final word: it overrides any layer on that axis.

The six axes

A policy layer (Caps) is a sparse object — only the axes it sets appear; the rest fall through. Platform-seed defaults shown:

max_cpu
CPU cores per run  ·  default 128
max_ram
RAM GB per run  ·  default 1024
max_gpu
GPUs per run  ·  default 8
max_timeout_sec
wall-clock cap per run  ·  default 86400
max_concurrent_org
in-flight runs per org  ·  default 10
max_concurrent_user
in-flight runs per user  ·  0 = no per-user cap

The first four are clamped (an over-cap ask is quietly reduced, with a note). The two concurrency axes are rejected with a 429.

The reservation ledger & 429

Concurrency is enforced by a reservation ledger: a submit reserves a slot keyed by the run’s id, and the slot is released when the run reaches a terminal phase. Because the reserve is atomic at the cap boundary, two submits racing at the limit can’t both slip through — the loser gets 429. Pipeline stages and sessions reserve the same way, so they count against the same quota.

Preflight a submit (see a quota hit before it happens)

GET/POST /v1/policy/preflight runs the exact admission decision a submit would make — clamps + the live ledger counts — without enqueuing. Same auth as submit (runs:submit). Use it to grey out fields or check before a big sweep.

curl "$BASE/v1/policy/preflight?stack=mujoco&cpu_cores=200&gpu_count=2" \
  -H "Authorization: Bearer slt_…"

Here CPU is clamped to the cap and the org is already at its concurrency limit, so allowed=false with the 429 a real submit would return:

{
  "org": "org_acme", "stack": "mujoco",
  "allowed": false,
  "clamped": { "cpu_cores": 128, "ram_gb": 0, "gpu_count": 2, "timeout_sec": 0 },
  "clamp_notes": [ "cpu 200→128 (POLYGON_MAX_CPU)" ],
  "reason": "org concurrency limit reached (10/10)",
  "http_status": 429,
  "concurrency": { "org_inflight": 10, "org_limit": 10,
                   "user_inflight": 3, "user_limit": 0 }
}

Because preflight reads the same ledger the gate uses, its verdict matches the real submit exactly. clamp_notes is never null; http_status is 0 when allowed.

See & set an org’s policy (admin)

GET /v1/policy returns the resolved view: the platform seed, the stored org/project/user layers, and the folded effective caps with per-axis provenance. Admin-gated (operator admin token, or an org owner/admin confined to its own org — a cross-tenant ?org= is 404).

# view (superuser may target any org with ?org=)
curl "$BASE/v1/policy?org=org_acme" -H "Authorization: Bearer $POLYGON_ADMIN_TOKEN"

# set org + per-project + per-user layers (unset axes fall through)
curl -X PUT "$BASE/v1/policy?org=org_acme" \
  -H "Authorization: Bearer $POLYGON_ADMIN_TOKEN" -H "content-type: application/json" \
  -d '{
    "org":      { "max_concurrent_org": 20, "max_gpu": 4 },
    "projects": { "wing-study": { "max_gpu": 8 } },
    "users":    { "usr_intern": { "max_concurrent_user": 2, "max_gpu": 0 } }
  }'

The platform layer isn’t settable here (it’s the env seed). An empty doc removes all refinements. The policy engine is enabled per deployment — otherwise these routes return 503.

Preview as a user

GET/POST /v1/policy/preview resolves the effective caps for a named subject (and optional project/stack) without enqueuing — the admin “what would this person actually get?” tool. It folds the same layers the live gate uses. With a stack, it also returns the EffectiveManifest — the stack’s config surface intersected with the caps.

curl "$BASE/v1/policy/preview?org=org_acme&subject=usr_intern&project=wing-study&stack=mujoco" \
  -H "Authorization: Bearer $POLYGON_ADMIN_TOKEN"
# → {
#   "org_id": "org_acme", "subject": "usr_intern", "project": "wing-study",
#   "effective": { "max_gpu": 0, "max_concurrent_user": 2, "max_concurrent_org": 20, …,
#                  "provenance": { "max_gpu": "user", "max_concurrent_org": "org", … } },
#   "effective_caps": { "MaxGPU": 0, "MaxConcurrentUser": 2, … },
#   "stack": "mujoco",
#   "effective_manifest": { "max_cpu": 128, "max_gpu": 0, … }
# }

Here the intern’s user layer pins max_gpu: 0 (provenance user) even though the project allows 8 — the highest-precedence layer that sets the axis wins.