Core concepts.

A job is Trawl's unit of work: an asynchronous AI web-extraction task. You submit two required inputs and get back a schema-conforming JSON object:

• json_schema — a JSON Schema describing the shape of the result you want. The top level must be a non-empty object with "type": "object"; bare arrays and primitives are rejected with a 400. To extract a list, wrap it in an object property.
• description — a natural-language description (up to 2,000 characters) of what to extract. This drives the agent; the schema shapes the output.

Optionally, add suggested_urls (up to 50 http/https seed URLs) to point the agent at known sources. Seeds are optional — without them, the agent searches the web itself to find authoritative sources. A job is not “URL in, data out”; it's “question in, structured answer out”, with URLs as an optional hint.

When the job finishes, the extracted data is stored on the job row in the result field — you can always read it back with a plain GET. Attaching a webhook is optional (see below).

There is no Trawl SDK yet — the HTTP API is the integration surface today, and generated SDKs will follow the pattern of the other ProductCraft products later. See the quickstart for the first call and the API reference for every field.

Creating a job returns 201 immediately with status queued. A worker picks it up, flips it to running, performs the extraction, and the job lands in exactly one terminal state.

Terminal states — succeeded, failed, cancelled — are frozen. Late or duplicate worker results against a terminal job are ignored, so a cancel is never clobbered by a slow extraction finishing afterwards.

Timeouts and the safety net

Each job carries a wall-clock budget, timeout_s (default 120 s, configurable 1–3,600). A background reaper runs every 5 minutes and fails anything stuck:

• A running job is failed once it has been running longer than timeout_s plus a 60-second grace period.
• A queued job that no worker ever picked up is failed after 30 minutes.

Reaped jobs get error: "job exceeded its time budget (reaped)" and fire job.failed to their webhook if one is attached. A stuck job always reaches a terminal state — you never poll forever.

Cancellation

POST /v1/workspaces/:workspace_id/jobs/:id/cancel cancels a queued or running job and returns the job with status cancelled and completed_at stamped. Only queued or running jobs can be cancelled — terminal jobs are rejected. Cancellation flips the record; an extraction already in flight runs out its own budget and its late result is discarded. Cancelled jobs fire no webhook event.

Trawl's worker is not a page scraper with a CSS-selector config. It's an LLM agent driving a real headless Chromium browser — it navigates, reads rendered pages, follows links, and decides its own next step. Because it's a real browser, JavaScript-heavy pages and SPAs render normally.

Source strategy

• With suggested_urls: the agent starts from your seeds (the first is already open when the run begins) and browses on from there.
• Without seeds: the agent uses web search to find authoritative sources, then browses to them.
• When a page fails — dead link, DNS error, HTTP error — the agent sees the navigation error and tries other sources rather than aborting. A job only fails outright when it can't finish within its step and time budget.

Cross-page memory

The agent distills each page it reads into durable memory before moving on, so a single job can legitimately aggregate data from several pages. Every extra page costs steps and time — size timeout_s accordingly (the 120 s default suits jobs touching a handful of pages; raise it for research-style jobs).

Schema enforcement — twice

Your json_schema is enforced at the moment the agent produces its answer and re-validated before the result is recorded. Output that doesn't conform fails the job with result failed schema validation — malformed or non-conforming JSON is never delivered. The agent is also explicitly instructed to use null for values it cannot find rather than inventing them, so make hard-to-find fields nullable (or leave them out of required). Full guidance in schema design.

Honest failure

If the agent finishes but assesses that it could not actually satisfy the request, the job fails with agent could not complete the extraction — even if its payload happened to validate. “Couldn't find it” is never shipped as success.

Boundaries

• Public pages only — login walls and authenticated content are out of scope today.
• Every navigation is checked against private, loopback, link-local, and cloud-metadata addresses; the agent cannot be steered at internal infrastructure. Seed URLs pointing at private addresses are rejected at submit time with a 400.

There are two ways to learn a job's outcome, and they are complementary, not either/or:

• Polling. The result lives on the job row. GET /v1/workspaces/:workspace_id/jobs/:id until status is terminal, then read result (or error). Simplest integration; no receiver endpoint needed.
• Webhooks. Create a webhook (a workspace-level resource), pass its id as webhook_id when creating jobs, and Trawl POSTs a signed event to your HTTPS endpoint when each job reaches succeeded or failed.

Exactly two event types exist: job.succeeded and job.failed. There is no per-event subscription — a webhook fires for whichever jobs reference it. Timed-out (reaped) jobs fire job.failed; cancelled jobs fire nothing.

Every delivery is signed with an HMAC-SHA256 signature in the X-Trawl-Signature header (t=<timestamp>,v1=<hex> — the same Stripe-style recipe as ProductCraft Mail and Waitlist webhooks), using a per-webhook signing secret shown exactly once at creation. Verification recipe and receiver patterns are in the webhooks guide.

Delivery semantics — one attempt, durable record

Trawl makes one delivery attempt per event — there is no retry-with-backoff schedule. This is safe because the webhook is a notification, not the data channel: a missed delivery is never lost data, since the result stays on the job row. Instead of retries you get observability and protection:

• Every attempt (status code, response snippet, error, latency) is recorded and inspectable via the webhook's deliveries endpoint.
• After 20 consecutive failures, or a failure streak lasting 24 hours, the webhook is auto-disabled and further dispatches to it are dropped. Re-enable it with PATCH { "active": true }, which also clears the failure streak.
• Deliveries are at-least-once in rare crash-recovery cases — dedupe on the X-Trawl-Event-Id header.

Webhook URLs must be HTTPS and publicly resolvable at delivery time; each attempt has a 10-second timeout and any 2xx response counts as success.

Trawl is workspace-scoped, like every ProductCraft product. All routes live under /v1/workspaces/:workspace_id/..., and every query is filtered by that workspace — a caller authenticated against one workspace can never read or mutate another's jobs or webhooks (cross-workspace ids surface as 404, never data).

Authentication

Authenticate with a workspace API key (PAK) minted at console.productcraft.co, sent as Authorization: Bearer pcft_live_.... Your platform session also works for ad-hoc calls, but PAKs are the integration credential.

Service activation

The workspace must have the trawl service enabled. If it isn't, every route returns 403 with code SERVICE_NOT_ENABLED before any data is touched.

Permissions

Every route is gated by a trawl.* action. A request whose credential doesn't grant the required action gets a 403 naming the missing permission.

Workspace owners hold every permission; members default to read-only (trawl.read, trawl.list, trawl.webhook.read). Scope PAKs to the minimum set your integration needs — a poller that creates jobs and reads results needs trawl.create and trawl.read, nothing else.