# Using Async Jobs

An **async job** lets a client start a plugin call and get its answer later,
instead of holding an HTTP request open for the minutes a plugin may take.

Jobs are **optional**. A plugin call without the headers below behaves exactly
as it did before: synchronous, same body, same status codes. Only plugin routes
take jobs; ops and primitives stay synchronous.

API reference: [docs/reference/API.md](../reference/API.md#jobs).

---

## The problem jobs solve

Every call is synchronous HTTP. A `get-following` that scrolls for 10 minutes
holds the client's connection for 10 minutes, and nothing keeps its result: if
the client (or a proxy in front of the workers) dies mid-call, the plugin still
runs to the end, but its answer is lost.

With a job the worker answers `202` at once and runs the plugin in the
background. The result then reaches the client in one of two ways, and is kept
no longer than that way needs.

---

## Two ways, never both

| | Webhook | Polling |
|---|---|---|
| Header | `X-Callback-Url: <absolute http(s) URL>` | `X-Result-Ttl: <seconds>`, `0 < ttl ≤ 600` |
| Who moves | Penelope delivers the result to the client's URL | The client fetches it at `GET /api/v1/jobs/{job_id}` |
| When | at 0, 1, 5 and 10 minutes after the job ends, until one is answered `2xx` | any time until `ttl` seconds after the job ends |
| Data deleted | once delivered, or after the 4th failed attempt (or a `4xx`) | when `ttl` runs out |

Sending both headers answers `400`, as does a bad value; nothing starts.

```bash
BASE=http://localhost:5000/api/v1
AUTH="Authorization: Bearer $PENELOPE_KEY"

# webhook: Penelope delivers the result to the client
curl -s -X POST "$BASE/main/plugin/instagram/open-account" -H "$AUTH" \
  -H "X-Callback-Url: https://scheduler.lan/hooks/42" \
  -H "Content-Type: application/json" -d '{"handle": "alice"}'

# polling: Penelope holds the result 10 minutes after the job ends
JOB=$(curl -s "$BASE/main/plugin/instagram/get-following" -H "$AUTH" \
  -H "X-Result-Ttl: 600" | jq -r .job_id)
curl -s "$BASE/jobs/$JOB" -H "$AUTH"     # "state": "running", then "done"
```

---

## Starting

The worker runs, before answering, every check a plain call would run first:
the bearer key (`401`), the headers and parameters (`400`, or `422` for a
malformed body), the page (`404`), its lease (`423`). Then it claims the page
for the job and answers:

```json
{
  "success": true,
  "page_id": "main",
  "job_id": "5b1d82d4dc257267",
  "status_url": "/api/v1/jobs/5b1d82d4dc257267",
  "ttl": 600.0,
  "callback": null
}
```

`ttl` is the polling hold (`null` for a webhook), `callback` the webhook URL
(`null` for polling).

### One job per page

A page runs at most one job. The worker answers `409` at once, without starting
anything, when the page:

- already runs a job (`"Page main already runs job 5b1d…"`), or
- is not idle: a plain call is running on it.

Other callers on the page (plain plugin calls, ops, primitives) wait for the
job as they wait for any busy page: up to 30 s, then `409`. The memory purge
skips it. There is no queue: to run several jobs at once use several pages (up
to `MAX_PAGES`), to run them one after another wait for each to end.

---

## Webhook

When the job ends, the worker POSTs the job, as `GET /api/v1/jobs/{job_id}`
would show it, to `X-Callback-Url`:

```json
{ "success": true, "page_id": "main", "job_id": "5b1d…", "state": "done",
  "plugin": "instagram/open-account", "started_at": 1792844301.50, "finished_at": 1792844331.57,
  "expires_in": 609.9, "status": 200, "result": { "success": true, "page_id": "main", "...": "..." },
  "callback": "https://scheduler.lan/hooks/42",
  "delivery": { "attempts": 1, "delivered": false, "error": null } }
```

with these headers:

| Header | Value |
|---|---|
| `Content-Type` | `application/json` |
| `X-Penelope-Job` | the job id |
| `X-Penelope-Attempt` | `1` to `4` |
| `X-Penelope-Signature` | `sha256=` + hex HMAC-SHA256 of the raw body, keyed with `PENELOPE_KEY` |

`status` and `result` are exactly what the plain call would have answered (see
[Polling](#polling)).

**Attempts.** The four attempts are at 0, 1, 5 and 10 minutes after the job
ended. Any `2xx` delivers it. A network error, a timeout (10 s), `429` or a
`5xx` waits for the next attempt; any other answer (a `4xx`) gives up at once.

**Deletion.** Once delivered, the job is deleted; a `/media/` link in its
result stays valid 5 more minutes, for the download. Given up, the job and its
media are deleted at once. Either way the worker keeps nothing past about 10
minutes after the job ended.

**Receiving.** Answer `2xx` only once you have stored the body. The same job
can arrive twice (a stop between the POST and its answer): key on
`X-Penelope-Job`. Verify the signature before trusting the body:

```python
import hashlib, hmac

def verify(body: bytes, signature: str, key: str) -> bool:
    expected = "sha256=" + hmac.new(key.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)
```

While a webhook job runs, or waits for its next attempt, `GET
/api/v1/jobs/{job_id}` shows it (with `delivery` so far); once delivered or
given up it answers `404`.

The webhook leaves from the host's own address, not through the persona's
browser proxy. The worker POSTs wherever the caller says: any client with the
key can point it at any URL the host reaches, so keep the key with trusted
clients only.

---

## Polling

`GET /api/v1/jobs/{job_id}` (auth) while it runs:

```json
{ "success": true, "page_id": "main", "job_id": "5b1d…", "state": "running",
  "plugin": "instagram/get-following", "started_at": 1792844301.50, "finished_at": null,
  "expires_in": null, "status": null, "result": null, "callback": null, "delivery": null }
```

and once it ends (`state` is `done`, or `interrupted`: see [Restarts](#restarts)):

```json
{ "success": true, "page_id": "main", "job_id": "5b1d…", "state": "done",
  "plugin": "instagram/get-following", "started_at": 1792844301.50, "finished_at": 1792844901.57,
  "expires_in": 594.9, "status": 200, "result": { "success": true, "page_id": "main", "...": "..." },
  "callback": null, "delivery": null }
```

- `status` and `result` are exactly what the plain call would have answered:
  the same response model, or the error envelope with its status
  (`{"success": false, "error": "...", "page_id": "main"}` with `409`, `408`,
  ...). The outer `success` only says the poll worked: check `status` and
  `result.success`.
- The result can be read any number of times until `expires_in` runs out
  (`ttl` seconds after the job ended, 600 at most); then it is deleted and the
  job answers `404`, as an unknown id does. A `/media/` link in it stays valid
  as long.
- Polls are not work: they are left out of the request log, but they do count
  as client activity for the waker (see [Sleepy workers](#sleepy-workers)).

---

## Jobs and the rest

### Leases

A job honours leases like a plain call: send `X-Lease` with the call that
starts it. The job runs under that lease, which never expires while the job
works the page (see [USING_LEASES.md](USING_LEASES.md#expiry-and-renewal)).
Neither polling nor the webhook renews it: its TTL (600 s at most) starts again
from the moment the job ends, so the next call of the flow must come before it
runs out.

### Media links

A plugin that produces a file returns a one-shot `/api/v1/media/{token}` link,
normally dropped after 300 s. A job keeps the file as long as it needs the
link: for polling until the result expires, for a webhook until 5 minutes after
delivery (deleted at once if given up). The files sit on disk (the persona's
`store/media/`), not in RAM, and count against the media cap (`MEDIA_MAX_MB`,
4 GB of unclaimed files) all that time. The result's own `expires_at` field
still shows the original 300 s.

### Sleepy workers

A job counts as a request in flight until it ends, and a webhook job until it
is delivered or given up (up to 10 minutes later): the waker never stops a
`sleepy` worker or recycles an `always` one under it. A polling job stops
counting when it ends: the worker's idle clock starts, and a `sleepy` worker
stops after its idle timeout (1 minute by default). Its held results stay on
disk until their `ttl`: to poll a stopped worker, `start` it again through the
waker ([WAKER_API.md](../reference/WAKER_API.md#post-apiv1personasnamestart))
and poll as usual.

### Request log and workload

The call that starts a job logs its own request (status `202`, a few ms), and
the job logs a pair of rows of kind `job`: the plugin's route, the status the
plain call would have answered, and how long the plugin really ran. The waker's
workload counts the job and skips the `202`.

---

## Restarts

Jobs and media links live in the persona's store, `store/penelope.db` (SQLite)
and `store/media/`, so a worker restart (an update, a recycle, a crash, an idle
stop, a new proxy session) keeps them, within their deadlines. Leases stay in
memory and are gone ([USING_LEASES.md](USING_LEASES.md#when-leases-disappear)).

The plugin's work does not survive: the browser and its pages restart with the
worker. At the next start every job still running becomes `interrupted`:

```json
{ "state": "interrupted", "status": 503,
  "result": { "success": false, "error": "job interrupted: the worker stopped before it ended", "page_id": "main" } }
```

and counts as ended at that start: a polling one is held `ttl` from then, a
webhook one gets its four attempts from then. A webhook job the stop caught
between attempts resumes at its next attempt; attempts missed while the worker
was down collapse into one. A job past its deadline when the worker comes back
(10 minutes for a webhook, `ttl` for polling) is deleted without a last try.

An interrupted job may have done part of its work on the site before the stop.
Check the page's state (the family's state reader) before starting it again.

Known limits:

- No queue, no cancel: a running job runs to its end, or to the next stop.
- At most 1000 jobs held at once (running or holding a result); one more
  answers `503`.
- Expired jobs and files are swept every minute.

---

## Implementation notes

| Piece | Location |
|---|---|
| Headers, page claim, run, webhook attempts, start-up recovery | `worker/lib/jobs.py` — `options`, `submit`, `_run`, `_deliver`, `_post`, `recover`; `WEBHOOK_ATTEMPTS_S`, `MAX_TTL_S` |
| SQLite store, sweeps | `worker/lib/store.py` — `store/penelope.db` (tables `jobs`, `media`), `sweep_loop` |
| Plugin call → job | `worker/routers/plugin.py`, the generic handler |
| Polling | `worker/routers/jobs.py` |
| Page claim | `ManagedPage.job_id`, `job_allows` and `try_acquire` in `worker/lib/page_management.py`; `current_job` read by `BrowserNavigator._get_page_to_use` |
| Media on disk, kept | `worker/lib/media_store.py` — `put`, `hold`, `expire`, `drop`, `claim`, `sweep`, `drop_orphans` |
| Models | `worker/models/jobs.py` — `JobAcceptedResponse`, `JobResponse`, `JobDelivery` |