PLATFORM_V2_CONNECTORS

Platform v2: dispatch connectors and account secrets

This guide describes the checked-in v2 scaffold, not a deployed service. The production Python Worker, its existing queues, auth, and scraper adapters are unchanged. No v2 Cloudflare resources, migrations, subscriptions, or secrets have been provisioned by these code changes. Run the account commands below only after separately approving their cost and effects.

Why the connector no longer needs supervisor edits

Ordinary Worker service bindings are enumerated in the calling Worker’s Wrangler config. Adding a new service binding therefore requires changing and redeploying the supervisor. The v2 supervisor instead declares one Workers for Platforms dispatch namespace binding, V2_CONNECTORS. The namespace resolves the script name harvy-<connector-id> at runtime. There is also one producer/consumer Queue, harvy-v2-work, rather than a new producer binding for every connector. These are separate v2 resources; never point them at the production v1 Worker.

authenticated POST /v2/runs
  → validate explicit environment + bounded input
  → check registered, enabled, capability-verified connector in v2 D1
  → Workflow: namespace.get("harvy-<id>") → POST /v2/discover
  → validate item IDs, URLs and scope → persist v2 D1 items → shared Queue
  → supervisor queue consumer: claim D1 lease → namespace.get("harvy-<id>") → POST /v2/process
  → connector writes v2 R2 artifact and returns its path
  → supervisor emits v2 K2 event, finalizes D1 item/run and acknowledges Queue message

The connector is not automatically approved just because its code exists or was deployed. Discovery and work require an explicit D1 registration and activation. ArtifactFS/Artifacts may later trigger builds from versioned source, but neither Artifacts nor Worker initialization creates bindings or approves a connector. No Artifacts integration is configured here.

Contract and trust boundaries

Each v2 connector implements GET /health (JSON connector must match its ID), authenticated POST /v2/discover (returns bounded items with connector, run_id, environment_id, HTTPS url and source_url, kind), and authenticated POST /v2/process (takes the claimed item and returns { "artifact_path": "...", "record_count": 0 }; count is optional). /v2/process replies 422 for permanent source/item errors and 503 for transient errors. The shared consumer retries transient failures with the existing D1 lease and at-most-eight-attempt logic. After the last attempt or a permanent error, it marks the item failed in D1 and acknowledges the message; the configured Queue DLQ catches delivery/handler failures, not these application-level failures. Invalid or permanent items cannot be silently routed to a different connector. R2 writes should be safe to repeat because Queue delivery and K2 publish may retry.

platform-v2/shared/contracts.ts validates all run IDs and explicit environment IDs; existing JudyRecords, Truth Social and SAM.gov run inputs keep their source-specific validators. A new connector’s generic JSON input is capped at 16 KiB at the supervisor, not considered a source-specific security validator: implement bounded inputs and an HTTPS origin/path allowlist in its own discovery and process handlers. The supervisor caps discovery at 1,000 items; the connector may need to break larger runs into smaller batches. Connector code must reject cross-source URLs, embedded credentials, unauthorized scopes, and malformed inputs. Do not treat a person’s name in a source record as verified identity. Public progress is aggregated from shared-scope events only; private artifacts must not be placed in a public cache.

The shared supervisor token authorizes the registration and run API. A separate V2_CONNECTOR_TOKEN secures connector calls from the supervisor. Keep connector Workers off workers.dev (workers_dev: false) and do not expose /v2/process on a public route. Set the same connector token on the supervisor and approved connectors without checking it into source or logging it. The supervisor does not pass provider credentials; each connector receives only its approved resource/secret bindings. D1 controls registration and leases, KV holds non-secret optional flags, R2 stores artifacts, K2 carries telemetry. A disabled connector blocks new runs; already queued work for a registered connector can still drain, so use Queue pause/recovery procedures to stop processing.

Secrets Store is central storage, not automatic authorization

Create provider credentials once in account-level Secrets Store. A binding explicitly attaches a named secret to a Worker at deploy time. await env.SPIDER_API_KEY.get() retrieves its value; workers scope means the Workers service may consume it, not that only a particular Worker can. The binding list is the per-connector allowlist. JudyRecords and Truth Social request Spider/Proxy only; SAM.gov requests its API key only. The supervisor must never hold the provider keys or a binding to the entire store.

Use the account’s Secrets Store ID (a47356e8f71f432e8fc18ee717af2d64) in the connector config’s secrets_store_secrets entries. Each provider secret needs workers scope; the CI/deployment credential attaching it needs Account Secrets Store Edit, not merely Read. Separate secret administration from deployment when possible, and do not let an unreviewed connector manifest choose arbitrary secret names. The single-store/open-beta limits (currently 100 production secrets) should be checked against your account before scaling onboarding. Rotating a shared secret changes it for all bound connectors. Audit logs cover access and create/update/delete actions, but do not substitute for per-Worker binding review. Local Secrets Store entries are different from remote values and may be unencrypted on disk: use dummy local credentials and keep .wrangler/state out of Git.

Compatibility gate: the documented Workers integration supports Secrets Store bindings, and Workers for Platforms documents user-Worker resource bindings, but Secrets Store on a dispatch-namespace user Worker has not been verified in this account. Before switching collection traffic, deploy one inert connector in the v2 namespace with a harmless test secret and confirm its get() works without exposing the value. If unsupported, stop: do not proxy provider secrets through the supervisor as an unreviewed workaround. Do not call the integration live until that test passes.

Operator rollout (not executed)

1. Confirm Workers for Platforms is enabled for this account. Workers Paid and Workers for Platforms are different subscriptions; the latter has separate pricing. Verify the desired dispatch namespace and billing before creating resources.
2. Provision new v2 D1, KV, R2, K2 stream, Workflow, harvy-v2-work Queue and its DLQ. Replace placeholder IDs/names in platform-v2/supervisor/wrangler.jsonc. Create the namespace harvy-v2-connectors; do not reuse any v1 binding. Apply platform-v2/migrations/0001_platform_v2.sql, 0002_v2_projection_releases.sql and 0003_v2_connectors.sql to the v2 D1 only after approval. No migration is automatic at Worker startup.
3. Verify Secrets Store metadata, scopes, quota and binding permissions; add SPIDER_API_KEY, WEBSHARE_PROXY_URL, or source credentials only if approved and needed. Never echo values, pass them on the command line, put them in Wrangler vars, or write them to notebooks. Configure V2_SUPERVISOR_TOKEN and V2_CONNECTOR_TOKEN as Worker secrets. Replace per-connector Secrets Store IDs with the actual store ID.
4. Check locally with npm run typecheck:platform-v2 and npm run test:platform-v2. When the account-side prerequisites are approved and verified, deploy each connector to the dispatch namespace (not as an ordinary public Worker), for example npx wrangler deploy --dispatch-namespace harvy-v2-connectors --config platform-v2/judyrecords/wrangler.jsonc. Deploy the supervisor after its namespace exists and v2 bindings resolve. Check the exact command/options with the account’s installed Wrangler version before running it.
5. PUT /v2/connectors/<id> with { "enabled": false, "capability_verified": false } using Authorization: Bearer <V2_SUPERVISOR_TOKEN> to register safely. Then check the deployed Worker and the specific source’s API/collection entitlement, output/provenance, secrets scope, and expected load. Once verified, repeat with both booleans true. Enabling performs an internal /health identity check. Truth Social remains blocked until its Spider Unlimited social-domain support is proven; SAM.gov remains blocked until API-key and sample-query capability is proven. Imported catalog entries are not deployments or approval.
6. Call POST /v2/runs with a connector ID, explicit environment_id, unique idempotency_key, and the connector’s validated input; inspect GET /v2/runs/<id>, D1 work-item state, queue/DLQ, R2 artifact path and K2 progress. Do not report a live run as successful on the basis of a health check or a local typecheck. Measure cache hit rate only on eligible public reads; never cache private run/connector responses to meet a target.

For another connector, copy platform-v2/templates/<kind> to platform-v2/<id>, use a lowercase ID matching ^[a-z][a-z0-9-]{0,62}$, set its script name to harvy-<id> and set all three code-side IDs to <id>. Fix the template $schema relative path after copying; attach only v2 R2 plus approved provider secrets and its internal token. Implement the template TODOs, add source-specific tests and typecheck the copied connector. Deploy it into the namespace and register/enable it as above. No supervisor source, queue binding or ConnectorId union edit is needed. A copied template is deliberately not ready for a real source until its example host, validation, normalization and artifact paths have been replaced.

Recovery and known limits

• Failed registration /health means the script is absent, has the wrong ID, or is unhealthy. Check its namespace, script name and Worker logs before enabling; never silently fall back to a public URL.
• A 409 on run creation means disabled/unverified registration or a KV flag explicitly disabling it. Input validation errors return 400; old idempotency keys with different input return 409. Secret binding authorization failures require checking both the deployer’s Secrets Store Edit permission and the secret’s workers scope.
• To pause new runs, PUT registration with enabled: false. To stop existing work, pause/drain the v2 Queue separately, inspect failed D1 items, leases and DLQ, then resume or replay with the same idempotency behavior. Do not delete artifacts or reset D1 to roll back a Worker. Preserve old connector versions until queued items can drain.
• The shared Queue starts with batch size two and a 15-minute D1 item lease; unlike separate per-connector queues, a noisy or rate-limited connector can delay others. Monitor per-connector lag, failures, source limits and DLQ; split queues or add a rate-limited routing tier only when observed load justifies it. Long-running items need a measured timeout/lease strategy before live use. Worker/Workflow retries can repeat provider requests or R2 writes, so choose deterministic artifact paths and idempotent output handling.
• No account-side dispatch/Secrets Store compatibility, Queue delivery, v2 migration or live collection was exercised by this implementation. Artifacts build/registration automation is not implemented; registration is an authenticated operator API, not initialization side effects.

Sources: dynamic dispatch, user-Worker bindings, Secrets Store Workers binding, access control, audit logs, Artifacts build/deploy.