Platform v2 first implementation slice
The isolated v2/ code adds a supervisor API, explicit D1 run/item schema, JudyRecords discovery through Spider Unlimited links/bounded crawl, and Truth Social public-profile/timeline processing based on the existing v1/scrape/truthsocial.py behavior. Separate provider queues feed connector Workers, which write JSON artifacts to v2 R2; durable Workflow orchestration is defined by the v2 supervisor. Truth Social remains capability-gated and disabled by default because Spider Unlimited social-domain support has not been verified for this account. No logged-in cookies or follower lists are used.
The Wrangler configs intentionally contain placeholder D1/KV resource IDs. Provision separate v2 D1, KV, R2, connector queues, and DLQs before deployment; bind matching V2_CONNECTOR_TOKEN secrets to supervisor and each connector, V2_SUPERVISOR_TOKEN to supervisor, and Spider/Webshare secrets only to connectors that need them. harvy-workflows source is absent from this repository; this slice keeps v1 untouched and defines the v2 Workflow in the supervisor Worker instead. Judy query/session search is not implemented yet: Judy discovery currently starts from explicit, allowlisted seed URLs. The Python v1/scrape/truthsocial.py remains unchanged for the existing adapter; v2 implements an isolated Worker port with tests and keeps it disabled until Spider Unlimited social-domain support is verified.
The next v2 slice adds lifecycle K2 events, K2 projectors to the universal Basin stream and Analytics Engine, shared-only public progress (Analytics Engine SQL with exact D1 fallback), cached public progress and dataset manifests, a Workers Builds event consumer, and immutable R2 release manifests. It uses the existing K2 harvy_job_events stream and Basin harvyql_stream (the v2 bindings remain separate). Before deployment, create Basin/Analytics Engine/K2 consumer bindings as configured, three K2 subscriptions (basin, analytics, progress) and store their IDs under v2:k2:subscriptions:<name> in KV, configure the build-event subscription queue, and set V2_K2_TOKEN plus V2_ANALYTICS_READ_TOKEN as secrets. The manifest schema is ready, but committing manifests into a Git-backed Artifacts repo needs a git-capable publisher (the later Container phase); the current release API writes manifests to the requested R2 bulk-dataset prefix.
Source catalog and procurement connector
Implemented connector Workers live in their own directories under v2/catalog/connectors/: judyrecords/, truthsocial/, and sam-gov/. Reusable site-search form handling lives at v2/catalog/connectors/adapters/search/; it is an adapter, not a connector. v2/catalog/connector-scopes.json describes candidates, not executable directories. The Telegram collector remains in v1/scrape/telegram.py until a v2 connector is built and tested.
v2/catalog/connector-scopes.json is a disabled-by-default inventory generated with python3 v2/scripts/import_platform_v2_source_catalog.py '<local checkout>/Ingestion/Stream Catalog' from the referenced Civontology Stream Catalog checkout plus the user-provided 53 directory/CMS/social scopes. The import contains 99 STR_ notes and 11 REF_ notes; it preserves catalog disposition separately from implementation status and copies no upstream code. The upstream crosswalk claims 106 imported definitions at AA_Stream Map.md line 3, while its disposition headings total 117 (20 + 22 + 64 + 11, lines 13, 28, 56, 125); the Stream Index lists 95 Proposed implementations and no Working streams (lines 19–28). Reconcile these counts before treating the catalog as complete. The entries are catalog candidates, not live connectors.
The executable SAM.gov v2 API connector lives in v2/catalog/connectors/sam-gov/. It requires SAM_GOV_API_KEY on harvy-sam-gov, mandatory posted-date bounds (at most one year), at most 100 API records per page and 25 pages per run, plus explicit capability verification. API keys are added only to outbound requests and redacted from captured response JSON. The referenced OpenTIR SAM scraper is AGPL-3.0 and was inspected only as a reference; none of its code or its browser/proxy/storage stack was copied. The Civic Scraper package was not added. Data-portal directories are catalog discovery sources, not permission to crawl or ingest every linked dataset.