Domain Intelligence Platform architecture overview

What it is

A continuously-running pipeline that observes every new domain registered worldwide (via Certificate Transparency), captures a screenshot, performs WHOIS + DNS enrichment, archives everything to cloud object storage, and exposes the resulting dataset through fifteen specialised search/analytics frontends.

Core capabilities

• Real-time discovery — Certstream firehose → bucketed Redis lanes → fleet workers
• Screenshot capture — headless Chrome (rod-pool) at mobile + desktop viewports, JPEG+thumbnail to Backblaze B2
• WHOIS — per-TLD parsers covering ~25 RDAP/legacy registries, enriched with admin/tech/contact
• DNS enrichment — bulk zdns scans (47 columns: A/AAAA/MX/NS/TXT/SPF/DKIM/CAA/...)
• .com availability — in-RAM zone-diff against the daily CZDS .com zone (162M SLDs)
• Drop-catch & expiry tracking — automated per-day pipelines for .in, .com, .io expiries

Scale

PostgreSQL 18 with the canonical table v3.domains (BIGSERIAL row_id, ~539M rows, fillfactor=50, pg_repacked). Snapshot blobs live in Backblaze B2 behind cdn.backup.im.

Architecture (high level)

                       ┌────────────────────┐
                       │ Certstream firehose│  (CT logs, .com/.in/.io/...)
                       └──────────┬─────────┘
                                  │
                  ┌───────────────┴───────────────┐
                  │ HUB · server25.js · :3000     │   23.137.104.135
                  │ • REST API (200+ endpoints)   │   ├─ Redis 6379  (queues, KV)
                  │ • UI: 30+ HTML dashboards     │   ├─ Bull queues  (legacy)
                  │ • baseline cursor coordinator │   └─ B2 uploader  (cdn.backup.im)
                  └───────────────┬───────────────┘
                                  │       Tailscale mesh (100.x.x.x)
            ┌─────────────────────┼─────────────────────┐
            │                     │                     │
            ▼                     ▼                     ▼
   ┌────────────────┐   ┌────────────────┐    ┌────────────────┐
   │ Fleet workers  │   │ NATS broker    │    │ PostgreSQL v3  │
   │ (Go binary,    │──→│ JetStream      │←───│ database.dom.to│
   │  28+ VPSes)    │   │ RESULTS_PIPELINE    │ partitioned    │
   │ ss/whois/dns   │   │ + KV streams   │    │ FDW shims      │
   │ + "all" lane   │   └────────┬───────┘    └────────────────┘
   └────────────────┘            │                     ▲
                                 ▼                     │
                          ┌──────────────┐             │
                          │ pipeline-svc │─────────────┘
                          │ (Go writer)  │  apply: WHOIS / DNS / SS
                          └──────────────┘
  

All inter-host traffic flows over a Tailscale mesh, so VPSes from any provider can join the fleet in minutes. The hub is the source of truth for configuration; everything else is replaceable.

Data pipeline (one domain end-to-end)

  1. CT log entry         certstream → ct_log_monitor_pg.js
  2. Bucket + dedupe      Redis: ss:active / ss:passive / whois:passive / dns:passive
  3. Worker pull          fleet-worker (Redis BLPOP) or baseline cursor claim
  4. Render               rod-pool Chrome → JPEG @ 375×667 + 1440×900
  5. Enrich               whois (TCP 43 / RDAP) → ParsedWhois
                          zdns (A/AAAA/MX/NS/TXT/SPF/DKIM/CAA/...)
  6. Emit                 NATS publish → RESULTS_PIPELINE stream
  7. Apply                pipeline-svc → bulk UPDATE v3.domains + insert v3.screenshots
  8. Archive blob         B2 upload (a-z sharded) + thumbnail
  9. CDN serve            cdn.backup.im
  

Workloads currently running

Work distribution: cursor-as-counter

Workers don't lock or "claim" anything. The cursor is just a monotonic counter.

-- Shape of the row in v3.baseline_cursor (one per workload)
workload    text        -- ss / whois / dns / all
next_row_id bigint      -- the counter
direction   text        -- forward (0→max) or backward (max→0)
range_size  int         -- how many rows per claim

-- claim_next_range() in plpgsql:
SELECT * FROM v3.baseline_cursor WHERE workload=$1 FOR UPDATE;
new_start := next_row_id;
new_end   := new_start + range_size;
UPDATE v3.baseline_cursor SET next_row_id = new_end WHERE ...;
RETURN (new_start, new_end);

Properties

• Workers serialise on the cursor row for microseconds only — each gets a unique range.
• If a worker dies mid-range, the range is lost. Re-coverage happens naturally on the next cycle wrap-around (cursor walks back to 0 / up to MAX).
• Worker durability is local: BadgerDB on each worker stores the in-flight range so a worker restart resumes where it left off.
• No heartbeat, no claim release, no stale-claim detection. Removed in migration 0009 after the original claim/release model wedged the fleet for 8 hours when 4 workers died holding claims.

Fleet

Single Go binary (fleet-worker) running on every VPS as a systemd unit. Auto-updates from a release manifest with stable / beta / canary lanes.

Provider mix (representative)

DeluxHost · CycloneServers · Hetzner · DigitalOcean · Berohost · Solidvps · Advin · Packets · 24fire · Greencloud · Sthosting · ProprUS · DediRock · UltraVPS — usually €1–35/year per host. All on Tailscale; provider-agnostic.

Databases & storage

Public sites (15× dom.to subdomains)

All hosted on a single 512MB IPv6-only VPS (77.76.13.194), nginx-fronted, PM2-managed, ports 3001-3015.

Tech stack

Evolution highlights

• v0.14 — single fleet-worker binary replaces fleet of separate Python/Node scripts
• v0.17 — hub-controlled, autonomous defaults (worker.yml becomes optional)
• v0.18-0.19 — pipeline-svc + per-host class-tiered batch sizing + sweep observability
• v0.20-0.21 — schema-native payload format; release lanes (stable/beta/canary)
• v0.23 — NATS+BadgerDB scaffolding; HTTP-only fallback path
• v0.24 — owned-claim DNS lane: worker durably owns its range, no DB-side claim/release
• v0.25 — "all" workload: whois+dns then OR-gated SS in one pass; backwards walk for newest-first coverage
• migration 0009 — drops claimed_by gating from claim_next_range; cursor becomes a pure counter

What's next

• Move fleet_activity_log off v3 (telemetry was throttling DNS UPDATEs)
• Replace home-grown pipeline_health_monitor with Zabbix/Netdata
• Cut over screenshot writers/readers from FDW to v3.screenshots_native
• Re-enable certstream subdomain collection at full fleet capacity