The OSINT Workflow That Survives Contact With Reality

The OSINT problem in 2026 is not access. Telegram channels, ship transponders, corporate registries, leaked databases, satellite tasking, and a million scraping tutorials have made collection nearly free. The problem is that free collection produces an ocean, and most teams respond by building a bigger bucket. The workflow that survives contact with reality is the one that treats noise as the default state of the world and engineers four disciplines against it: requirements-driven collection, machine-enforceable source grading, governed entity resolution, and an analyst review loop with real feedback capture.

01Collection Planning: Requirements Before Feeds

The first failure mode is collection greed. A team stands up scrapers for forty sources because they might matter someday, and six months later nobody can say which feed answered which question. The fix is old tradecraft wearing new infrastructure: priority intelligence requirements (PIRs) come first, and every collection task must cite one.

A collection task is a typed object, not a cron entry. It carries the requirement it serves, the owner who will be asked "is this still worth it," the legal and terms-of-service basis for touching the source, the cadence, and — critically — an expiry date. Collection that nobody renews stops. This one field kills the zombie-scraper problem that quietly eats most OSINT budgets.

from pydantic import BaseModel, Field
from datetime import date

class CollectionTask(BaseModel):
    task_id: str
    pir_id: str                      # the requirement this serves — mandatory
    source_id: str
    method: str                      # api | scrape | manual | purchase
    cadence: str                     # continuous | hourly | daily | on-demand
    legal_basis: str                 # tos-permitted | public-record | licensed
    owner: str                       # a human who answers for this feed
    expires: date                    # zombie-scraper killer
    expected_yield: str              # what "working" looks like, in one line

def active_tasks(tasks: list[CollectionTask], today: date) -> list[CollectionTask]:
    return [t for t in tasks if t.expires >= today]

02Source Grading Machines Can Enforce

The Admiralty (NATO) system — source reliability A–F, information credibility 1–6 — has survived for decades because it separates two questions people constantly conflate: do I trust the source and do I believe this specific claim. A reliable source can relay a false claim; a garbage source can hand you a true one. The modern failure is that teams treat grading as a culture value instead of a schema constraint. If reliability and credibility are not required fields at ingest, they will not exist at analysis time.

Two engineering rules make grading real. First, grades propagate: any derived record — a correlation, a summary, an alert — inherits the weakest grade in its lineage, automatically. Second, grades decay: a claim credible in January is not automatically credible in July, so temporal validity is part of the envelope, and stale claims fall out of automated pipelines until re-observed.

RELIABILITY_ORDER = "ABCDEF"          # A best
CREDIBILITY_ORDER = "123456"          # 1 best

def worst_grade(inputs: list[tuple[str, str]]) -> tuple[str, str]:
    """Derived records inherit the weakest grade in their lineage."""
    rel = max(inputs, key=lambda g: RELIABILITY_ORDER.index(g[0]))[0]
    cred = max(inputs, key=lambda g: CREDIBILITY_ORDER.index(g[1]))[1]
    return rel, cred

def eligible_for_auto_correlation(rel: str, cred: str) -> bool:
    return rel in "AB" and cred in "12"

03Entity Resolution Without Hallucinated Merges

Entity resolution is where OSINT systems quietly commit their worst sins. Two records — "D. Okonkwo, Lagos, logistics" and "David Okonkwo, freight broker" — get merged by a fuzzy-match heuristic, and from that moment every downstream product treats them as one person. If the merge was wrong, you have manufactured intelligence. Nobody will catch it, because merges are invisible once made.

The survivable design treats a merge as a governed action with a risk band, exactly like the action-routing model in our Governed Autonomy playbook. Cheap blocking (shared identifiers, geo-temporal overlap, name phonetics) generates candidates. A scorer produces match confidence with named evidence. Then routing applies: high-confidence, low-impact merges auto-apply with audit; anything touching a sanctioned entity, a person of interest, or a published product requires an analyst to approve the merge with the evidence in front of them. And every merge is reversible — you store the pre-merge state, because some of them will be wrong.

class MergeProposal(BaseModel):
    left_id: str
    right_id: str
    score: float                      # 0..1 from the matcher
    evidence: list[str]               # named signals, not a bare number
    impact: str                       # low | medium | high

def route_merge(p: MergeProposal) -> str:
    if p.impact == "high":
        return "analyst_approval"     # POI, sanctions, published products
    if p.score >= 0.92 and p.impact == "low":
        return "auto_apply_with_audit"
    if p.score >= 0.75:
        return "analyst_queue"
    return "reject_keep_candidates"   # store the pair; never silently merge

04Provenance: The Claim Ledger

The unit of OSINT truth is not the document — it is the claim: a single assertion, extracted from a source, with a grade, a timestamp, and a validity window. Documents are containers; claims are what you reason over. Keeping claims as first-class records gives you three capabilities that flat pipelines never get: contradictions become visible objects ("source A asserts X, source B asserts not-X") instead of silent overwrites; deception becomes detectable as patterns of coordinated claims from correlated sources; and every published sentence can be walked back to raw collection in one query.

CREATE TABLE claim (
  claim_id      UUID PRIMARY KEY,
  entity_id     UUID,                          -- resolved subject (nullable pre-resolution)
  predicate     TEXT NOT NULL,                 -- located_at | affiliated_with | owns | ...
  value         JSONB NOT NULL,
  source_id     UUID NOT NULL REFERENCES source(source_id),
  document_id   UUID NOT NULL,                 -- container it was extracted from
  reliability   CHAR(1) NOT NULL,              -- A..F
  credibility   CHAR(1) NOT NULL,              -- 1..6
  observed_at   TIMESTAMPTZ NOT NULL,
  valid_time    TSTZRANGE NOT NULL,            -- claims decay
  extracted_by  TEXT NOT NULL,                 -- analyst id or model@version
  supersedes    UUID REFERENCES claim(claim_id),
  contradicts   UUID[] DEFAULT '{}'            -- contradictions are data
);

Note the extracted_by column. When a model does the extraction, the model identity and version are part of provenance. When a model's extraction later proves wrong, you can find every claim it touched. That is what provenance means operationally: errors have a blast radius you can query.

05The Analyst Review Loop

Automation upstream exists to protect the scarcest resource in the pipeline: analyst attention. The review loop that survives has three properties.

1. Triage by decision-value, not arrival time. The queue is ranked by which PIR a claim serves, its grade, and its novelty against the existing claim graph — not by recency. A C3 claim that contradicts a standing assessment outranks fifty A1 claims that confirm what you already know.
2. Confidence bands, not binary verdicts. Analysts confirm, reject, or mark "credible but uncorroborated." The middle state matters: it keeps honest uncertainty in the system instead of forcing false precision.
3. Every override is captured as labeled evidence. When an analyst rejects an auto-correlation or downgrades a source, the reason is a structured field, not a Slack message. Those records become the eval datasets that tune matchers, grading priors, and triage ranking — the same correction-to-evidence loop that powers the Artemis self-improvement architecture.

Products published from the desk carry their confidence with them: key judgments state the grade of supporting claims, dissenting claims are listed rather than deleted, and each judgment links to its claim lineage. A reader who wants to challenge the assessment can attack the evidence, which is exactly what you want — that is what makes an intelligence product citable.

06Scenario: One Claim, End to End

A regional Telegram channel posts that a sanctioned freight company has resumed operations at a specific port. The collection task that captured it cites PIR-7 (sanctions evasion, maritime), so it enters the graded pipeline instead of a folder. The source is graded D (unproven channel); the claim gets credibility 4 — plausible, uncorroborated. As a D4 claim it is firewalled from publication and auto-merge, but it ranks high in triage because it contradicts a standing assessment that the company is dormant.

An analyst tasks two corroboration checks: port-call data and ship-transponder tracks. AIS shows a vessel historically linked to the company at the port in the claimed window — a B2 claim from a licensed feed. The correlation engine proposes linking the vessel record to the company entity; because the entity is sanctioned, the merge routes to analyst approval with evidence attached. The analyst approves the link, upgrades the composite assessment to "probable," and publishes a product whose key judgment cites both claims with grades and lineage. The Telegram source's reliability prior ticks upward — earned, recorded, and reversible. Total analyst touches: two. Everything else was structure.

Related briefs