The Record

A complete record

Illustrative. The schema, field names and value formats are exactly as produced. The content is invented — we do not publish real declarations as examples.

          {
  "schema_version": "1.6",
  "pair_id": "01KZQSC2HZRV6DC7PAJ17DGKWD",
  "release_id": "rel-2026-08-11T06-50-15-332Z",
  "declaration_id": "dec_9f2c41b807ae5d63",

  "declaration": {
    "text": "a family estate car under 40k, petrol, in Zurich",
    "scrub_level": "regex_ner",
    "inferred_region": "CH",
    "region_suppressed": false,
    "declared_at": "2026-08-07T14:00:00.000Z",
    "unlinkable_token": "zk:9a41c00e17d2b3f8"
  },

  "classification": {
    "taxonomy_code": "CT-1",
    "taxonomy_version": "3.1",
    "taxonomy_dictionary_hash": "sha256:2414682d049b92be...",
    "label": "Automotive",
    "confidence": 0.82,
    "classifier_id": "aice.v1",
    "classifier_source": "live"
  },

  "match": {
    "kind": "served_ad",
    "ranker_id": "matcher.cf-ad-agent",
    "payload": {
      "ad_format": "native",
      "headline": "Certified pre-owned estates, Zurich",
      "advertiser_id": "adv:512"
    }
  },

  "candidate_set": {
    "considered": 20,
    "served": 1,
    "not_evaluated": 0,
    "eliminated_by": { "cctld_geo": 6, "llm_floor": 9, "post_fetch_geo": 4 }
  },

  "outcome": {
    "grade": "deep_engagement",
    "grade_weight": 0.4,
    "events": [
      { "kind": "impression", "at": "2026-08-07T15:00:00.000Z" },
      { "kind": "click",      "at": "2026-08-07T15:00:00.000Z" }
    ],
    "human_verification": {
      "verified": true,
      "verifier_kind": "declarant",
      "user_response": "yes_match"
    }
  },

  "labels": { "relevance": "yes_match", "engagement": "deep_engagement" },

  "provenance": {
    "operator": "profila",
    "consent_ref": "policy:profila-terms:2026-08",
    "consent_scope": ["ad_matching"],
    "key_id": "profila-corpus-2026-08",
    "signature": "ed25519:r8iNn40SBSDSQsUcgfepv7Uhi..."
  }
}
        

The unit of the corpus is one JSON object per line (JSONL), gzipped, in numbered part files.

The two orthogonal labels

This is the most important structural decision in the schema.

          labels.relevance    yes_match | no_match | no_response      ← the person's judgement
labels.engagement   the behavioural grade                    ← what they did
        

They are deliberately separate, because they disagree — and the disagreements are the most informative records in the corpus. A click on something the declarant then rejected is not a positive example. Collapsing these two into one score, as click-optimised datasets do, destroys precisely the information you would pay for.

The outcome ladder

Nine rungs, ordered, each with a weight.

The top rung is the one no ad network can produce: an outcome confirmed both by the market and by the person who declared the intent.

The grade is a pure function of the events the record carries. It can be independently re-derived from the exported record without reference to our database, and our validator does exactly that on every record before publication.

not_served sits outside the ordered ladder so it can never be reached from an event set — because no set of events describes an absence of events. It is assigned from the fact that nothing was matched. Both it and no_engagement carry weight 0, and conflating them would teach a model that unmet demand is a failed advert rather than a missing one.

Optional blocks

• <code>refinement</code> — present only when this declaration revised an earlier one. The block’s presence carries meaning: it means this declaration corrected something. It records the prior and new taxonomy codes, the prior label, whether the text changed, and the step number. Chain members share a chain_id, so a whole trajectory can be reconstructed with a group-by rather than by walking a linked list.
• <code>candidate_set</code> — present when the counterfactual was captured. Note not_evaluated, held separately from eliminated_by: on one of our two search paths the ranker stops at the first acceptable result rather than judging the rest, and recording those as “rejected” would claim a decision that was never made.

The corpus is a projection, never a table

There is no “corpus table”. A record is assembled at export time by joining the declaration, its classification, the served content, the event stream, and the verdict.

This is deliberate and has one large consequence: there is no second copy of the data to keep in sync, and no stored artifact that can drift from the operational truth. Every release is derived fresh.

The whole pipeline runs on Cloudflare’s edge — Workers for compute, D1 for operational storage, R2 for the published corpus, Workers AI for classification and PII scrubbing. Two databases are separated by purpose: an operational store, and an EU-pinned store for personal data. The corpus is built from the operational store only.

A release is a rebuild, not an append

          releases/<release_id>/part-00000.jsonl.gz
releases/<release_id>/_manifest.json
releases/<release_id>/_datasheet.json
releases/CURRENT                          → the live release id
        

Each release is built completely from the current state of the database, and the previous release is deleted once the new one is live.

That single design choice is what makes erasure work. When a person deletes their account, their rows leave the database, and they are therefore simply absent from the next release — no suppression list, no rewriting of published files, no dependence on a licensee honouring a tombstone.

The ordering is careful: the new release is built and published before the old one is removed, so a failure mid-build leaves the previous release intact rather than leaving the corpus briefly empty.

The validation gate

Every record is validated before it can be published. Fourteen rule families.

A record failing any rule is withheld from the release and written to a dead-letter table with the rule that rejected it. The release datasheet reports the withheld count, so a short release can never be mistaken for a small queue.

No record in the current release was rejected by the gate. The withheld count ships with every release; the record counts themselves are published once the corpus passes 100 contributors.

• The gate is itself checked. It runs once, at write time, per record — so three classes of defect are invisible to it: rules that only make sense across records, drift in already-published records, and the gate silently failing open. A scheduled re-validation re-derives the entire corpus from the database and re-checks it, writing nothing. It reports failures per rule, because a rule that has stopped firing shows up as a missing number rather than as extra green. It found two real defects on its first production run. Current status: ok: true — no rule reporting an error.

Signing — verify before you trust anything

Every record is signed with Ed25519 over a canonical JSON serialisation: recursively key-sorted, whitespace-free, with the signature field excluded. The public key and its identifier are published at a stable endpoint.

          def canonical(obj):
    """Recursively key-sorted, whitespace-free, signature excluded."""
    if isinstance(obj, dict):
        items = sorted((k, v) for k, v in obj.items() if k != "signature")
        return "{" + ",".join(f"{json.dumps(k)}:{canonical(v)}" for k, v in items) + "}"
    if isinstance(obj, list):
        return "[" + ",".join(canonical(v) for v in obj) + "]"
    return json.dumps(obj, separators=(",", ":"))

bad = [r["pair_id"] for r in records if not verifies(r)]
assert not bad, f"{len(bad)} records failed signature verification"
        

A licensee can verify, independently and offline, that a record came from us and has not been altered — using only the published key and the canonicalisation rule. We tested this by writing a verifier from the published description alone and confirming it both validates authentic records and detects tampered ones.

Do this first. It is the difference between a dataset you were handed and one you have checked.