Deduplication across platforms: the complete technical reference

Deduplication starts as a distributed identity problem. A single conversion can appear from a browser pixel, a backend webhook, a checkout system, a queue retry, and an analytics export. Every one of those records can be truthful. None of them is useful if the system cannot decide whether they are copies of one conversion or genuinely separate economic events. The more systems involved, the less safe it becomes to rely on implicit clues such as arrival time, event name, or customer email alone.

Eventual consistency makes this worse. Browser events are usually fast but fragile. Server events are durable but often delayed by queueing, anti-fraud checks, payment confirmation, or warehouse staging. A platform may receive the browser copy first, the server copy first, or only one of them for a while. Dedup therefore cannot depend on delivery order. It has to depend on a stable contract that remains true even when the same conversion arrives through multiple transports minutes or hours apart.

Retry behavior turns dedup into an idempotency problem. A retry is not supposed to create a second commercial fact. It is only supposed to complete the delivery of the first one. The moment a retry path generates a new event_id, the transport layer stops being idempotent and the platform receives what looks like a brand new conversion. Good dedup architecture therefore lives at the seam between distributed systems design and measurement logic.

The exact vocabulary differs by network, but the structure is similar across all major ad platforms: one shared conversion identity plus supporting browser, click, or hashed match context. The table below is a practical implementation map rather than a claim that every destination exposes the same public diagnostics.

A correct event_id has three requirements. First, it must be unique per conversion event. Not per user, not per cart, and not per session. Second, it must persist across retries and recovery flows so that a failed delivery can be replayed without producing a second identity. Third, it must be deterministic from the point of view of the system that owns the conversion. Even if the ID is random, the platform-facing value must be fixed once assigned.

There are two sane generation strategies. UUID v4 is the cleanest when the event producer has authoritative storage and can persist the result at creation time. Deterministic hashing is better when multiple systems may need to derive the same identity without a central write step, for example when purchase events can originate from both checkout completion and payment confirmation workers. In that case, a canonical input such as order_id, merchant_id, and a timestamp bucket or event-type suffix can generate the same ID every time.

Timestamp alone is never enough. It is not unique under load, it is vulnerable to clock skew, and it makes independent retries or partial replays look like distinct conversions if the timestamp is regenerated. The same order confirmed twice within one second can collide, while one delayed retry a minute later can fork the identity of the same purchase. Timestamps are useful as part of a canonical payload. They are dangerous as the whole key.

Window behavior is where teams often get surprised. Platforms do not only care about whether the identifiers match. They also care about when the second copy arrives relative to the first and whether the surrounding browser or click context is still valid. The table below should be read as an operational reference for implementation and debugging, not a substitute for platform docs on specific campaign types.

The safe mental model is that retries operate on a delivery record while dedup operates on a conversion record. TrackLayer keeps those concerns separate. When a network request to Meta, TikTok, or another destination fails, the retry worker does not rebuild the event from scratch with a fresh identity. It reuses the existing conversion envelope, the same event_id, and the same platform-scoped payload inputs.

This matters even when the failure happens after the remote platform received the request but before TrackLayer observed the response. In distributed systems, that is the classic ambiguous success case. The platform might have accepted the event, while the sender believes it failed. The only defense is idempotent replay. If the retry carries the same event_id, the platform has a chance to recognize it as the same conversion. If the retry uses a new ID, the ambiguity becomes a guaranteed duplicate.

Deterministic IDs are useful when multiple services need to agree on the same conversion identity without asking a central event registry in real time. The key is to hash a canonical form, not a loosely concatenated string assembled ad hoc by each service.

import { createHash } from "node:crypto";

type PurchaseIdentityInput = {
  merchantId: string;
  orderId: string;
  eventName: "Purchase" | "Refund";
  timestampBucket: string; // for example: 2026-04-23T14:35Z
};

function canonicalize(input: PurchaseIdentityInput) {
  return [
    input.merchantId.trim().toLowerCase(),
    input.orderId.trim().toLowerCase(),
    input.eventName,
    input.timestampBucket,
  ].join("|");
}

export function buildDeterministicEventId(input: PurchaseIdentityInput) {
  const canonical = canonicalize(input);
  return createHash("sha256").update(canonical).digest("hex").slice(0, 32);
}

const eventId = buildDeterministicEventId({
  merchantId: "acme-eu",
  orderId: "ord_104928",
  eventName: "Purchase",
  timestampBucket: "2026-04-23T14:35Z",
});

// eventId → 7d5e8d4ff1c96e6caa4e5c7d1ad23942

The important detail is canonicalization. Trim whitespace, normalize case where appropriate, pin separators, and define the exact timestamp granularity. If one worker hashes raw order IDs while another lowercases them, deterministic generation stops being deterministic.

Warehouse checks are where dedup bugs become obvious. Platform dashboards often show the symptom after a delay, while your warehouse can usually show the root cause immediately.

select
  platform,
  event_name,
  event_id,
  count(*) as delivery_count
from platform_deliveries
where delivered_at >= current_date - interval '30 days'
group by 1, 2, 3
having count(*) > 1
order by delivery_count desc, platform, event_name;

select
  merchant_id,
  order_id,
  count(distinct event_id) as distinct_event_ids,
  min(created_at) as first_seen_at,
  max(created_at) as last_seen_at
from tracklayer_events
where event_name = 'Purchase'
  and created_at >= current_date - interval '60 days'
group by 1, 2
having count(distinct event_id) > 1
order by last_seen_at desc;

select
  date_trunc('week', delivered_at) as week,
  platform,
  sum(case when dedup_status in ('deduplicated', 'merged') then 1 else 0 end)::decimal
    / nullif(count(*), 0) as dedup_ratio
from platform_deliveries
where delivered_at >= current_date - interval '180 days'
group by 1, 2
order by week asc, platform;

select
  m.event_id,
  m.event_name,
  m.received_at,
  m.value,
  m.currency
from meta_receipts m
left join tracklayer_events t
  on t.event_id = m.event_id
 and t.event_name = m.event_name
where m.received_at >= current_date - interval '30 days'
  and t.event_id is null
order by m.received_at desc;

Internally, TrackLayer treats dedup as a system invariant rather than a destination option. The platform adapter is allowed to transform field names, but it is not allowed to reinterpret conversion identity.