`_id` generation for large chuncks of data generates duplicates ids

[ErikPillon]

The current _id generation strategy is not safe across restarts because it relies solely on the current millisecond and an in-memory counter that resets when the process exits.

A robust fix would be to stop overriding MongoDB's _id entirely, or derive _id deterministically from _key if the project truly needs a custom primary key.

[br-g]

Hello @ErikPillon and @FestinBiju,
Thanks for raising this.

Some context on why _id is currently overridden: we use it as a chronological sort key. When several documents share the same _key, the _id timestamp is what decides, at query time, which one is the most recent. Reusing _id for ordering was a deliberate choice to avoid storing a separate timestamp field and save a bit of disk.

That ordering guarantee also leaks into the public API; _id is sent over MQTT/WebSocket and consumers rely on it to sort messages chronologically. So whatever we change, we need to preserve a key that increases with event time at (at least) millisecond resolution.

My one concern is relying on MongoDB's _id for ordering. An ObjectId's embedded timestamp only has second resolution, and within the same second the order comes from a random value plus a counter rather than from time. For high-frequency data like ours that means many documents per second would no longer be ordered by arrival time.

To make sure we close the exact gap you hit: could you share the conditions that produced the collision on your end? Re-ingestion of the same chunk, a restart mid-ingestion, or concurrent writers? That'll confirm which fix actually solves it

[br-g]

Some additional context:

OpenF1 stores this data in MongoDB time series collections, and the reason we don't upsert is that time series collections don't allow it; MongoDB explicitly forbids "upsert: true" on time series update commands.
We rely on time series collections because they cut storage by roughly 90% overall. That saving is largest for our highest-volume streams, location and car_data, which dominate disk usage, so moving off time series collections to regular ones (which would allow upserts) isn't really on the table.