Revision: Rewriting the Ingest Collector
Diff notes on polling versus event-driven ingest, with the operational tradeoffs made explicit.
Run the collector
Same records, same worker, different delivery contract. Advance the clock to see where each design spends its time. The window stops at the configured limit so you can inspect the result instead of waiting for an imaginary system to become done.
Observation window ends at 5.00s.
records
- No records yet.
event stream
same burst, two paths
Polling vs event-driven ingest
- — simple mental model
- — interval becomes a bottleneck
- — more idle work
- — reacts to actual events
- — less unnecessary polling
- — more moving parts to reason about
The change in one screen
// polling
setInterval(async () => {
await collectNewRecords();
}, intervalMs);
// event-driven
events.on("record.created", async (record) => {
await ingest(record);
});The second version removes idle polling. It also makes delivery, retries, and ordering someone else's problem only if you explicitly design for them.
The polling version isn't automatically wrong. It becomes the wrong shape when the interval adds work, hides delay, or makes recovery harder to reason about.
The exhibit uses the same records for both modes. Polling places a record in a source collection and discovers it on the next check. Event-driven ingest emits a record.created event and delivers it after the configured delay. Both paths hand records to the same one-at-a-time worker, so the comparison is about delivery rather than throughput theater.
The failure is deterministic: every third record fails once, then retries after a short delay. That makes attempts, retry count, queue depth, and completion order visible without turning the demo into a slot machine. Change the maximum attempts if you want to see the failure boundary instead of the recovery path.
The observation window defaults to five seconds and stops automatically. It does not stop as soon as the queue empties, because an empty polling check is part of the story. The final idle-poll count shows the work the collector performed while waiting for something to happen. Event-driven ingest avoids those checks, but adds delivery and retry contracts that still need an owner.
Detection latency measures the time between record creation and the collector discovering or receiving it. It is not end-to-end processing latency. The completion order shows what the worker finished, not a promise that real distributed systems preserve order.
This is a design comparison, not a claim that I built or operated a fleet telemetry system. The useful question is where delivery, retries, ordering, and failure ownership should live before the first production incident answers it for you.