17,000 events a second, live in one week
Replacing a legacy stack that had stopped scaling with a real-time pipeline ingesting 3 TB a day — delivered in a single week, and paid for by decommissioning what it replaced.
US retailer, 3 TB/day telemetry
17K/secevents ingested, 3 TB/day
In-store Wi-Fi access points see something no till system does: how people move. Which entrance they use, where they pause, how long they stay, which departments they pass without stopping. It is one of the few sources of physical-world behavioural data a retailer already owns.
The problem is volume. Presence events from access points across a store estate arrive constantly, in quantities that make "just put it in the warehouse" an expensive answer.
Challenges
A traditional stack, at the wrong scale. The existing technology could not manage the rapidly growing volume of high-speed data. This is the classic inflection point where a system that worked for years stops working over a few months, because the data grew and the architecture did not.
Capture from network hardware. The data originates in networking equipment, not an application database. There is no convenient nightly export.
Real-time ingestion and ETL. Yesterday's footfall is a report. Today's is an operational decision — where to move staff, what to bring forward.
The architecture
We built and deployed a next-generation ETL pipeline, plus a connector offloading processed data into the analytics warehouse.
Results
| Measure | Outcome |
|---|---|
| Ingest throughput | 17,000 events per second |
| Daily volume | 3 TB per day |
| Time to build and deploy | One week |
| Legacy stack | Decommissioned — thousands of dollars saved |
The one-week figure is the one people disbelieve, so it is worth being precise about what it means. It was not a week of inventing a streaming platform from nothing. It was a week because the ingestion and ETL frameworks already existed — built, hardened and reused across engagements — so this project was configuration and connector work rather than construction.
That is the compounding return on framework investment described in our retail platform work: the third project on a shared foundation is dramatically cheaper than the first.
What the business did with it
Two operational uses, both unglamorous and both immediately valuable:
- Merchandise planning at store level — what to stock where, informed by how people actually move through a specific store rather than by chain-wide averages.
- Workforce management — staffing to observed footfall patterns instead of to a schedule set months earlier.
What we would take from this
The savings arrive from decommissioning, not from building. The new pipeline cost money; retiring the legacy stack it replaced is where the finance case landed. Migrations that run both systems indefinitely never realise their business case.
Throughput numbers are meaningless without the shape of the data. 17,000 events per second of small presence records is a very different problem from 17,000 large documents. Quote both, or the number tells the reader nothing.
Frameworks are what make a week possible. Without them this is a multi-month project, and no amount of pressure changes that.
We do not name clients. Engagements are described by sector and scale because confidentiality obligations outlast the work, and consent we cannot produce is consent we do not have.