Six times faster model training, across 30 million customers

Propensity models answer a commercially simple question: how likely is this customer to buy this thing? Get it right and every downstream decision — which offer, which channel, which moment — improves. Get it slow, and the answer arrives after the decision has been made.

A US retailer's existing pipeline had reached the point where scale, not accuracy, was the binding constraint. It could not process the volumes the business had grown into, so it did what such systems do: it sampled. The models were trained and scored on a fraction of the customer base, and everything downstream inherited that limitation.

What was actually wrong

Three problems, and only one of them was about model quality.

Scale. The legacy implementation could not handle processing at the volumes involved — hundreds of millions of records.

Sampling as a workaround. Because full processing was infeasible, sampling logic stood in for it. That is a reasonable engineering compromise and a quietly expensive one: the customers omitted from a sample are disproportionately the unusual ones, and unusual customers are exactly where propensity modelling earns its keep.

Iteration speed. Model training took over 18 hours. When a full training run costs the better part of a day, experimentation stops. Nobody tries a speculative feature at four in the afternoon.

What we built

A 360-degree customer view. The pipeline drew on clickstream activity, in-store behaviour and demographic data, combining them into a single customer-level representation rather than several partial ones.

Category-level, dynamic feature engineering. Rather than one global feature set, features were derived per category — product category and subcategory, age and gender buckets, with brand and price endpoints for every customer. A customer's propensity for homeware is not predicted well by the same signals that predict their propensity for apparel, and a single flat feature set forces a compromise between them.

Explicit handling for unknown data. Real customer data is full of gaps. Treating "unknown" as a value in its own right, rather than discarding the record or silently imputing, is what let the pipeline run against the whole dataset instead of the well-behaved part of it.

Validation and evaluation as pipeline stages. Model validation was built into the pipeline with a metrics dashboard, rather than being a manual step after training. This is the change that most improved the team's working life: comparing versions stopped being an exercise in remembering what was run.

The numbers that mattered

The results recorded at the time:

• The complete dataset, four months of it, for 30+ million customers — replacing the sampling logic entirely.
• Training time from over 18 hours to three, for 19,000 models.
• Scoring improved by 14%, with the unknown-data handling running against 100% of the dataset.
• Model tuning and optimisation across versions "drastically reduced" by the built-in validation and evaluation stages.

The training time is the one we would highlight. A six-fold reduction is not merely convenient — it changes what the team can attempt. At 18 hours you get one experiment per day and you choose it conservatively. At three you get several, and you can afford to be wrong.

That 19,000 models are involved is worth pausing on too. This is not one model; it is a model per meaningful segment, which is what "category-level" means in practice and why the training cost mattered so much.

Where the value showed up

Better propensity scores fed personalised search and recommendations, discount optimisation and campaign management. The internal estimate attributed approximately $20 million of additional revenue to those improvements combined.

We quote that figure with its framing intact: it is an estimate, recorded in our own project documentation, attributing revenue across several initiatives that the propensity work contributed to rather than solely caused. Attribution at that level is genuinely hard, and we would rather present the number honestly than either drop it or imply a precision it does not have.

The engineering results above — dataset coverage, training time, model count, scoring improvement — are directly measured and are the ones we would stand behind without qualification.

The general lesson

The instinct when a model underperforms is to reach for a better algorithm. Here the algorithm was rarely the constraint. Sampling was, and sampling existed because the pipeline could not do better.

Fixing the engineering — full-dataset processing, faster training, explicit handling of missing values — improved the model more than a change of technique would have, and it improved the team, because experiments that take three hours get run and experiments that take eighteen do not.