The Waitrose effect, but in your data

Houses near a Waitrose sell for more. Estate agents have a name for it, the Waitrose effect, and the numbers behind it are real: a premium of ten to twenty-odd percent, more in London. The tempting reading is that the supermarket lifts the street. The honest reading is that nobody knows which way it runs. Maybe the shop raises the prices. Maybe Waitrose just opens where the prices were already high and the money was already there. The correlation is solid. The cause is anyone's guess.

I had that in my head for most of a week, working on something that has nothing to do with supermarkets.

The client trains apprentices. Construction in some places, digital in others, and a standing question about where to put which. They had a hunch, the usual hunch, that there was demand they weren't serving and didn't know how to find.

So we went looking in the free stuff first. The Office for National Statistics publishes a great deal of local-area data, broken down by place, and gives it away. Most people never touch it because on its own it's a wall of rows that doesn't say anything. We loaded a chunk of it into ClickHouse (a database built for chewing through very large tables fast) and pointed it at the client's own records: where their courses already ran, where they filled, where they didn't.

The correlations came back in seconds. That part is cheap now. What took the time was reading them.

The interesting bit wasn't where the two datasets agreed. It was the gaps. Places where the public data said the conditions were right for a digital course, and the client wasn't running one. On a dashboard a gap like that looks like money: untapped demand, go and win it.

But a gap has a second reading, and it's the Waitrose one. Maybe nobody's there because the market already tried and walked away. Maybe the correlation that flagged it is the same trick as the supermarket: the signal is real, the cause is backwards, and the "opportunity" is a place that was never going to work.

You can't tell the two apart from the data. That's the whole point of the Waitrose effect. A correlation will not tell you which way it runs, no matter how clean it looks, and a database that finds a thousand of them an hour will not tell you either. It just gives you a thousand questions instead of one.

In this case the early read is that the gaps are real. The conditions hold up, the absence looks like nobody having got there rather than everybody having failed. So we're doing the only thing that actually settles it. We're running small tests in a few of those places to see if the demand answers back. I can't tell you how it lands, because we don't know yet. That's the honest state of it.

The thing I'd hand to a sceptic is this. Free data is free because it arrives raw, undifferentiated, the same wall of rows for everyone who bothers to download it. There's no edge in the data. The edge is in the join. Public numbers crossed with your own records, read by someone who knows the business well enough to tell a real gap from a mirage, produce something neither half had on its own.

That's the part worth paying for, and it's the part a dashboard can't do for you. The database finds the correlation in seconds. Deciding whether it means anything is still a day's work and a few quiet tests. Worth doing. Just don't mistake the first step for the last one.