Tom Coupé with 10 examples of business experiments, including dating without pictures, women who make you want to spend, Googling up smaller snacks, Bill Gates & chickens, moving from Tonga to NZ causes stress & more

Today's Top 10 is a guest post from Tom Coupé, an Associate Professor in the Economics and Finance Department at Canterbury University.

As always, we welcome your additions in the comments below or via email to david.chaston@interest.co.nz.

And if you're interested in contributing the occasional Top 10 yourself, contact gareth.vaughan@interest.co.nz.

See all previous Top 10s here.

Improving performance by learning from experiments

For businesses and countries to be successful, it’s important to understand what helps them to perform and what hinders their performance. But given that so many factors can potentially affect their performance, distinguishing between what matters and what doesn’t matter is far from easy.  My top 10 is about one increasingly popular way to get to correct answers, the use of real life experiments by both businesses and researchers. If done well, experiments can provide correct answers because they allow us to effectively separate the influence of one specific driver of performance, from all other possible drivers of performance. 

In this top 10, you’ll find examples of businesses doing experiments to understand their consumers better or to increase their sales, businesses experimenting on their employees and researchers experimenting on themselves. You will also find some articles with practical advice on how to run a business experiment as well as an example of a ‘natural’ experiment in New Zealand.

1. We experiment on human beings (so does everybody else).

Let’s start with a fun, insightful, well-written blog post by OKCupid, a dating website, which describes what happened when they removed pictures from their website and what happened if they told people they were great matches even though their internal algorithm, which predicts the quality of matches, suggested the opposite.

“I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.”

2. This Lingerie Company A/B Tests The World's Hottest Women To See Who Makes You Click "Buy".

Next, a great illustration that there is nothing random (or better, that nothing has to be random) in what you see when browsing through a company catalogue. But also that getting rid of randomness can create a competitive advantage.

 “Sex doesn't sell, so forget the boudoir shot. Blondes don't work. Props distract. Couches are fine. Playing with hair is ideal. Those are some of the insights the lingerie company Adore Me has learned from testing the photos of models wearing its sexy products online. For each bra, Adore Me shoots multiple versions of images to run on its website. The distinctions between the pictures might include different models wearing the same set in the exact same position, or the same model in the same set in a different position, for example. Then, like any good tech company, it tests the options to find out which one sells better.”

3. “How Google Optimized Healthy Office Snacks”.

Companies not only experiment on their clients, companies can also experiment on their employees. Here an example of how Google tried to improve the eating habits of its employees.

“In a field experiment in another Google microkitchen, we targeted the most popular snack item: bulk M&Ms. They had been self-serve from bulk bins into four-ounce cups; most employees filled the cup. After taking a baseline measure of consumption, we replaced loose M&Ms with small, individually wrapped packages. This simple intervention reduced the average serving by 58%, from 308 calories to 130.”

4. “Paid Search Ads Pay Off for Lesser-Known Restaurants”.

Experiments can also be used to answer more general business questions like ‘does advertising work?’. Most businesses hope it does but very few will be able to give you convincing estimates of the return on investment in advertising. Michael Luca of Harvard Business School and Weijia Dai of Lehigh University teamed up with Yelp to see whether restaurants benefit from advertising their services on Yelp. To be able to answer this question, they gave away for free millions of dollars of advertisement space. Maybe interest.co.nz too is interested in finding out how effective advertising on its website is? 

“Luca proposed a large-scale experiment in which Yelp would give away several months of advertising spots for free to randomly selected businesses. “Geoff sort of cringed,” Luca recalls, at the thought that he would effectively be giving away the equivalent of millions of dollars worth advertising exposures.”

5. Bill Gates wants to give the poor chickens. What they need is cash.

It is not only business that can benefit from experiments. Governments can do too, as experiments can be used to answer important public policy questions. In this article, Christopher Blattman, professor at the University of Chicago, explains to Bill Gates why he thinks it is better to give cash to poor people, rather than giving chickens, and what can be done to figure out who has the right answer.

“I’m also writing to say that both of us are counting our chickens before they hatch. (Sorry — I couldn’t resist.) Given the state of current research, neither of us can say definitively whether livestock or cash is better — or under what circumstances one might be better than the other. But the answer is in easy reach if only someone would conduct a thorough experiment to test the two approaches. It is irresponsible of us not to do so.”

6. Using Randomized Trials in Policy: Oliver Hauser on the UK Behavioural Insights Team.

In the United Kingdom, there is even an organization, the Behavioural Insights Team, which started as a government unit but is now co-owned by the UK government, and does experiments to try to improve government policy.

“To illustrate these, Oliver shows us a trial they ran on the "Tax Return Initiative" with HM Revenue and Customs. Famously, they tried to prime people to be more honest on tax payment. They ran an RCT including a link directly to the tax form rather than a link to the website. In a randomized trial of approximately 2,000 people in each group, click-through rates were 23.4% compared to 19.2%. In more recent study of tax forms in Guatemala, they have carried out several million observations.”

7. Just do it? But HOW? 24 productivity experiments I tried, plus a QS time management recap.

Arguably less scientific and fraught with methodological issues, but at least as amazing: self- experimentation. In psychology and medicine, there are many examples of experiments that researchers run on themselves. I am not aware of academics in economics or business doing this but I found this relevant example:

“Some time ago I was asked for the ultimate productivity tip, and instead of giving a straightforward take-away, I said that in the end the answer is “it depends.” That wasn’t a cheap shot because what works for you might not work for the next guy, and vice versa. Sound familiar? It’s the same case for medications, meditation, and most anything else we humans do. That’s why it’s best to experiment, examine your results, and decide based on the data. In other words, quantify!”

8. Why Businesses Don’t Experiment.

While more and more businesses do run experiments, many firms refrain from doing so. Dan Ariely of Duke University discusses some reasons why companies fail to take advantage of the benefits of doing experiments.

“Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups—a dozen people riffing on something they know little about—to set strategies. And yet, companies won’t experiment to find evidence of the right way forward.

I think this irrational behaviour stems from two sources. One is the nature of experiments themselves. As the people at the consumer goods firm pointed out, experiments require short-term losses for long-term gains. Companies (and people) are notoriously bad at making those trade-offs. Second, there’s the false sense of security that heeding experts provides.”

9. How to Design (and Analyze) a Business Experiment.

The rational managers of companies and government agencies interested in doing experiments and benefiting from them can find some practical advice in this article

“This article lays out seven steps to ensure that your experiment delivers. These principles draw on the academic research on field experiments as well as our work with a variety of organizations ranging from Yelp to the UK government.”

10. Natural Experiment Evidence on the Effect of Migration on Blood Pressure and Hypertension.

What about New Zealand? I am sure there are businesses and possibly even government agencies in New Zealand that are running experiments. My Google search didn’t give me any examples, however. (If you know of examples, please leave them in the comments or email me!) but I did find this study of a so-called ‘natural’ experiment created by New Zealand’s immigration system. In New Zealand, a lottery has been used to select some applicant-migrants to receive permanent residency. This lottery acts just like an experiment, and by comparing applicants who won the lottery to those who didn’t one can find out how migrating to New Zealand affects, for example, one’s health. Interestingly, the results show migration from Tonga to New Zealand increases blood pressure and stress.

“In this paper, we use a natural experiment, comparing successful and unsuccessful applicants to a migration lottery to experimentally estimate the impact of migration on measured blood pressure and hypertension. …

 In 2002, another channel was opened up for immigration to New Zealand through the creation of the Pacific Access Category (PAC). This allows for a quota of 250 Tongans to immigrate to New Zealand each year regardless of their skill level or socioeconomic status. …

Many more applications are received than the quota allows, so a ballot is used by the New Zealand Department of Labour (DoL) to randomly select from amongst the registrations”

We welcome your help to improve our coverage of this issue. Any examples or experiences to relate? Any links to other news, data or research to shed more light on this? Any insight or views on what might happen next or what should happen next? Any errors to correct?

We welcome your comments below. If you are not already registered, please register to comment or click on the "Register" link below a comment.

Remember we welcome robust, respectful and insightful debate. We don't welcome abusive or defamatory comments and will de-register those repeatedly making such comments. Our current Comment policy is here.

5 Comments

A comment from a minor bit player

Your article stimulates some interesting thoughts. The main areas I would like to comment on if I may are (of course I may be wrong and may have misinperpreted your message)

What you espouse is "behavioural economics" as distinct from micro and macro disciplines. Your article is most refreshing in that the vast majority of articles written and published in New Zealand are Macro studies which I find less than useful most of the time

But, hey, that's what we get.

New Zealand is a market of 4½ million people which is in fact a crucible of large experiments. To be of any use one needs to be able to assess and review and analyse the result of the experiments. Problem in the New Zealand context we never get the data so we cant evaluate the results

Take the immigration experiment
We should be tracking the economic production for every immigrant brought in under the HNW Investment visa category and the Skilled Work Visa. We have their date of arrival, passport number, their IRD number, annual earnings, tax paid, job changes, industry employed in, time spent with each employer, medical subsidies, welfare costs etc

But we don't. We keep splurging out macro GDP estimates, GDP per capita estimates. Every report and every article is littered with the word "estimates"

Wow. That makes so much sense. And dosen't seem that hard to do! We could do it at a high level so we don't go down to an individual level. That would be a start. Wow

Agreed, an interesting and thought provoking contribution. We all have to get along forming our beliefs on some blend of personal experience and in rarer cases on scientific method. Of course scientific method may require conclusions to change as more evidence comes along even in this "don't confuse me with the facts" world.
On a earthier level and well outside the 'opinion only' world, gardeners might be interested in a series of practical videos examining common assumptions in the backyard food growing world.
https://www.youtube.com/watch?v=_9WoW8Rtp2E&list=PL5mfR-r4BXH3UTGH_3UAG6...

We should experiment with UBI here. The only reason the Government doesn't want to is that if it works it undermines their faulty ideology, and it could transform NZ into world beaters. A small scale experiment here would produce useful information. If it would produce start ups and more IPOs we could produce a lot more technical value and intellectual property.

There's plenty of people with vision in NZ but the Government is focused on creating roadblocks for all businesses.

Let's obtain the results-data and analysis from Finland who are conducting a UBI experiment