Today's Top 10 is a guest post from Tom Coupé, an Associate Professor in the Economics and Finance Department at Canterbury University.
As always, we welcome your additions in the comments below or via email to firstname.lastname@example.org.
And if you're interested in contributing the occasional Top 10 yourself, contact email@example.com.
Improving performance by learning from experiments
For businesses and countries to be successful, it’s important to understand what helps them to perform and what hinders their performance. But given that so many factors can potentially affect their performance, distinguishing between what matters and what doesn’t matter is far from easy. My top 10 is about one increasingly popular way to get to correct answers, the use of real life experiments by both businesses and researchers. If done well, experiments can provide correct answers because they allow us to effectively separate the influence of one specific driver of performance, from all other possible drivers of performance.
In this top 10, you’ll find examples of businesses doing experiments to understand their consumers better or to increase their sales, businesses experimenting on their employees and researchers experimenting on themselves. You will also find some articles with practical advice on how to run a business experiment as well as an example of a ‘natural’ experiment in New Zealand.
Let’s start with a fun, insightful, well-written blog post by OKCupid, a dating website, which describes what happened when they removed pictures from their website and what happened if they told people they were great matches even though their internal algorithm, which predicts the quality of matches, suggested the opposite.
“I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.”
Next, a great illustration that there is nothing random (or better, that nothing has to be random) in what you see when browsing through a company catalogue. But also that getting rid of randomness can create a competitive advantage.
“Sex doesn't sell, so forget the boudoir shot. Blondes don't work. Props distract. Couches are fine. Playing with hair is ideal. Those are some of the insights the lingerie company Adore Me has learned from testing the photos of models wearing its sexy products online. For each bra, Adore Me shoots multiple versions of images to run on its website. The distinctions between the pictures might include different models wearing the same set in the exact same position, or the same model in the same set in a different position, for example. Then, like any good tech company, it tests the options to find out which one sells better.”
Companies not only experiment on their clients, companies can also experiment on their employees. Here an example of how Google tried to improve the eating habits of its employees.
“In a field experiment in another Google microkitchen, we targeted the most popular snack item: bulk M&Ms. They had been self-serve from bulk bins into four-ounce cups; most employees filled the cup. After taking a baseline measure of consumption, we replaced loose M&Ms with small, individually wrapped packages. This simple intervention reduced the average serving by 58%, from 308 calories to 130.”
Experiments can also be used to answer more general business questions like ‘does advertising work?’. Most businesses hope it does but very few will be able to give you convincing estimates of the return on investment in advertising. Michael Luca of Harvard Business School and Weijia Dai of Lehigh University teamed up with Yelp to see whether restaurants benefit from advertising their services on Yelp. To be able to answer this question, they gave away for free millions of dollars of advertisement space. Maybe interest.co.nz too is interested in finding out how effective advertising on its website is?
“Luca proposed a large-scale experiment in which Yelp would give away several months of advertising spots for free to randomly selected businesses. “Geoff sort of cringed,” Luca recalls, at the thought that he would effectively be giving away the equivalent of millions of dollars worth advertising exposures.”
It is not only business that can benefit from experiments. Governments can do too, as experiments can be used to answer important public policy questions. In this article, Christopher Blattman, professor at the University of Chicago, explains to Bill Gates why he thinks it is better to give cash to poor people, rather than giving chickens, and what can be done to figure out who has the right answer.
“I’m also writing to say that both of us are counting our chickens before they hatch. (Sorry — I couldn’t resist.) Given the state of current research, neither of us can say definitively whether livestock or cash is better — or under what circumstances one might be better than the other. But the answer is in easy reach if only someone would conduct a thorough experiment to test the two approaches. It is irresponsible of us not to do so.”
In the United Kingdom, there is even an organization, the Behavioural Insights Team, which started as a government unit but is now co-owned by the UK government, and does experiments to try to improve government policy.
“To illustrate these, Oliver shows us a trial they ran on the "Tax Return Initiative" with HM Revenue and Customs. Famously, they tried to prime people to be more honest on tax payment. They ran an RCT including a link directly to the tax form rather than a link to the website. In a randomized trial of approximately 2,000 people in each group, click-through rates were 23.4% compared to 19.2%. In more recent study of tax forms in Guatemala, they have carried out several million observations.”
Arguably less scientific and fraught with methodological issues, but at least as amazing: self- experimentation. In psychology and medicine, there are many examples of experiments that researchers run on themselves. I am not aware of academics in economics or business doing this but I found this relevant example:
“Some time ago I was asked for the ultimate productivity tip, and instead of giving a straightforward take-away, I said that in the end the answer is “it depends.” That wasn’t a cheap shot because what works for you might not work for the next guy, and vice versa. Sound familiar? It’s the same case for medications, meditation, and most anything else we humans do. That’s why it’s best to experiment, examine your results, and decide based on the data. In other words, quantify!”
While more and more businesses do run experiments, many firms refrain from doing so. Dan Ariely of Duke University discusses some reasons why companies fail to take advantage of the benefits of doing experiments.
“Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups—a dozen people riffing on something they know little about—to set strategies. And yet, companies won’t experiment to find evidence of the right way forward.
I think this irrational behaviour stems from two sources. One is the nature of experiments themselves. As the people at the consumer goods firm pointed out, experiments require short-term losses for long-term gains. Companies (and people) are notoriously bad at making those trade-offs. Second, there’s the false sense of security that heeding experts provides.”
The rational managers of companies and government agencies interested in doing experiments and benefiting from them can find some practical advice in this article
“This article lays out seven steps to ensure that your experiment delivers. These principles draw on the academic research on field experiments as well as our work with a variety of organizations ranging from Yelp to the UK government.”
What about New Zealand? I am sure there are businesses and possibly even government agencies in New Zealand that are running experiments. My Google search didn’t give me any examples, however. (If you know of examples, please leave them in the comments or email me!) but I did find this study of a so-called ‘natural’ experiment created by New Zealand’s immigration system. In New Zealand, a lottery has been used to select some applicant-migrants to receive permanent residency. This lottery acts just like an experiment, and by comparing applicants who won the lottery to those who didn’t one can find out how migrating to New Zealand affects, for example, one’s health. Interestingly, the results show migration from Tonga to New Zealand increases blood pressure and stress.
“In this paper, we use a natural experiment, comparing successful and unsuccessful applicants to a migration lottery to experimentally estimate the impact of migration on measured blood pressure and hypertension. …
In 2002, another channel was opened up for immigration to New Zealand through the creation of the Pacific Access Category (PAC). This allows for a quota of 250 Tongans to immigrate to New Zealand each year regardless of their skill level or socioeconomic status. …
Many more applications are received than the quota allows, so a ballot is used by the New Zealand Department of Labour (DoL) to randomly select from amongst the registrations”