Jun

20

Randomised Business Trials

June 20, 2008 | 2 Comments | Joshua Gans

There has been lots of discussion about the need for randomised trials in policy evaluation (just google “Andrew Leigh” and “randomised” to see what I mean). However, the same logic — that choices can use evidence — applies to business as well. In particular, David Pogue is interested in whether electronic books will deter book piracy. As a result, he and his publishers are proposing the following:

Early next month, the company will also start selling electronic versions of certain books with no copy protection. For a single price (cheaper than the printed-book price), the package will include the book in three formats: PDF, Mobipub (compatible with the Amazon Kindle), and Epub (soon to be compatible with the Sony Reader).

Anyway, I’ve agreed to try an experiment involving one of my books (“Windows Vista: The Missing Manual”): to offer it as part of that buy-the-electronic-versions program.

O’Reilly is also offering five other books in these formats, including Information Architecture for the World Wide Web, Third Edition and Open Sources 2.0. The company plans to announce other titles in July.

Now how Pogue and his publisher will use and analyse the data is still open. But it will be there. However, it is not clear how randomised the trial is. How were these candidate books selected? If it was out of hat then fine. But also, there is some contamination coming from the announcement in the New York Times. Nonetheless, at least it is a move in an interesting direction. Precisely the sort of thing that can help resolve the “someone is wrong on the Internet” feeling.


Comments

2 Responses to “Randomised Business Trials”

  1. Kevin Cox on June 20th, 2008 11:19 am

    Randomised trials are a nice idea for situations like medical experiments, finding a better breed of lupin. They are not necessary when it comes to information systems such as economic systems because as with the reported trial above if we collect appropriate information then the trial becomes an experiment. We can get the results hypothesise on why the results are the way they are and then change the system to see if our hypothesis is right or not.

    The problem with randomised trials is a practical one. Ok so you “prove” that there may be a causal effect but now you have to convince people to let you implement the change. By the time you get someone to implement your changes the world is now a different place and it is likely that what you have discovered is bypassed.

    With incremental experimentation or trials you keep learning and you do not have to convince people because they see it happening. That is, the development of the revised improved system is quicker and more certain and it will go in directions that you had no idea of when you started.

    Design economic systems to be learning experimental systems is likely to be a more productive approach than randomised trials.

    I would suggest business keeps doing things the way they are and not get into randomised trials. Not because randomised trials are wrong or a bad idea just that they are less effective than a dynamic learning system which is what experimental trials are all about.

  2. Club Troppo » Missing Link Daily on July 22nd, 2008 9:00 am

    [...] Joshua Gans joins Andrew Leigh in the Quest for the Holy Randomised Trial. [...]