Navigating the treacherous waters of ethics compliance for field experiments

[The following contribution is the unedited version of the comment that The Australian published today under a different title. The Australian is pay-walled, so I put the draft here, with a few links added for good measure. For the record, I prefer the headline that I proposed … ]

Two years ago, University of Queensland (UQ) researchers Paul Frijters and Redzo Mujcic published a couple of working papers that documented substantial, statistically significant discrimination on the part of Brisbane bus drivers, especially against people of colour. Interestingly, the authors also found that Asians were not discriminated again, a finding which I find hard to reconcile with the anecdotal evidence I have seen in Sydney, for example. There were many other important findings in the Mujcic-Frijters study which quantified various aspects of discrimination in an innovative manner that could be a template for further studies.

UQ initially issued a press release to announce these findings and outlets such as The Sydney Morning Herald picked up the study that curiously – given its important quantification of white privilege — was never published and did not get publicity afterwards. Last week, in the wake of two contributions in The New York Times and Forbes by Yale Law School professor Ian Ayres, we learned why. Not a pretty story it is.

According to numerous contributions in Core Economics Today, The Australian, The Brisbane Times, The Washington Post, and many others, it seems that a day after the press release, UQ administrators told Frijters – the supervisor of then-Ph.D. candidate Mujcic – that the research had to be pulled out of the media and was not to be published because it had not received appropriate ethics clearance. It also has emerged that this discovery seems to have been prompted by the Brisbane bus company (Translink) media adviser asking how a study that seemed to involve fare evasion could receive ethics clearance.

Fare evasion? No, not really for all I know. Frijters & Mujcic had 29 young adult testers – duly mixed by gender, ethnicity, and attire – board Translink buses and insert an insufficiently funded fare card into the scanner. The testers then told the bus-drivers, “I do not have any money, but need to go to [a station about 1.2 miles away]”. The bus-drivers were thus prompted to make a call whether to allow the testers to stay onboard.

In the flurry of contributions and discussions in various outlets over the last few days, facts and open questions have emerged. Was deception involved? (Having written extensively about the issue of deception in economics and other social sciences, I believe, no: The testers had insufficiently funded fare cards and as part of their research had to go to the destination that they gave the bus-drivers. Hence there were no deceptive statements. Of course, bets are off on this one if the testers had indeed money on them and / or valid fare cards.) Did Frijters, Australia’s Young Economist of the year in 2009 and by all measures one of Australia’s most productive economists, follow the ethics procedures then in place at UQ? (If indeed no deception was involved, apparently, yes.) Was it fare evasion? (No. The bus drivers could decline the request for a free ride.) Were bus drivers entitled to let people free ride that had insufficiently funded fare cards? (Apparently, as persuasively argued by Rabee Tourky in Core Economics Today, yes. According to him, in the wake of the Daniel Morcombe murder Brisbane bus drivers were instructed to use judgment when such requests were made.) Should the bus company have been informed about this research beforehand? Should bus-drivers have been debriefed that they just had participated in an experiment? Was it appropriate for the researchers to publish location and carrier?

Some of these questions do not have easy answers, as is becoming now clear. With the benefit of hindsight, ethics clearance on a higher level might have been a good idea, be it only to protect the researchers from over-the-top administrative responses including at some point a demotion for Frijters that UQ ultimately had the good sense to reverse and Mujcic’s inability to get his dissertation research published. Hindsight, of course, is 20/20. As matter of fact, likely Nobel Prize laureate John A. List has defended field experiments that involve lack of ex-ante informed consent, or ex-post debriefing in the 11 July 2008 issue of Science, a leading and highly influential general-interest magazine. Arguing that notifying experimental subjects is often neither possible nor necessary because of the minimal risk that is involved, he puts it thus: “[I]n a natural field experiment, the analyst manipulates experimental conditions in a natural manner, whereby the experimental subjects are unaware that they are participating in an experiment. This approach combines the most attractive elements of the laboratory and of naturally occurring data: randomization and realism.” When challenged on this proposition, he responded in the 31 OCTOBER 2008 issue thus:“Ethical issues surrounding human experimentation are of utmost importance. Yet, the benefits and costs of informed consent should be carefully considered in each situation. Those cases in which there are minimal benefits of informed consent but large costs are prime candidates for relaxation of informed consent.” This seems to describe the Frijters & Mujcic study well and is in any case, for better or worse, an increasingly accepted point of view among behavioural and experimental economists.

Ethics clearance requirements – implemented in response to scores of well-documented examples of Human Subject Research abuses (google them!) and universities’ concerns for liability and reputation — have become a considerable drain on researchers’ scarce resources in Australia, and – it has to be said — other places. I am sure every behavioral and experimental economist here in Australia will attest to that. Occasionally ethics clearance requirements are applied in questionable ways in that the gate-keepers feel entitled to make judgment calls on the quality of research. I consider this an unacceptable infringement of academic freedom. It is in any case clear that – in light of List’s position, for example, as well as the Frijters & Mujcic situation — it is time to have a conversation about these issues, so as to lift the administrative and regulatory fog.

That ethics clearance requirements have escalated, and unreasonably so in many a case by List’s standard, seems widely acknowledged in that a number of universities in Australia have fast-track procedures for low-risk research. It seems that UQ in 2012, when Frijters & Mujcic did their study, had such procedures in place although apparently there was some wiggle-room for interpretation. The Frijters-Mujcic study hence seemed to warrant at best an updating of procedures that were found wanting. I simply do not see a case for suppressing this research given its unquestionable importance, and given that no deception seems to have been involved.

Academic freedom is an important good and should be defended vigorously. It is deplorable that the ongoing UQ saga has come at a high cost to Frijters and his former Ph.D. student. Their time could have spent much more productively although I am confident that the saga has provided Frijters already with many examples for his ARC-funded research into socially undesirable behavior.

Given that UQ administrators continue to insist that the university has responded appropriately, and given the facts that have emerged over the last few days, it seems clear that UQ administrators at this point are an interested party and as such ought to excuse themselves from further attempts at assessments of what happened. The accusation of maladministration that is now in the open, is too important an accusation, and ought to be dealt with by truly independent investigators. UQ, and Frijters, should commit to implement whatever recommendations such investigators come up with. It is to be hoped that Frijters soon can do again what he does best: Research that addresses important real-world problems in innovative ways and that gets noticed, and recognized world-wide, for exactly that reason.

27 thoughts on “Navigating the treacherous waters of ethics compliance for field experiments”

  1. I apologize for repeating a question from a previous post, but I’d honestly like to hear a convincing argument that having a tester wear a replica Australian army uniform does not constitute deception, if the tester is not a member of the Australian army.

    (There’s nothing from what I can tell in the research report to indicate whether the testers were all in the army, but the recruitment mechanism did not seem likely to produce an all-army set of testers.)

    Like

    1. I believe that Gigi Foster answered your question already; I can not improve on her answer.

      Like

      2. That’s the response I was referring to. I would argue that it is neither illegal nor deceptive, at least if the testers did not deviate from the script. If there was no deviation from the script there was at best an invitation to a misleading inference. Most economists would not consider this deception although some might be uncomfortable about some such strategy in the context of lab experiments (due to reputational consequences for the lab, which in a field setting are something most economists do not worry about much typically.)

        Like

      3. I don’t want to belabor the point, but “I would argue that…” is a statement about the potential of an argument, and the statement about most economists is an unsubstantiated claim about what a group of persons would think.

        I’m interested in an actual argument that it is not deception to attempt to make bus drivers believe something that is not true, which in this case is the belief that the tester was a member of the Australian army.

        Like

  2. Nice op-ed.

    If they had no money on them, how do they get back from their destination? As you say:
    “The testers had insufficiently funded fare cards and as part of their research had to go to the destination that they gave the bus-drivers. Hence there were no deceptive statements. Of course, bets are off on this one if the testers had indeed money on them and / or valid fare cards.”

    Like

    1. Good questio. The authors give some indication on p. 8 of the study (December 2014 version, the one accessible on Ayres site.)

      Like

  3. To JLZ (can’t reply directly to your ” I don’t want to belabor … ” argument so here goes): The world is not black – white much as you might like to believe so. There is in this case a grey zone where reasonable people can disagree (and where the argument hinges possibly on important details). You may want to read up on current discussions of deception in economics. Such as this … see the many references in there, too: http://spq.sagepub.com/content/71/3/222.refs

    Like

    1. L.J Zigerell- you can hire an army costume for a fancy dress party which you are invited to and catch public transport to get there. This sort of thing happens all the time. If it was illegal, the stores would not stock them.

      From the paper, I can see that the RA’s never told the bus driver or anyone that they were actual army personnel (which they weren’t in any case). It is like wearing your national team’s football shirt in public or to a stadium; a lot of people would notice it, some may react positively, some (rivals) are likely to react in a negative manner by shouting or even picking a fight with you. Some people even wear clothes to make others believe that they belong to another group; for example, you can wear the opposite team’s shirt. This is life. So should we go out and question every single person on the street and ask them why they are wearing a particular uniform? C’mon, grow up!

      Like

      1. Hi Zin. Your argument about legality would be relevant if I had made a claim about legality. I’m also not sure why you would discuss the possibility of questioning persons wearing uniforms on the street.

        —

        I think that the bus bias study is great and important research, but I do not see how the study is exempt from clearance at the institutional level. The strongest argument is that: (1) Research at the University of Queensland that involves deception must receive clearance at the institutional level; (2) The bus bias research design was for testers to wear replica army uniforms so that the bus drivers would believe that the tester was in the army; (3) That belief that appears to be false, which means that the study involved deception and thus needed clearance at the institutional level.

        I don’t see how your comment breaks that chain of reasoning.

        For what it’s worth, the best argument that I have been able to come up with is that the deception in this case was not necessary to the study because the study would have been the same had authentic army personnel been used as testers or if the testers had been persons who honestly lacked funds and sincerely needed to travel to station X. However, the fact that the study could have been conducted without deception does not appear to erase any deception that was used, for the purposes of determining the required level of ethics clearance.

        —

        Your football shirt example requires more context: sometimes wearing a team shirt is not deceptive, and sometimes wearing a team shirt is deceptive. It would be deceptive if I wore a team shirt in order to gain entry to a secured area because my plan was to make a security guard think that I was a member of the football team.

        Similarly, it would not be deceptive to simply wear a replica police officer uniform to a Halloween party with friends. However, it would be deceptive to wear a replica police officer uniform, walk down the street, and ask complete strangers incriminating questions and then ask to see their driver license.

        Like

      2. You are quite the troll, aren’t you? Once more: Why don’t you first make yourself knowledgeable about what constitutes deception in academic research in economics and what not. See my earlier comment and reference(s) … . Repeating the same flawed chain of reasoning doesn’t improve the quality of your arguments.

        Like

      3. Hi Andreas.

        Your article states: “Economists’ proscription of deception targets the explicit and intentional provision of erroneous information” (p. 223). Presuming that some testers were not in the army, it seems that deception is involved in purposefully wearing a costume (“explicit and intentional provision”) to invoke a false belief about the tester being in the army (“erroneous information”).

        Your article seems to restrict the definition of “explicit” to verbal statements (“in other words, lying” p.222), but I don’t see anything in the dictionary or in common usage that would require that restriction. If you know that the true response to a numeric question is three, I can’t imagine that saying “one” is deception but raising a single finger is not deception.

        Imagine a research study that involves a six-year-old child walking down the street dressed as a clown pointing an orange toy gun at a pedestrian and then saying “I’m a police officer: put your hands up.” Then imagine a research study that involves a muscular 30-year-old walking down the street in a replica police uniform pointing a replica gun at a pedestrian and saying “Put your hands up.” If I am interpreting your arguments correctly, only the study involving the six-year-old child would involve deception.

        —

        I’m honestly interested in an argument that deception is not involved in purposefully wearing a costume to invoke a false belief, because that appears to be a necessary argument to make in order for the bus bias study to have not needed ethical clearance at the institutional level. I have not seen that argument made, and I don’t see any reason to continue this thread unless such an argument is forthcoming.

        Like

      4. Andreas, LG Zigerell has made a patient (and, in my view, relevant) argument. I don’t see how anything s/he says merits you calling him/her a troll.

        Like

      5. I stand corrected, Jeremy. His follow-up clarified that he had an interesting point that, however, was not clear to me from what his previous post.

        L.J Z, yes, when we wrote our papers we restricted the definition of “explicit” to verbal statements (“in other words, lying” p.222), that’s because our work on deception was dealing with lab situations.

        Your point is an interesting one and one that I had not reflected on before, again that is because these kind of questions do not come up in the lab, at least not in economics experiments …

        What happened in this experiment is not completely clear; see p. 9 of the December 2014 version of their paper. Mujcic & Frijters employed both students from various faculties and non-student members of the outside community. Were false beliefs invoked with replicas? Was it deception? If, for the sake of the argument it was, should such a study (whose cleverness and importance you seem to acknowledge) be approved because of its minimal- risk nature?

        Like

      6. “If, for the sake of the argument it was, should such a study (whose cleverness and importance you seem to acknowledge) be approved because of its minimal- risk nature?”

        Hi Andreas. I would vote to approve the study if I were on the ethical review board, and I would recommend that the manuscript be published if I were a peer reviewer and the study had the required ethical clearance. I’d even argue that the deception in the study was not necessary to the study, if that loophole were relevant to the manuscript being published without the required ethical clearance.

        Like

  4. Andreas – Your argument about what economists think deception (or isn’t) and what the rules should be isn’t as interesting as what the rules actually are. I think we’re all in agreement that the Human Ethics rules are largely – but not always – inappropriate (and very onerous) for the kind of work that gets done in Business and Commerce faculties. The University of Queensland – like all Australian universities – would base its human ethics research committee work on the National Health and Medical Research Council guidelines. Those guidelines, going back to at least the late 1990s/early 2000s are remarkably consistent on this point.

    From the 2001 Human Ethics Handbook:
    “As a general principle, deception of, concealment of the purposes of a study from, or covert observation of, identifiable participants are not considered ethical because they are contrary to the principle of respect for persons in that free and fully informed consent cannot be given.”
    Now, of course, there are some kinds of research that can only be conducted is that manner. So the Handbook spells out the safeguards, including:
    “(f) participants will be able to withdraw data obtained from them during the research without their knowledge or consent; and
    (g) such activities will not corrupt the relationship between researchers and research in general with the community at large.”
    So the bus drivers and the bus company have simply exercised their rights under the guidelines to withdraw their data.

    Now, even if the argument that there was no deception involved is accepted (for argument sake) then there is still the problem of “Limited Disclosure” as outlined in the 2007 National Statement on Ethical Conduct in Human Research.
    “Research involving limited disclosure covers a spectrum, from simply not fully disclosing or describing the aims or methods of observational research in public contexts, all the way to actively concealing information and planning deception of participants. Examples along the spectrum include: observation in public spaces of everyday behaviour; covert observation, for example of the hand-washing behaviour of hospital employees; undisclosed role-playing by a researcher to investigate participants’ responses; telling participants the aim of the research is one thing when it is in fact quite different.”
    So if not actual deception, at the very least, there is undisclosed role playing in this study. Again, this is permitted once the appropriate safeguards are met, including:
    ” (e) whenever possible and appropriate, after their participation has ended, participants will be:
    (i) provided with information about the aims of the research and an explanation of why the omission or alteration was necessary; and
    (ii) offered the opportunity to withdraw any data or tissue provided by them.”
    There is more, but I think we all get the idea.

    There is a breach of the NHMRC guidelines on research ethics involving human subjects. These guidelines have been in place for over a decade, if not much longer.

    To be sure reasonable people can disagree on what the rules should be, but as far as I’m aware this is what the rules actually are.

    To me it looks like the bus company, on behalf of its employees, has withdrawn permission for the use of the data. The University of Queensland has complied with that request.

    Like

    1. That’s not my understanding, Sinclair, but then I am not privvy to procedural details. And that’s for the better in this saga as far as I am concerned.
      Yes, I think we’re all in agreement that the Human Ethics rules are largely – but not always – inappropriate (and very onerous) for the kind of work that gets done in Business and Commerce faculties. That’s why I wrote that we need to have a discussion about these kind of things lest we cannot do research here in Australia that researchers can do elsewhere without problem. (Recall my List reference.)

      It seems that even at the U of Q studies without informed consent have been approved — see the Mazerolle et al article that someone posted in this thread.

      As to the 2001 Human Ethics Handbook excerpt, the operative word seems ot be “identifiable participants”; the data here were completely anonymized if I understand correctly, so that passage does not seem to apply.

      And, again, back to the key questions: Who was possibly at risk and what was that risk?

      Like

      1. The identifiable participants are the bus drivers. Brisbane bus drivers are now at risk of being stereotyped as being “racist”.

        Furthermore the study involved the bus drivers being approached after the fact and being quizzed on their choices. That in itself is a survey (if not actually stalking) requiring a plain language statement, and informed consent, and ethics approval.

        The problem with this study is that it required ex ante approval and did not get that approval. To be fair it looks like the authors were advised that they didn’t need approval (or that approval was granted).

        Like

  5. http://link.springer.com/article/10.1007/s11292-012-9160-1

    http://www.ceps.edu.au/programs/vulnerable-communities/queensland-community-engagement-trial-qcet-

    NO informed consent; wasted people’s times; ‘business as usual’ experiment; randomized field experiment; the researchers were academics using Qld police as experimenters; the subjects were uninformed citizens of Qld; recently published in the ‘Journal of EXPERIMENTAL Criminology’.

    Jim Rose, LJ Zigerell and others please enjoy.
    p.s. imagine how many of the people being stopped in this study felt bad afterwards, depressed, questioned their appearance etc. and then went home in a reduced mental state, or missed their important job interview/kid’s birthday/hospital visit of a loved one?!

    Like

    1. Hi R.

      The question that has been under discussion is whether the bus bias study had the required ethical clearance. I’m not sure how the linked study addresses that question.

      If you have evidence that the linked study lacked the required ethical clearance, then we can start a discussion about how unfair it is that one study is published and the other study is suppressed.

      Like

    2. New Zealand privacy law allows employers to have mystery shoppers come along to test customer service. However, the employer must notify staff that a mystery shopper will be coming along in the next few weeks.

      Queensland has similar information privacy principles in its privacy laws, but how they interact and are enforced on academic organisations is not easy to obtain online. Different rules appear to apply to employers.

      The studies you referred to were done in conjunction with the Queensland police.

      Like

  6. L.J Zigerell ,akes a lot of valid points about verbal versus non-verbal deception.

    Most of all, the experiment itself was about non-verbal cues.

    It was about how bus drivers react to the information they get from the way people look, the way they dress and how they react differently depending on differences in these details.

    Many a con game is based on the way the con artist looks. They look like a trustworthy person, smile a lot, they dress well, or use a lot of other non-verbal props.

    The entire trade of magicians is based on misdirection. While they are talking to you, they do something in the background that fools you.

    This debate is not about academic ethics. It’s about honesty of dealings with others and the extent to which they comply with the law.

    Like

    1. Hi Jim,

      from what I can see, it must have been some experiment for yourself, LJ, and Jeremy to be so intensely captured by it. This is usually what happens when something very important is done, and then non-researchers start picking on and arguing about the small (mostly irrelevant) details.

      I don’t see how con artists, magicians, witches, wizards, mind readers etc. are relevant here (please read Andreas’ above academic article more carefully).

      Like

      1. The whole point of ethics panels including outsiders who are just ordinary members of the public with no association with the profession is the proposed project passed the normal sniff test of an ordinary person regarding how you should deal with strangers.

        Like

Comments are closed.