Data Science Checklists (Part 2): Can Checklists Help?

Josh Feldman · February 2, 2020

Now that we have established the complexity of putting AI principles into practice, we can begin discussing whether checklists can help. At first glance, it would appear that they cannot. Checklists seem designed to handle simple problems, maybe complicated problems, but certainly not complex problems. In The Checklist Manifesto, Gawande writes that one way to understand the benefit of a checklist is that "[t]hey remind us of the minimum necessary steps and make them explicit" (36). This type solution seems well suited to simple and complicated processes. In a simple problem that can be solved via a recipe, this is exactly what one would be looking for: a tool that ensures that the most critical steps in the recipe are followed. In complicated tasks, a tool that reminds one of the minimum necessary steps to solve the problem can be useful given that expert knowledge is fallible. Ultimately, since both simple and complicated problems have solutions that can be applied in all instances of the problem with relative success, checklists can be written to capture this underlying order.

Note: The following three blog posts will accompany a package called "checklist" that I developed. After installing the package, it will trigger a checklist in the command line whenever users call "git push" if they also insert a short shell script in their shell configuration (available here).

In complex problems, this is not the case. Since each instance of the problem is different from the last, the solution is also dependent on the case at hand. A checklist designed to remind the user of the minimum necessary steps may help with parts of the problem that are invariant, but this tool would not help tame the underlying uncertainty. Returning to our example complex problem of raising a child, one could imagine a checklist including the item "feed them". This would certainly be useful advice to heed and is a necessary step in raising any child well. The issue is that this only handles a small pocket of order in an otherwise complex and uncertain problem. The checklist reminding a parent to feed a child may be useful, but it does not help a parent understand the aspects of raising their child that are specific to their son or daughter.

Checklists Can Handle Complexity

The most interesting argument made in The Checklist Manifesto is that, against our intuition, a well-designed checklist can help with complex problems too. Gawande offers the example of the construction industry to illustrate this surprising capability of checklists. AnOhio State University study found that the rate of serious building collapses was 0.00002% per year. How does the construction industry achieve such a high success rate, given the complexity of the task? The chaotic nature of the physical world and the individual features of each building means that there is high uncertainty going into any given project. Construction being complex does not preclude it from being complicated as well. Immense expertise is required to build a building safely. To handle the complicated components of the problem, interdisciplinary construction teams develop expansive checklists outlining every step necessary to complete the project. The interesting aspect of this workflow is what happens when unexpected obstacles inevitably occur. The checklist designed to handle the complicated aspects of the problem was replaced with another checklist, designed to handle the complexity of the problem. Gawande describes the complexity checklist as follows:

"It was also a checklist, but it didn't specify construction tasks; it specified communication tasks. For the way the project managers dealt with the unexpected and the uncertain was by making sure the experts spoke to one another—on X date regarding Y process. The experts could make their individual judgments, but they had to do so as part of a team that took one another's concerns into account, discussed unplanned developments, and agreed on the way forward. While no one could anticipate all the problems, they could foresee where and when they might occur."

By ensuring the correct experts communicated, the complexity checklist helped ensure that when unpredicted events occurred, the team got back on the right track. Even though the situations in which this checklist is used are unexpected with high degrees of uncertainty about the future, it demonstrates that there are still concrete steps that one can take to manage complexity. In the case of construction, this step was communication between experts. The checklist ensured this occurred.

There are three important features to highlight with this type of checklist. First, the checklist does not mandate what should be done to solve the problem. Because the problem is complex, such instructions do not exist. Second, following this checklist does not guarantee success in the same way that following a recipe does. The complexity of the problem can be managed, but not eliminated. Finally, though communication between experts is one approach to managing complexity, it is not the only one. This is a feature of checklists to manage complexity that Gawande overlooks. He insists that all checklists follow "the same basic design" of having "a mix of task and communication checks to manage the problem of proliferating complexity" (101). As will be demonstrated in Part 3, when we propose a checklist to help data scientists create algorithms that align with a given set of principles, more than mere communication between experts will be needed.

We can now return to the question of whether checklists can help data scientists put AI principles into practice. The answer is yes, but it is conditional on whether the checklist was designed with the complexity of the problem in mind. Two other groups have attempted to design such a checklist, but neither have managed to adequately address this complexity. Thefirst checklist was designed by Mike Loukides, Hilary Mason, and DJ Patil and thesecond, called deon, by the companyDrivenData. The second checklist is heavily influenced by the first but has additional checks in response to cases exposed by tech ethics journalism, which they feel the original checklist could not address. The problem with these two checklists is that they presuppose the data scientist has appropriate definitions for a number of contentious concepts, for example: fairness, harm, securing user data, clearly explaining what users are consenting to, and bias. As we established in part 1, much of the complexity involved in putting principles into practice arises from needing to define these terms. For instance, the insurance industry could check off the item "Have we tested for fairness with respect to different user groups?" for their policy pricing algorithms, but only with respect to their flawed and economically motivated notion of actuarial fairness. In our checklist, we will need to propose checks to create processes that help ensure these concepts are appropriately defined.

Defending Checklists from Being Co-Opted

A reasonable counterargument to the claim that checklists can help data scientists put AI principles into practice is that these checklists will be co-opted by technology companies. Ochigame makes the case for this general scepticism of self-regulation. Ochigame argues:

"The emphasis on ideas of "fair algorithms" and "ethical AI" constitutes a strategy for containing the controversies, preserving legitimacy without major concessions, and diverting attention from proposals of legally enforceable restrictions that could threaten lucrative operations. In short, the promotion of algorithmic fairness is part of a corporate strategy of self-regulation." (11)

Essentially, Ochigame suggests the notions of fair and ethical algorithms are branding tools forwarded by corporate interests to limit the scope of potential regulation. If technology companies self-regulate, then they argue that there is no need for government intervention. In the context of checklists, a technology company might advertise that a given algorithm passed a series of checks and consequently claim that the algorithm cannot be immoral or misaligned. This is certainly a possibility, and one to be wary of.

To guard against the co-opting of the checklist we create in part 3 by technology companies, we will make two design choices. First, the checklist will be targeted towards data scientists, rather than individuals higher up on the corporate ladder. In doing so, we hope to capitalize on the emerging distinction between engineers and the companies they work for. This phenomenon is bestillustratedby the #TechWontBuildIt movement, in which employees at large technology companies are protesting contracts they deem unethical. By designing our checklist for data scientists, we hope to encourage data scientists to reflect critically on their work and better recognize when their algorithms, and by extension the company they work for, do not align with their principles. Not only will this help data scientists identify immoral technology, but it will also help them recognize if the checklist becomes an argument against government regulation. Ultimately, we see the checklist as less of a tool for self-regulation, which implies companies regulating themselves, and more as a tool to help engineers regulate the companies they work for.

The second design choice we make is including a header on the checklist that states two warnings about the checklist. First, we want to make it clear that following this checklist in no way guarantees and moral, ethical, or fair algorithm. Second, this checklist does not preclude government regulation and, quoting the first recommendation fromAI Now's 2018 Annual Report, "Governments need to regulate AI by expanding the powers of sector-specific agencies to oversee, audit, and monitor these technologies by domain". Hopefully, by designing this tool for data scientists and including these caveats at the top of the checklist, it will be less likely that large tech companies co-opt this tool as an argument against government regulation.

Now that we have established that a properly designed checklist can help with complex problems, in part 3 we introduce the Design Justice principles and propose a prototype checklist.


A. Gawande, The Checklist Manifesto, New York: Metropolitan Books, 2010.
S. Glouberman and B. Zimmerman, "Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like?," Commission on the Future of Health Care in Canada, Canada, 2002.
K. Wardhana and F. C. Hadipriono, Study of Recent Building Failures in the United States, 2003.
M. Loukides, H. Mason and D. Patil, "Of oaths and checklists: Oaths have their value, but checklists will help put principles into practice.," O'Reilly, 17 July 2018. [Online]. Available:
"An ethics checklist for data scientists," Deon, 2018. [Online]. Available:
Unicorn Riot, ""Tech Won't Build It: The New Tech Resistance" Discussion Panel," Unicorn Riot, 11 July 2018. [Online]. Available:
Google Employees Against Dragonfly, "We are Google employees. Google must drop Dragonfly.," Medium, 27 November 2018. [Online]. Available:
AI Now, "AI Now Report 2018," AI Now, 2018.

Twitter, Facebook