Nicholas Carr has an in-depth piece, The Great Forgetting, in this November’s Atlantic Monthly Journal. This article dives into the question of just what happens when we put ever-greater trust in automation that may be safer than we are, but not totally, completely, one hundred percent safe. The article starts with stories about airplane autopilots, and these are indeed excellent examples of what I call Adjustable Autonomy in Robot Futures. The interesting thesis is that increased reliance on automation has the unfortunate side effect of leading to disastrous human error when humans do, in fact, need to take over. Whether from lack of practice or a paucity of situation awareness, the human’s relationship to control changes when we are not really in charge, but just the understudy who might just be called up at a moment’s notice. At one point Carr notes that a solution posited includes irregularly forcing the human to do the work. Imagine the balanced self-driving car: every few days, it throws up its robo hands and announces: now, you drive!
A very useful trope that is an important contextual consideration is defined by Carr as the substitution myth. The basic idea is that we caricature automation as perfectly replacing some human activity with machine dexterity, when in fact the human is not simply freed by the automation. Our work doesn’t disappear- it is replaced by new work because the nature of our relationship to the newly “autonomous” system changes. In other words we live in a complex system. Automation doesn’t replace humans, it simply redefines our responsibilities and accountability in complex and hard-to-predict ways.
Of course, as Carr points out, one theoretical solution to the autopilot (read: automation in general) problem is, if people make errors when automation sometimes turns control over to people, just make the machine be in charge absolutely all the time. I become interested in just what happens to society when we hew towards this vision but, along the way, discover that 100% is a bit harder to reach than we first thought. What interesting dystopian situations face us then?