Table of Contents
|
Can Computers Decide what is Right and Wrong?
It is dangerous to use computers to ensure adherence to rules, laws and regulations. When rules are interpreted by humans, the humans are capable of interpreting the rules with discretion and understand that quite often, the rule should not be adhered to 100 % in every case. Computers are not so good at making such judgements. Because of this, it can be unsuitable to put to much of rule-checking into software. |
Introducing the Issue by an ExampleExample: Suppose you equip a motorcar with a computer-controlled device (a breathanalyzer, in Swedish alkholås), which makes it impossible for you to drive when your blood alcohol level is above a certain level. And suppose there is an exceptional case. John has a heart attack, and the only person available to drive to the hospital, Mary, has drunk a little too much. The computer stops Mary from driving the car, and John dies. They do not have a telephone in their remotely located summer cottage, and their mobile phone is out of reach for the network it is using. My argument: This is an example of a computer making a decision. The computer decides that Mary is not allowed to drive the car, and in this particular case, the decision made by the computer might be ethically wrong. This is no easy issue, the best may still be to have such a computer device in the car. But the example illustrates the problem and the danger of programming computers to make decisions about right and wrong. And some (not all) such breathanalyzer actually has an override button, even if the user of course has to report and explain why s/he used the override button, or lose their drivers's license. Counter-argument: No, the computer did not make any decision. The decision was made by the humans who programmed the computer. They may have weighed pros and cons, and decided that the advantage with such a drunk-driving-protection device is worth the risk that in some exceptional cases the outcome may be wrong. My counter-counter-argument: This is becoming a discussion of the meaning of words. You do not accept that the computer made a decision. OK, let us then say that the computer made a ruling, or whatever word you prefer to apply to the case where a computer prohibits you from doing something. You are avoiding the ethical issues: In what way should we program computers to control human beings. I am not a fanatical liberal who is against all laws and rules. I am quite willing to accept that in some cases it may be ethically right to program computers to prohibit you from drunken driving or stop children from downloading bomb-making recipes from the Internet. But I am advocating that in those cases you are programming the computer into making decisions, or rulings or whatever word you prefer to use. And this can be dangerous and you should be aware of the risks. Solution: In this special case, a solution might be to allow the driver of the car to communicate with SOS Alarm, and allow them to send a code to override the breathanalyzer lock in some very exceptional cases. By doing this, we are moving the final decision from computers to humans. |
Explanation of ProblemOne mode of human communication is the setting of rules. Some human beings make a list of rules. The rules may be a law, a local ordinance, ethical rules of a professional organizations, company rules for employees or published in other ways. The human beings may also introduce ways of enforcing the rules, such as courts of law, committees on ethical conduct, etc. This is sometimes (not always) necessary even though the rulings made are sometimes wrong, like convicting innocent people. But we accept that this risk must be taken because without law and order society would not work. An example: Even if it is forbidden for a pedestrian to step into the street against a red light, there are special cases where this rule does not apply. Suppose a child runs out on the street, and the only way to stop the child from getting run over by a car is to run out and catch the child. Such special cases are easily handled by humans. No human court would sentence a person for running against a red traffic light in such a case. But it is not so easy to teach a computer to understand such exceptions. The danger is that people do not always understand, that putting such rules or laws into a computer, and programming the computer to enforce the rules, is something very different from having humans implement the rules by human decisions [Grip 1975, Palme 1975]. Humans can understand the special conditions of special circumstances. A human might decide, in the example above, that in this special case the importance of getting John to a hospital is higher than the risk of Mary driving while intoxicated. Counter-argument: A counter-argument made at the ACM meeting was that in this case it was humans who made the decisions, by programming the car computer, but their decisions were wrong. They did not take all circumstances into account. They should have made a more advanced program, which could take into account the special circumstance of the heart attack situation. Counter-counter-argument: What you are doing, with this kind of argument, is to make the computer program more advanced and complicated, to reduce the risk that the decisions/rulings made by the computer are wrong. The path you are treading may make things worse instead of better. More complex and advanced computer rulings may increase the risk of wrong decisions instead of reducing them. The right solution may sometimes instead be to accept that the computer is not perfect, and thus that all rules do not have to be enforced by computers alone. Italian Strike ExampleA well-known method for strikes is to continue to work, but to adhere 100 % to all rules while working. This rapidly causes many businesses to a complete standstill, or at least makes them work much slower and less efficiently before. The reason for this is that even good and benevolent rules can have disastrous effects if adhered to 100 %, the way a computer would do if programmed to enforce them. Filling in a Form ExampleIf you fill in a form manually, you have the option of adding an accompanying sheet of paper explaining why you filled in the form in a particular way. In most forms, where there is a "yes-no" question, it is quite possible to omit the checkboxes and write a more nuanced explanation below or above. The human who receives the form will read and understand and interpret this. When the form becomes computerized, this freedom to not follow the prescribed way of filling in the form often disappears. Society Evolves by Many People Doing Things in Better WaysAn important way in which human society is evolving is that many people make small and large decisions to try out new and better ways of doing things. If the computer program stops them from doing things in other ways that those specifically allowed, this will prevent people from finding better ways of doing things, and thus stop imrovements [Hoare 1975, Palme 1997]. Sometimes this may be necessary. For example, there is a human tendency to stop performing actions which are necessary only to avoid seldom occurring risks. Example: A pilot forgets an item on the pre-flight check list, or a night watchman forgets to go to a normally empty part of the building. In such cases, it may be necessary to use technical means to ensure that the human follows the rules, for example the night watchman must turn a key to show that he has passed that part of the building. But this does not forbid the night watchman from disobeying the rules in special cases, for example skip the empty corridor if there is a thief in another part of the building. The danger is when the computer does not allow you to do things in other ways than those foreseen when programming it. People are in fact very clever in circumwenting restrictions in order to do what has to be done. See for example the picture below:
Two-sided Communication is Better Than Enforcing One SolutionIt is a well-known fact that power is addictive. That is why we design human societies with so many safe-guards against giving individual people too much power. We must understand that the power to control other humans by design of computer software can also be addictive [Hoare 1975]. Only by understanding this, can we stop people from putting too much control of humans into their software. I am not arguing that there should be no control of humans by software. The common "Are you sure?" dialog boxes are often motivated, even though they are sometimes a nuisance. But those who design computer software should be aware of the risks of putting too much control of humans into the software. This will reduce the possibillities for people to influence their environment, and will create a feeling of helpnessness, which may cause dissatisfaction and depression [Seligman 1975]. When two humans communicate regarding a task, the outcome of their discussion is usually a twosided or so-called win-win solution, a solution where the needs of both are taken into account [Harris 1969, Gordon 1970]. When the task is controlled by a computer (even though a human did originally program the computer) the interaction necessary to achieve a win-win solution is usually not possible. It is well-known that such situations easily cause frustration and dissatisfaction and also often mean that a less good way of performing the task has to be used. User influence |
Human beings hade a need to be able to influence their life. They will be more happy and satisified, and will be able to do a better job, if they can influence their life, and use their abilities to perform their tasks better and better. | |||||||||
Conventional solution: Let the users influence the development of the software system they are going to use. | Alternative solution: Design the software so that the users can, themselves, modify it according to their present and future needs. | ||||||||
Problems with the conventional solution:
|
Problem with the alternative solution:
|
||||||||
Corrective action: Users require new features of the software, developers are overloaded with work to adjust the software, there is a huge backlog of tasks, the software gets more and more complex through many haphazard extensions. | Corrective action: Educate special so-called "local experts", who work locally in the local user groups, and help users with extension of the software to their needs. |
Grip 1974: |
ADB-system och kommunikation (Data processing and communication). Hermods-studentlitteratur, Lund, Sweden, 1974. |
Gordon 1970: |
P. E. T. - Parent Effectiveness Training, by Thomas Gordon, 1970. |
Harris 1969: |
I'm OK - You're OK, by Thomas A. Harris, 1969. |
Hoare 1975: |
Software Design: a Parable . In Software World, vol. 5, No. 9 &10, 1975. |
Martin 1997: |
Empowering Educators and Parents: Content Advisories for the Internet. By C. Dianne Martin, Proceeedings of the ITiCSE ACM conference, June 1997. |
Palme 1975: |
Interactive Software for Humans , by Jacob Palme. An abbreviated version was published in In Management Informatics vol. 7(1976) pp 4-16. |
Palme, 1997: |
User influence on software design may give less good software . |
Palme, 2000: | A
personal history of CMC, by Jacob Palme. Honorary, in Creative Crossroads Electronic Honorary Publication, Dedicated to Yvonne Wærn on Her Retirement. |
Palme, 2010: | CAN COMPUTERS DECIDE WHAT IS LEGAL AND ILLEGAL?, by Jacob Palme. Information and Communicaton Technologies, Society and Human Beings, IGI Global, Hershey PA. USA, November 2010. |
Seligman 1975: |
Helplessness: On depression, development and death, by Martin E. P. Seligman, W.H. Freeman, San Francisco1975. |
Vetandets värld 2006: | (In Swedish, audio) Den mänskliga faktorn som förklaring till olyckshändelser.in Vetandets värld, Sveriges Radio, 27 Feb 2006 |