Trust in Computer Systems and the Cloud. Mike Bursell

or not performed), and the activity was not imbued with any further significance. If, however, we give or associate our example with some value—say, a signal that money should be transferred into my bank account from the flag-waver and their business partner's bank account—it is clear that a lot more is at stake. If, for example, my friend chooses to favour the business relationship over our friendship, my friend might collude with the flag-waver and tell me that they raised the red flag even when they did not. Or, my friend might tell me the flag-waver has not raised the flag when they have, colluding with a third party with the intention of defrauding both me and the third party by somehow accessing the funds in my bank account that I do not believe to have been transferred.

      The channels for reporting on actions—i.e., monitoring them—are vitally important within trust relationships. It is both easy and dangerous to fall into the trap of assuming they are neutral, with the only important one being between me and the acting party. In reality, the trust relationship that I have to a set of channels is key to maintaining the trust relationships that I have to the main actor that is the monitor—who or what we could call the primary trustee. In trust relationships involving computer systems, there are often multiple entities or components involved in actions, and these form a chain of trust where each link depends on the other: the chain is typically only as strong as the weakest of its links.

There is a further complication to be considered, which is how rarely we have control over—and can therefore have significant trust towards—the reporting or monitoring of channels in a relationship to a computer system. Channels are often under the control of the acting party or can be compromised in ways over which I have very little control. Let us look back again at the example of the bank transfer via a web browser. The reporting of the transfer is via the application, which is presumably written by the bank or at least is under the control of the bank: I have no control of this channel. Even if I am using a downloaded application and it is running on my computer or mobile phone—and even if it is open source,⁹ meaning I have the chance to check that what it is displaying is “correct”—I cannot be sure that it is “valid” as the information it has comes from the bank, in the final analysis. In terms of compromise, I can at least take some steps to decrease vulnerabilities and, therefore, the chance of compromise, for the pieces over which I have control: I can keep my computer or phone as secure as possible, check that the versions of any downloaded applications—web browser or banking application, for instance—are verified as being from the expected suppliers, and check that the connection they have to the bank is as secure as I can make it.

      In the end, however, it is up to my bank to provide valid information and ensure its correctness, though I will be the one who pays for these measures and am likely to bear any cost of invalid data: security economics raises its head again. This discussion about trust chains and monitoring will reappear later in the book as an important issue when designing and managing trust.

One final point to bear in mind about Gambetta's definition of trust is his description of the subjective probability with which an assessment is made about whether actions will be performed. We dropped this phrase in our definition, replacing it with assurance, as subjective probability frankly seems difficult to quantify or apply. The use of the word assurance should not be taken to imply 100% certainty, but words like subjective are problematic. The first reason is that the subject in this case may not be human. The second is because if we are to tie trust relationships back to discussions of risk, then objectivity and external qualifiability are insufficient: we also require external quantifiability.

      We have examined, then, a variety of different trust definitions in the human realm, though none of them seems a perfect fit for our needs. Before we throw all of these out with no further consideration, however, there is an interesting question about the overlap between human-to-human relationships and human-to-computer relationships when the computer has a closely coupled relationship with an organisation. This is different to case 3 that we discussed in Chapter 1, when we discussed the relationship between a bank and its systems, and more like case 2, where my trust relationship to the bank, and the bank's relationship to me, are characterised by interactions largely with their computer systems. In this case, punishment or other social impacts (positive or negative) may be more relevant, as we may be able to relate them to people rather than to the computers with which the actual interaction takes place. We will return to this question later, once we have addressed questions around trust to institutions—which is related but distinct—later in this chapter.

Game Theory

Much of the recent discussion around trust in the human-to-human realm has revolved around game theory and cooperation, either separately or in conjunction. Game theory is the study of social interactions from a theoretical point of view, typically looking at strategies for interaction that yield particular outcomes (usually the “best” for an actor or set of actors). Different styles of interaction, competition, and cooperation are studied, and mathematical approaches are applied, usually with the proviso that the actors are “rational”¹⁰ though not necessarily in possession of perfect information about their situation.

      The Prisoner's Dilemma

The best-known example of game theory is the Prisoner's Dilemma. The classic scenario of the Prisoner's Dilemma is where two members of a criminal gang are arrested and held separately from each other so that they cannot communicate. Each of them is presented with a choice: they may either stay silent or betray the other. Each is told the consequence of whichever action they may decide to take, combined with that of the fellow gang member. There are four outcomes, of which two are the same for both criminals, reducing the total number of distinct outcomes to three:

       Both prisoners stay silent, in which case they are both sentenced to one year in prison.

       One prisoner stays silent, but their colleague betrays them, in which case the betrayer goes free but the silent prisoner receives a sentence of three years in prison.

       Both prisoners betray the other, in which case they both end up in prison for two years.

      The rational position for each prisoner to take is to betray the other because betrayal provides a better reward than staying silent. Three interesting facts fall out of this game and the mountains of theoretical and experimental data associated with it:

       If the prisoners play repeatedly but know the number of repeated games, then the most rational strategy is to punish the other for bad behaviour and keep betraying.

       If they do not know the number of repetitions, then the most rational strategy is to stay silent.

       In reality, humans tend to a more cooperative strategy when playing variants of this game, working together rather than betraying each other.

I have attended events where groups of people—unaware of the theory—have participated in multiple rounds of this game or a modified version of the Prisoner's Dilemma (sometimes, for instance, the payoffs are adjusted and counts kept of notional money won and lost). It is fascinating to watch people trying out strategies whilst reacting to past rounds and also being locked into a history of their own behaviour that they cannot change. Much of the foundational modern work around the Prisoner's Dilemma—and broader game theory—was done by Robert Axelrod.¹¹ He noted the same points and posited that cooperation—in such games or more broadly—is a positive evolutionary trait. It encourages behaviours that are likely to benefit the survival of the species adopting them. He also suggested ways to encourage cooperation, based on computer models contributed by various academic institutions:

       Enlarge the shadow of the future (make players more aware of future games and less bound into their—and their fellow