1. The number of potential terrorist targets is essentially infinite. 2. The probability that any individual target will be attacked is essentially zero. 3. If one potential target happens to enjoy a degree of protection, the agile terrorist usually can readily move on to another one. 4. Most targets are "vulnerable" in that it is not very difficult to damage them, but invulnerable in that they can be rebuilt in fairly short order and at tolerable expense. 5. It is essentially impossible to make a very wide variety of potential terrorist targets invulnerable except by completely closing them down.

1. Any protective policy should be compared to a "null case": do nothing, and use the money saved to rebuild and to compensate any victims. 2. Abandon any effort to imagine a terrorist target list. 3. Consider negative effects of protection measures: not only direct cost, but inconvenience, enhancement of fear, negative economic impacts, reduction of liberties. 4. Consider the opportunity costs, the tradeoffs, of protection measures.

This paper attempts to set out some general parameters for coming to grips with a central homeland security concern: the effort to make potential targets invulnerable, or at least notably less vulnerable, to terrorist attack. It argues that protection makes sense only when protection is feasible for an entire class of potential targets and when the destruction of something in that target set would have quite large physical, economic, psychological, and/or political consequences. There are a very large number of potential targets where protection is essentially a waste of resources and a much more limited one where it may be effective.

Apologies in advance, for the length of this post…

In a perfect world…
… we’d know which specific threat agent was going to act against us and know the capability of that threat agent in absolute terms (e.g., pounds per square inch), as well as know (through testing) what our resistance capabilities are in those same absolute terms. If we had this information AND assuming this information was precisely correct all of the time, vulnerability becomes a clear and simple binary consideration — we will be or we won’t be.

Stating the obvious (anyway)
Losses occur when threat events take place that we’re vulnerable to. This is true whether we’re talking about weather events, human error, or malicious acts. Obviously, we don’t experience loss with every threat event, which means we’re only vulnerable sometimes — i.e., less than 100% of the time. This means there is some probability associated with whether we’ll be vulnerable to any given threat event. The process of measuring vulnerability is intended to help us understand what that probability is likely to be.

Simplest approach
Perhaps the simplest approach is to identify the threat community you’re analyzing risk against and simply estimate your ability to resist the capabilities of that threat community. For example, we might estimate that our web application is capable of resisting all but the top 2% of the cyber-criminal threat community — i.e., two out of a hundred hackers have the skill and resources to defeat the application’s security.

This works as a quick-and-dirty solution, and in many cases is good enough. Read on if you’re interested in a somewhat more involved approach.

Uncertainty
Unfortunately, in the real world we usually don’t know:

• Which threat agent is going to act next,
• What their capabilities are, or
• What our resistance capability is going to be

Making matters even more challenging:

• We don’t have an absolute measurement scale for some threat categories (e.g., human capability)
• Our measurements are imprecise (e.g., we can’t measure force or resistance perfectly)
• One or more of the values being measured may vary over time (e.g., hurricane wind speed varies throughout the lifetime of the storm, and strength can change throughout the lifetime of a control )
• One or more of the values being measured may vary across a population (e.g., not all hurricanes have the same wind speed)

When absolute scales apply
(Warning: This is an illustration and not an engineering exercise, for those who might want to argue details.)

Some types of threat categories can be measured using absolute scales (e.g., wind speed in miles per hour), which makes things a bit more straightforward. For example, thru testing we could estimate that a structure should be capable of resisting wind forces between 150 and 200 MPH.

By using a distribution to describe this measurement, we account for the fact that under some circumstances wind speeds of less than 150 MPH might compromise the structure, while in some circumstances the structure may be able to withstand speeds greater than 200 MPH.

If we wanted to measure the structure’s vulnerability to a specific type of storm (e.g., a tornado) we could plot a similar distribution for tornado wind speeds (black curve below). This distribution reflects the fact that wind speeds vary from tornado to tornado, ranging from under 100 MPH to over 300 MPH, with most falling in the 200 MPH range. (Keep in mind this is just an illustration and isn’t intended to reflect actual tornado data.)

In order to determine the probability of being vulnerable, we’d use a Monte Carlo function to:

1. Take a random value from the tornado distribution and from the structural resistance distribution
2. Compare the values — i.e., for this iteration, determine whether wind speed was greater than resistance
3. If wind speed was greater, increment a counter that tracks the number of vulnerable instances
4. Repeat a thousand iterations (or ten thousand, a million, etc.),
5. After completing all of the iterations, the vulnerability counter divided by the number of iterations provides the probability of this structure being vulnerable to tornado winds

When an absolute scale doesn’t exist (the human threat community)
Human threat capability can be boiled down to skills and resources. Because skills and resources vary from individual to individual, we can characterize threat community capability as a distribution. At one end of the distribution are those threat agents who have the least capability, while at the other end are those who are the most capable. As seems to be the case for most things in nature (e.g., weather events), the distribution is probably pretty close to being bell-shaped (i.e., the majority of threat agents fall somewhere below those who are most capable and above those who are least capable).

A “100% secure” control (if such a thing existed) could be illustrated as existing outside of the threat community capability distribution. It would be 0% vulnerable.

More realistically, we can in most cases expect that some portion of the threat population would have the skill and resources to compromise a control (shown below).

Now, because of the uncertainties regarding threat capabilities and control strength, it would be more accurate to describe control strength as a distribution as well. For example, we expect the control is at least resistant to 90% of the general threat population, and may be resistant to as much as 99%+ of the population.

This is fine as far as it goes, but it doesn’t get us the answer we’re looking for in most circumstances. Most of the time it isn’t enough to know our vulnerability to the general threat population. In most analyses, we want to know what our vulnerability is to a particular threat community (e.g., cyber criminals, nation-state intel units, etc.). In that case, we’d have to plot the capability of the threat community in question (red distribution).

With that plotted, we can run our Monte Carlo function again, generating a probable vulnerability by taking random samples from the control distribution and the distribution of the specific threat community in question.

The key to measuring vulnerability in the absence of an absolute scale is to use the general population capability as the comparative baseline for both control strength and the capability of the threat community in question.

Considerations
Of course, because some malicious threat communities tend to share knowledge and tools, there can be an equalizing effect, which potentially narrows the width of the threat capability curve (shown below) but likely wouldn’t change its fundamental bell-shape. The good news is that this narrowing effect wouldn’t alter how we measure. The bad news is that it does affect vulnerability, which we know intuitively anyway.

Another consideration is the fact that the capability of the malicious population evolves over time — i.e., the curve shifts to the right along the continuum. For example, at one time in the past DES was considered invulnerable to brute force cracking. It isn’t any longer. In other words, we could say that the control stayed in place along the continuum, but the capability curve shifted to the right. This highlights the fact that it’s important to maintain a bead on how threat capability evolves, so that you can evolve your defenses as well. Also, this is good fodder for the importance of defense-in-depth.

Concerns
An obvious concern is the inexact nature of these estimates and the potential for the analyst to estimate badly for various reasons. We’ve covered this issue previously in other postings, so I won’t go into it in depth now. Suffice it to say that yes, this is an imprecise measurement fraught with all of the goblins that any measurement approach is subject to. That said, keep in mind a few things:

• The ability to estimate effectively can be significantly improved using calibration techniques
• There’s no such thing as a perfectly precise measurement, whether you’re using a laser or the width of your thumb to do the measuring. Therefore, the purpose of measurement is to reduce uncertainty, not eliminate it
• You can apply confidence levels to your estimates, both to describe the probability of actual values being outside of the estimated minimum and maximum, and to shape the peakedness/flatness of the curve
• Monte Carlo analysis is designed to help account for the uncertainty in measures
• You should never convey to management that these numbers are precise. In my experience management won’t have any problem with this, as the numbers they’re given from other business disciplines have precision challenges of their own.

Bottom line — If you’re trying to quantify risk, then you have to quantify vulnerability. This is one logical means of doing so. What’s more, it seems to accurately reflect how we subconsciously evaluate and quantify vulnerability anyway, only it brings the analysis to the surface. And by bringing it to the surface, it allows us to better understand and analyze risk scenarios.

If there’s interest, I can provide a couple of examples in a future post. Also, if there’s interest, I can include an example where the threat event is due to error rather than malicious intent.

Our submission looks at three cases in particular in which the ombudsman decided in favour of the banks and against bank customers over disputed ATM transactions. We found that the adjidicators employed by the ombudsman made numerous errors both of law and of technology, and concluded that their decisions were an affront to reason and to justice.

One of the cases has already appeared here on lightbluetouchpaper; the other two cardholders appeared on an investigation into card fraud on “Tonight with Trevor MacDonald”, and their case papers are included, with their permission, as appendices to our submission. These papers are damning, but the Hunt review’s staff declined to publish them on the somewhat surprising grounds that the information in them might be used to commit identity theft against the customers in question. Eventually they published our submission minuss the two appendices of case papers. (If knowing someone’s residential address and the account number to a now-defunct bank account is enough for a criminal to steal money from you, then the regulatory failures afflicting the British banking system are even deeper than I thought.)

The Financial Ombudsman Service, and its predecessor the Banking Ombudsman, have for many years found against bank customers and in favour of the banks. In the early-to-mid 1990s, they upheld the banks’ outrageous claim that mag-stripe ATM cards were invulnerable to cloning; this led to the court cases described here and here. That position collapsed when ATM criminals started being sent to prison. Now we have another wave of ATM card cloning, which we’ve discussed several times: we’ve shown you a chip and PIN terminal playing Tetris and described relay attacks. There’s much more to come.

The radio program is not yet available online; I’ll put in a link here when it appears. We clearly have them rattled; the ombudsman was patronising and abusive, and made a number of misleading statements. He also said that the “independent” Hunt review was commissioned by his board of directors. I hope it turns out to be a bit more independent than that. If it doesn’t, then consumer advocates should campaign for the FOS to be abolished and for customers to be empowered to take disputes to the courts, as we argue in section 31-32 of our submission.