It's not just terrorism; it's any rare risk in the news. The big fear in Canada right now, following a particularly gruesome incident, is random decapitations on intercity buses. In the US, fears of school shootings are much greater than the actual risks. In the UK, it's child predators. And people all over the world mistakenly fear flying more than driving. But the very definition of news is something that hardly ever happens. If an incident is in the news, we shouldn't worry about it. It's when something is so common that its no longer news - car crashes, domestic violence - that we should worry. But that's not the way people think.

Psychologically, this makes sense. We are a species of storytellers. We have good imaginations and we respond more emotionally to stories than to data. We also judge the probability of something by how easy it is to imagine, so stories that are in the news feel more probable - and ominous - than stories that are not. As a result, we overreact to the rare risks we hear stories about, and fear specific plots more than general threats.

The problem with building security around specific targets and tactics is that its only effective if we happen to guess the plot correctly. If we spend billions defending the Underground and terrorists bomb a school instead, we've wasted our money. If we focus on the World Cup and terrorists attack Wimbledon, we've wasted our money.

It's this fetish-like focus on tactics that results in the security follies at airports. We ban guns and knives, and terrorists use box-cutters. We take away box-cutters and corkscrews, so they put explosives in their shoes. We screen shoes, so they use liquids. We take away liquids, and they're going to do something else. Or they'll ignore airplanes entirely and attack a school, church, theatre, stadium, shopping mall, airport terminal outside the security area, or any of the other places where people pack together tightly.

These are stupid games, so let's stop playing. Some high-profile targets deserve special attention and some tactics are worse than others. Airplanes are particularly important targets because they are national symbols and because a small bomb can kill everyone aboard. Seats of government are also symbolic, and therefore attractive, targets. But targets and tactics are interchangeable.

The following three things are true about terrorism. One, the number of potential terrorist targets is infinite. Two, the odds of the terrorists going after any one target is zero. And three, the cost to the terrorist of switching targets is zero.

We need to defend against the broad threat of terrorism, not against specific movie plots. Security is most effective when it doesn't require us to guess. We need to focus resources on intelligence and investigation: identifying terrorists, cutting off their funding and stopping them regardless of what their plans are. We need to focus resources on emergency response: lessening the impact of a terrorist attack, regardless of what it is. And we need to face the geopolitical consequences of our foreign policy.

In 2006, UK police arrested the liquid bombers not through diligent airport security, but through intelligence and investigation. It didn't matter what the bombers' target was. It didn't matter what their tactic was. They would have been arrested regardless. That's smart security. Now we confiscate liquids at airports, just in case another group happens to attack the exact same target in exactly the same way. That's just illogical.

This essay originally appeared in The Guardian. Nothing I haven't already said elsewhere.

It's become a big deal in IT security, too. Many corporate customers are demanding ROI models to demonstrate that a particular security investment pays off. And in response, vendors are providing ROI models that demonstrate how their particular security solution provides the best return on investment.

It's a good idea in theory, but it's mostly bunk in practice.

Before I get into the details, there's one point I have to make. "ROI" as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It's an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn't make sense in this context.

But as anyone who has lived through a company's vicious end-of-year budget-slashing exercises knows, when you're trying to make your numbers, cutting costs is the same as increasing revenues. So while security can't produce ROI, loss prevention most certainly affects a company's bottom line.

And a company should implement only security countermeasures that affect its bottom line positively. It shouldn't spend more on a security problem than the problem is worth. Conversely, it shouldn't ignore problems that are costing it money when there are cheaper mitigation alternatives. A smart company needs to approach security as it would any other business decision: costs versus benefits.

The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money.

Of course, that $1,000 has to reduce the chance of being robbed to zero in order to be cost-effective. If a security measure cuts the chance of robbery by 40 percent -- to 6 percent a year -- then you should spend no more than $400 on it. If another security measure reduces it by 80 percent, it's worth $800. And if two security measures both reduce the chance of being robbed by 50 percent and one costs $300 and the other $700, the first one is worth it and the second isn't.

The Data Imperative

The key to making this work is good data; the term of art is "actuarial tail." If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one.

Cybersecurity is considerably harder, because there just isn't enough good data. There aren't good crime rates for cyberspace, and we have a lot less data about how individual security countermeasures -- or specific configurations of countermeasures -- mitigate those risks. We don't even have data on incident costs.

One problem is that the threat moves too quickly. The characteristics of the things we're trying to prevent change so quickly that we can't accumulate data fast enough. By the time we get some data, there's a new threat model for which we don't have enough data. So we can't create ALE models.

But there's another problem, and it's that the math quickly falls apart when it comes to rare and expensive events. Imagine you calculate the cost -- reputational costs, loss of customers, etc. -- of having your company's name in the newspaper after an embarrassing cybersecurity event to be $20 million. Also assume that the odds are 1 in 10,000 of that happening in any one year. ALE says you should spend no more than $2,000 mitigating that risk.

So far, so good. But maybe your CFO thinks an incident would cost only $10 million. You can't argue, since we're just estimating. But he just cut your security budget in half. A vendor trying to sell you a product finds a Web analysis claiming that the odds of this happening are actually 1 in 1,000. Accept this new number, and suddenly a product costing 10 times as much is still a good investment.

It gets worse when you deal with even more rare and expensive events. Imagine you're in charge of terrorism mitigation at a chlorine plant. What's the cost to your company, in money and reputation, of a large and very deadly explosion? $100 million? $1 billion? $10 billion? And the odds: 1 in a hundred thousand, 1 in a million, 1 in 10 million? Depending on how you answer those two questions -- and any answer is really just a guess -- you can justify spending anywhere from $10 to $100,000 annually to mitigate that risk.

Or take another example: airport security. Assume that all the new airport security measures increase the waiting time at airports by -- and I'm making this up -- 30 minutes per passenger. There were 760 million passenger boardings in the United States in 2007. This means that the extra waiting time at airports has cost us a collective 43,000 years of extra waiting time. Assume a 70-year life expectancy, and the increased waiting time has "killed" 620 people per year -- 930 if you calculate the numbers based on 16 hours of awake time per day. So the question is: If we did away with increased airport security, would the result be more people dead from terrorism or fewer?

Caveat Emptor

This kind of thing is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They've jiggered the numbers so that they do.

This doesn't mean that ALE is useless, but it does mean you should 1) mistrust any analyses that come from people with an agenda and 2) use any results as a general guideline only. So when you get an ROI model from your vendor, take its framework and plug in your own numbers. Don't even show the vendor your improvements; it won't consider any changes that make its product or service less cost-effective to be an "improvement." And use those results as a general guide, along with risk management and compliance analyses, when you're deciding what security products and services to buy.

This essay previously appeared in CSO Magazine.

Pouring over endless details of risks, regulations, taxonomies, and technologies can sometimes give us a narrow view of the world, so it seems worthwhile to take a minute to mark the 125th anniversary of the cataclysmic eruption of Krakatoa this week. For those of us that want to think big but can’t remember that far back, this week is also the 3rd anniversary of Hurricane Katrina’s devastating sweep across a wide stretch of the US Gulf Coast.

By now, I expect that most of you have read or are familiar with the 2007 book, The Black Swan, by Nassim Nicholas Taleb, which argues that these kinds of unpredictable, outlying occurrences are the ones that really shape businesses, countries, economies, and people. Taleb argues that although these “Black Swan” events are almost completely unforeseeable, we mistakenly try to explain the circumstances at the time and make predictions about similar events in the future.

In my ERM work with clients, and especially in the context of research I’ve been doing with my colleague Stephanie Balaouras on business continuity and resiliency, questions come up about how to plan for catastrophes... and they’re good questions. Were the CardSystems or TJX data breaches foreseeable? What about the Societe General debacle or the 2004 Indian Ocean tsunami? What’s next? Should these types of events be included in our risk assessments?

We’d like to get your opinion on these and other risks that may be on the very edge of the statistical tail. At what point do they belong in your risk register?

Of course, it’s possible to define mitigating controls for crises, disasters, or incidents without knowing for sure what they’re going to look like. That’s one of the hallmarks of a good crisis management plan. And that’s an important point, because trying to predict the next unforeseeable event can be a real challenge sometimes.

HANSEI - WHAT IS “RELENTLESS REFLECTION?” - And why we’re talking about it in the context of Risk Analysis.

Recall from yesterday’s post about how I got to thinking about the concept of Hansei-Kaizen, “relentless reflection” and “continuous improvement” and how we might apply that to risk management.  It’s a concept born of Toyota and is, in some way, the foundation for “Lean” production.

Call me biased, but I think that Hansei - the act of ‘relentless reflection’ made structured is the analytical function.  And I hate to debate (post-mortem) the father of Toyota quality success when he says that Hansei is the “check” in Plan/Do/Check/Act, but I think that Hansei also applies to the “Plan” of the P/D/C/A or Deming cycle.

You’ll recall the P/D/C/A cycle can be thought of even as an implementation of Scientific Method, in that it is Observation & Hypothesis Creation (P), Experiment (D), Analysis (Check), and Act (Revise/New Hypothesis, etc…).  Well then as such, the Hypothesis creation involves creating a model or creating an expected outcome for data using the currently accepted model.

So in our industry there is an opportunity for Relentless Reflection in both the Observation and Hypothesis (Plan) creation steps, and the Check step.  We create an estimate for control strength, or probable losses in the context of risk- then we go to Experiment step.  That hypothesis can be put it into production, have an audit, have a penetration test, whatever, in the context of the Do step.  BTW - using Hansei/Analytics in Plan is one way that strong analytical functions can really make penetration testing more useful - as a means to test the estimates and inputs into a model.  It’s Penetration Testing 2.0!  (<- tongue fully in cheek, yes)

Those who are versed in the reasons to merge Six Sigma and Lean together are probably already seeing where I’m going with this today.  But before you think that a simple DMAIC function is all that is needed to create proper “Hansei”, let me encourage you to keep reading.

Now if the analytical function can said to be “reflection”, why must it be relentless?

One word.  Change. There are essentially four separate “landscapes” or sources of change that we face (more on those tomorrow).  But anyone who has tried to manage system compliance, log management or policy exceptions knows that change is possibly the most difficult thing we security professionals must manage.  And when you think about it, there aren’t too many other business functions like information security where significant visibility and insight about the environment is needed for “complete” information (get bullish on Log Management is my recommendation).

HANSEI STEPS ADAPTED TO INFORMATION SECURITY

This is one of those quality control concepts that we can mangle adopt.  At Toyota, Hansei-Kaizen includes the following basic steps:

1. Initial problem perception
2. Clarify the problem
3. Locate area/point of cause
4. Investigate root cause (using an ask why 5 times approach)
5. Countermeasure
6. Evaluate
7. Standardize

Now it’s important to note that part of this includes the concept of Go See For Yourself, called “Gemba“.  Gemba can be translated as “the actual place” or “the place where virtue or truth is found.” At Toyota this might mean going to the shop floor to see the issue at hand in the production line.  But for us, that’s a problem because we live in the virtual world.  There’s usually not much use in hanging out in the wiring closets to try to see the problems.

But if you combine the concept of Gemba with the concept of “Nemawashi” –the process of discussing problems and potential solutions with all those affected- we can forge a similar concept using risk analysis.  That is discussing the issue and the risk associated with an issue (what some people would call “risk management”) with the business/LOB/data owner and let them accept authority and the risk decision.  We, the risk analyst, our goal is simply to perform items 1-5 (presenting countermeasure options that include transferring or accepting risk).  By going to the line of business and involving them, responsibility is shared.  Also, if you structure organizational behavior right, personal risk is transferred!

This sort of approach is also in harmony with concepts like “mutual ownership of problems,” or “genchi genbutsu,” (solving problems at the source instead of behind desks), and the “kaizen mind,” (an unending sense of crisis behind the company’s constant drive to improve).

One of the criticisms I have with the way most people try to implement DMAIC into “Lean”

REQUIREMENTS

Now to get this done, I really see three significant requirements.

1.)  A change in political structure.

2.)  Models that provide consistent, defensible analysis.

3.)  A Quantitative approach.  This means using actual units of measurement (not just amorphous percents, ordinal scales, etc.)  for risk and it’s subsequent factors.  Sure there are times when Q&D qualitative approaches are acceptable, but policy should be to have quantitative analysis whenever and wherever possible.

That last item - the quantitative approach - is really quite important.  And the reasons why will be discussed further in tomorrow’s post:

“What should we be reflecting about? & What is needed for reflection?”

P.S.  Your comments and suggestions, as always, are welcome.

P.P.S  Those who may be familiar with Lean/SixSigma/Kaizen sorts of mashups may be thinking - “hey, an Analytical step is built into SixSigma”.  Well, yes there is some prevision for analytical functions based on statistics, but I find SixSigma geared towards creating a State of Knowledge about operational processes, not towards creating a State of Wisdom for CISO’s around security & risks “big questions”.  In otherwords, the analytical function in DMAIC is in the context of Kaizen, and a different step than “reflective” analytics.