One of the main problems in counterterrorism today is that there are so many people and vehicles, and so much data and material, moving through globalization's myriad networks that it seems virtually impossible to track it all effectively. Nowhere has this problem been more acute than on the high seas.

In 2006, Adm. Harry Ulrich, then U.S. commander of NATO Naval Forces Europe, decided to do something about it. Despite having virtually no resources, his dream was to transpose the global air-traffic control system onto sea traffic.

Worldwide, aircraft are transparent, because they're all required to carry an identification beacon that allows them to be tracked leaving and entering airports, and monitored between airports, by a global network of sensors. Act suspiciously and somebody's fighter aircraft will soon be on your tail.

No such pervasive system currently exists globally for maritime traffic. While bigger ships carry an ID beacon similar to aircraft, without a shared monitoring network, that's like tracking only selected commercial jets and giving everyone else a pass.
So Ulrich, upon taking command, asked a simple question: "If we can do that in the air, why can't we do it on the sea?" He made a point of pioneering his sea-traffic-control effort first inside the Mediterranean, where NATO's southern naval forces have historically been concentrated, but his real target was waters off Africa -- the most ungoverned maritime space in the world.

Ulrich knew the U. S. Navy couldn't do it alone, much less bring Africa's meager coast-guard-like navies up to snuff so they could do it on their own. So he quickly created a network of assets -- both public and private -- to manage that space, modeling his monitoring system on international air-traffic control.

Ulrich began stitching together a network of shore-based sensors ringing the Mediterranean. His naval command then began initial monitoring by tapping into the International Maritime Organization's existing Automated Identification System, transforming NATO's ability to track ship traffic in the Med.

Almost overnight, NATO went from tracking dozens of ships on the Mediterranean to thousands, and instead of getting the data sometimes up to 72 hours late, now the contacts were being tracked in one to five minutes -- to an accuracy within 50 feet on the earth's surface.

When the classic big-firm systems integrators told Ulrich it would be too costly to pull it off, the admiral turned to the Volpe Center in Cambridge, Massachusetts, a U.S. Department of Transportation research center. Instead of hundreds of millions of dollars, Ulrich's initial network cost $900,000. The shore-based receivers are small, roughly the size of a radar dish you might find on a pleasure craft.

The strength of the system is a function of its reach: the more countries join, the larger the shared operational picture. By the time Ulrich retired at the end of 2007, he had enlisted 32 countries throughout the Mediterranean, the North Atlantic, along the west coast of Africa, around the Black Sea, and in the Pacific. Today, the network continues to spread around the planet.

With Ulrich's system in place, local police, coast guards, and border patrols catch most bad guys, obviating American military responses. As Harry told me for an article I wrote about his work in a fall 2007 issue of Esquire, "I don't do defense; I do security. When you talk defense, you talk containment and mutually assured destruction. When you talk security, you talk collaboration and networking. This is the future."

The admiral's legacy program, the Maritime Safety and Security Information System, earned the Volpe Center a prestigious "Innovations in American Government" award this month from Harvard University's Ash Institute for Democratic Governance and Innovation.

I’d love to have another blog even more specific - “Ok, that Assurance is Evidence Of What, Exactly?

Today he has a great article called:

What’s the matter with Risk Management?

And “in short, it’s everything.” It pretty much sums up why I had to grow to re-evaluate how our industry does risk, risk management, approaches controls & vulnerability and find a new way.   A couple of things jump out at me in reading Chris’ article:

1.)  Just because that Deming cycle sucks and is full of unknowns doesn’t mean “risk” doesn’t exist, nor that it isn’t of primary importance. Nor does it mean that in the absence of model & methodology, we won’t be “doing” risk analysis anyway - just in an ad hoc method and completely from “the gut”.

Our industry calls these unstructured risk analysis “Best Practices”, as it’s an easy and convenient way of sweeping the unknowns under the rug of bureaucracy and enforcing it via peer pressure.

2.)  What this “suckiness” does mean is that your model and methodology aren’t helping you. As Chris intimates, there is too much uncertainty in the inputs for his model (they are, in the language of Bayesians - too subjective to be useful priors).

Take for example how we might be approaching the “controls” part of our analysis.  Chris writes:

“2. What are the controls that we have to employ?
800-53, ISO 27001, PCI, etc.

Still kinda good, but we basically know that ISO is relatively voluntary and NIST supplies a control catalog and not policies. So here we have to take the control catalog, and mash our policies into it.”

I wouldn’t call this “kinda good” at all :)  These control catalogs only provide a hierarchy within which to look for evidence of  our ability to resist an attacker.  They are incapable of making any claim about the effectiveness of the controls when they are operated at 100% efficiency, or more importantly, what % efficiency our specific organization operates at.

Let’s use Chris Hayes’ Initech as our fictional example.

Initech has a control (a back door on a loading dock).  Now the locks on the door are 100% capable of locking the door.  This is different than saying that they are capable of frustrating all but the top 5% of lockpicking burgalars.  It is also diffferent than saying that in a sample of several “walk around audits” the doors are left open 20% of the time (they are not in compliance with policy 100% of the time).  Even worse, that 80% of the time the door is not propped open?  Yeah, tailgating is a known issue.

So we have several different variables here that we need to account for (and it’s just a door).  But the analogy stands that most “risk management” methodologies are “We have a door, yes/no?” And most GRC platforms, when asked for their “opinion” will simply say “door is needed” or, even worse, “a door policy is needed”.

3.)  Criticality and the Source of Value is all messed up in these Risk Management models.

Chris writes:

Someone wants me to tell them which boxes are more critical than others. This is mainly because of budgetary or operational reasons. To which I usually say “All of them, it is a system after all”.

This literally made me laugh out loud.  And this sort of “rate the firewall as Risk = 500 but rate the actual business application as Risk = 157″ thing is also endemic.  Now Chris is very smart here.  He correctly identifies that the value is tied to the business process the systems support, and not to a specific box.  Oh, we scan at the specific box level - but because of the nature of systemic failures - all the boxes in the process are inexorably interrelated.

One of the reasons I really like FAIR is that the losses are quantified (or qualified) based not on some amorphous value of the box or the process itself, but losses are linked to the actions that the threat will take. Take systems in a highly regulated industries as an example.  Usually the most probable losses aren’t due to system compromise per se, but in the disclosure the compromise causes (regulators are a threat source, after all).  But many “risk management” methodologies will say “online banking is worth $2 billion, the value of the systems is therefore $2 billion”.  And suddenly we’re telling executive management that there’s a 60% probability that they’ll lose $2 billion.

4.)  If the primary source of prior information for your “risk management” methodology is a vulnerability scanner - you’re doing it wrong.  Chris writes:

So we ran a scan and now we have a report. A snapshot in time to make all decisions. Where did these vulnerability ratings come from? Do I even know if my system is at risk? What if I spend my time on vulnerabilities that have no threat?

So first, my thoughts are that actual “vulnerability” must be a comparison of the force a threat can apply, and our ability to resist that force (this is a probability statement, btw).

Changing your thinking about vulnerability now helps us understand the problem in several new ways.  First, you can start to divorce yourself from the scanner.  After all, the scanner is simply providing you with current state information that is usually just relevant variance from policy. It doesn’t really tell you about real “weakness in a system” because the system is an interrelated mess of people, processes and IT assets.

5.)  Finally, most “risk management” approaches just *don’t* do a good job of helping us understand the how’s and why’s of managing risk. In the past, I’ve referred to these standards as really being “issue management” because they are at their heart, an act of discovery - a formal process around gathering prior information.  They are not, in and of themselves, capable of linking the issues discovered to the root cause.  And these root causes?  Yeah, they’re the things that create “risk”.  Not a threat, not a vulnerability, not the existence of an asset - the amount of risk that we have stems from our capability to manage it.

So Chris, I completely agree - but I wouldn’t give up yet.  There actually are a few of us who are focused on what you suggest:

Where to go from here: A fundamental revamp of how to deal with Risk. Where risk professionals focus on the treating the sickness and not the symptoms, and come up with some new success/actionable metrics.

Chris, there’s nothing I want to do more than that.

Now I’ve been “accused” of being a quant in the past (hi rybolov!) but in reality the only dogs I have in this fight are the model and the application of scientific method - and really, ethically speaking, I have to be tied to the latter while applying the former.

And I see a false dichotomy in this whole Quant vs. Qual thing.  We, as a profession, tend to create a political divide between the two which, if it even exists, I’d say is based more on our ignorance rather than our expertise.  After all, we are the profession that regularly multiplies across ordinal scales and uses wonderful models like R=VxTxI.   As someone  learning to deal in probabilities and rationalism, I have to recognize that this discussion is really just about the act of observation using different metrics of measurement.

But how we’re going about observing does not change the fact that there is measurement based on observation.  So if I’m working with you I can easily turn your qualitative scale into a quantitative one, and vice-versa.  Yes, Shrdlu, if we had the time, even your most seemingly Qual things could be Quant! (This flexible world view, btw, is an outcome of that new-fangled Bayesian thing).

COGNITIVE BIAS A-PLENTY

But back to what Rich is saying there about information security and risk - and he isn’t/won’t be the only one saying these sorts of things - we should try to understand what’s really going on rather than get caught up in the emotional hurricane.  Our profession suffers several forms of cognitive bias.  The nature of our jobs and what we do can cause us to be focused on the outcome and not the quality of the decision at the time it was made.  We want to bring in things from other professions that are useful, but at times we do view things outside our profession with false correlation to our own (unfortunately for those who write these sorts of articles, financial risk is completely different than operational risk).  We also have the tendency to focus on negative outcomes without acknowledging the positive outcomes (For example, I hear that Alan Greenspan’s new firm is up a couple of $billion in all this mess since he joined them, short sellers are doing quite well - must be because they have qualitative models or something -grin-).  The effect of these biases are compounded by the facts that proper correlation takes more work than we usually give it, and rational thought is not that easy when there’s a witch-hunt mentality.

What also floats in water? (link to Youtube)

WHAT SHOULD WE BE THINKING ABOUT?

So as you and I read opinions that seem to be the polar opposite of irrational exuberance (and there will be plenty between now and the election) we’ll have to ask ourselves, “what really failed here?”  At the risk (pun) of over-simplification:

• Was There an Error on the part of Probability Theory?

After all, Probability Science like all other fields of knowledge is always “advancing” as they say.  So perhaps probability theory is wrong somehow?

I’m personally disinclined to put the blame here, primarily because I would think that there would be evidence from other fields (like Quantum Mechanics) that something is amiss waaaaay before it hit a field like economics.

• Was There Error In The Model Used to Determine Risk?

Some people who understand real estate valuation and complex derivatives and financial risk want to put the blame here.  It’s a little too early to tell, but one thing is for sure - Financial risk is so different from operational risk I couldn’t begin to hazard an opinion on the subject.   But it would seem that this is really somewhere we might look.

• Was There Error In The  Scale Used (Quantitative vs. Qualitative)?

Honestly?  I find it extremely difficult to understand how this could be the source of financial ruin.

• Was There Error on the part of the Decision Maker?

What if all of the above were just fine, and the decision maker chose short term gain over long term stability?  What if this was (to simplify the matter greatly) a choice of “heads” over “tails” and the coin landed on tails?  What if the model represented the right risk (probability of negative outcome vs. positive outcome), but the complex derivative was sold to someone else who had poor “risk management” (ability to make a good decisions)?

Now I have no clue about complex derivatives, and I’m oversimplifying to be sure - chances are like most things, there are several problems that helped create the primary cause. But it seems to me that as we go into incident response mode for the economy, it’s more helpful to do so in a rational, logical manner.

OTHER THINGS WE MIGHT WANT TO CONSIDER

Consider the Source
Some authors (who I think tend to exploit outcome and hindsight bias,and then combine those with indirect ad hominem attacks in order to sell their books), are actually putting forth arguments against the use of analytics.  The source of this is a current epistemic debate between those who believe that only falsification is certain, and those who maintain that neither proof nor falsification are certain, there are only probabilities.    So before you go believing any “quadrants” of usefulness on faith - I encourage you to understand what is at the heart of the discussion.

We All Have to Live In The Real World
The sun will rise tomorrow, and someone will try to find the source of the problem and do a better job.  Now chances are, they’ll be doing it in a quantitative manner.  Chances are also that at some point their models will fail and we’ll need to build new ones.  And this will happen whether the field is cosmology, economics, meteorology, information security, or professional baseball.

WHAT ABOUT YOU, ALEX?

I’m far from certain and subject to change, but these days I lean towards Robin Hanson & MIchael Lewis w/regards to placing blame.

1. Our brilliant field engineer Dimitri McKay talks about the eternal topic of converting Windows event logs to syslog. Yes, Eric, we ALL know it is ugly, but that is the only way that actually works well across all systems ...
2. More on Windows and syslog: "Syslog ... 20 Years Later."  BTW, this is really not about syslog, but about Vista/2k8 finally getting an ability to natively centralize the event logs via event subscriptions ("It's only about twenty years behind schedule, if you're counting.")
3. Two fun pieces on correlation: 1 and 2. What often kills "a log correlation project"? "Whoever had worked on it had not had much time available to learn the way to properly configure the software" (from this)  and "correlation only really works when backed up by real data about what is the biggest problem in your environment, and how that problem manifests itself in the event logs." (from this) None of this is new, but a useful reminder nonetheless
4. Fun LogLogic podcast is here. The topic of this high-level discussion (CEO) is related to operational use for logs. I did one with them too; on logs and virtualization (will be up soon)
5. A couple of good posts on logging from Nemertes Research: "Sharpening Stones and Walking on Coals",  "Search or Destroy"
6. Reminder about a few useful Windows Vista and 2k8 events: 4802 (screensaver engaged) and 4803 (screensaver dismissed)
7. One person is wondering about the usefulness of logging after "experiencing" Linux auditd logging (kernel audit): "Logs are like a warm blanket; verbose logging means you can know what's happening on your systems if you keep up with the logs.  At the same time, logs become a burden very very easily, and they are easy to ignore." This post is a must read for us logging afficionados; producing too much log data is a sure way to make people hate you...
8. This also follows the same theme: people doubting the god-like power of logs :-) "So for an administrator to not care about logs was a shock." But would I argue that "log management is NOT a pain?" Now, would I? :-)
9. A classic about logging for application developers: "Building Secure Applications: Consistent Logging."  I am noticing a lot more discussions about logging in a developer community, e.g. see this and this (the latter, BTW, contains a lot of info on "why log" for developers). Overall, "getting logging right" is important (and will get more important in the future) and people need something NOW and cannot wait for the standards.  BTW, I am planning a mini-crusade on how to train application developers to include useful logging in their applications...
10. Finally, the "Is SIEM dead?" theme is continued in this fun post "Life after SIEM. Situational Awareness is next." Indeed, context is key for logs. BTW, if somebody mentions that I have "vendor bias", I will kick your ass! :-)

Enjoy!

Woodward: This is very sensitive and very top secret, but there are secret operational capabilities that have been developed by the military to locate, target, and kill leaders of al Qaeda in Iraq, insurgent leaders, renegade militia leaders, that is one of the true breakthroughs.

Pelley: What are we talking about here? Some kind of surveillance, some kind of targeted way of taking out just the people that you're looking for, the leadership of the enemy?

[...]

Woodward: It is the stuff of which military novels are written.

Pelley: Do you mean to say that this special capability is such an advance in military technique and technology that it reminds you of the advent of the tank and the airplane?

Woodward: Yeah.

It's here, 7 minutes and 55 seconds in.

Anyone have any ideas?