Sounds like the beginning of a joke, right?  So these two guys walk into a bar…

“The” Bruce Schneier and Marcus Ranum have an article up on TechTarget/Information Security Magazine called, creatively enough, “Bruce Schenier, Marcus Ranum debate risk management“.

Unfortunately, to get to the article, you’ll have to either already be a subscriber to IT Security, a subscriber to TechTarget, or go through the 20 minute process of signing up by giving TechTarget all sorts of “market information” about how you’re really Brandon Walsh, CSO of “The Peach Pit” Industries in Beverly Hills, CA 90210 (phone 714-867-5309).

For those of you who are already a TechTarget person, the link is above.  For those who aren’t, or those who just don’t have the time, I’ll summarize.  The “debate” is kind of awkward because both authors seem come to the same conclusion:

Risk Management, it’s something our profession should do, something humans do naturally, it’s necessary in business, but gosh - we don’t have enough data.

I’m not a cryptographer.  I don’t *nearly* have the insight on privacy and politics that Bruce has.  I’m not deep in IP communications.  I haven’t got a proven track record of innovation in IP Security products like Marcus has.  But here’s the thing, I hope you’ll never hear me pretend that I have the skill set to speak authoritatively on those subjects.  Heck, I wouldn’t claim to be a “risk” expert because I have a some insight into my shortcomings and what is needed to tackle such a complex problem.  But such a tepid article on something that (at least I think) is so important kind of, well, confuses me.

Why is it such a boring article?  I’m not sure.  Maybe because they’re just two guys who would rather debate the merits of specific controls or control activities (after all, their penetration testing debate was a huge success), but there’s no new information in the “debate”.  It’s the same old “insurance companies know risk because they have scads of data and we don’t have that” complaint. You know what?  I’m tired of hearing that line, so let’s talk about it.

HOW DO YOU KNOW WE DON’T HAVE THE AMOUNT OF DATA WE NEED TO DO RISK MANAGEMENT WELL?

Not particularly picking on Marcus, but in the article he uses the common complaint, “We lack the data to do risk management well.”  This mantra is repeated to the point where I’m blase’ about it.  But for some reason, this sentence really jumped out at me this time for two reasons.  It made me ask:

1.)  How do you know we don’t have the proper amount of data?

2.)  Can we even define “well” (i.e. what “good” risk management is) yet?

I really don’t know that the industry, especially concerning IT risk, is mature enough to really conclude that we don’t know (in the case of the former), nor that we can define (latter), conclusively.

PLAYING THE CONTRARIAN

Just because I’m feeling kind of zany this morning, let me suggest something.  Maybe there actually is lots of evidence out there for us to use.  Maybe:

1.)  It’s just that we don’t have particularly good models that provide context.

2.)  When that evidence isn’t an obvious phenomena that lends itself to easy measurement, we throw our hands up in disgust and fall back on “lack of data”, “can’t quantify risk”, “best practices work just fine” or any other number of arguments, no, excuses we use to justify our inability to be precise about the past (more or less the present or future - apologies to Niels Bohr).

IT’S IN THE WAY THAT YOU USE IT

Now I actually am happy to acknowledge that we don’t have enough data to be precise.  You, me, even smart guys like Marcus and Bruce - we’ll never be able to “engineer” risk management.  But you know what?  Neither can Insurance companies.  Sure, there are plenty of places where they have enough data to apply a traditional frequentist approach to risk valuations.   But there are plenty of times Insurers actually insure and they don’t have centuries or decades of data.  There are plenty of times when they rely on the “estimates” of subject matter experts.  There are many times they have enough information to be accurate rather than precise, and that’s good enough for them.

For that matter, it’s worth noting that there are plenty of scientific disciplines that have to deal in imprecise prior information, or evidence that’s fraught with uncertainty (what Ranum calls “squishy”, and what I’ve heard real honest to goodness physicists call “noisy”).  Unfortunately, we’re going to be like them.  Until we can read minds and predict the future, there will always be uncertainty in our measurements and posterior conclusions.  The trick is in how you deal with it and express it.  And while I really don’t know how much time Marcus or Bruce have really spent in the deep end on the subject of risk and its management - I have seen people doing brilliant things around risk (though they just aren’t mainstream).  Whether the tools are Bayesian methods, Monte Carlo engines, reductionist models of complex problems, there are risk analysts trying to deal with the problem.  These analysts are applying scientific method(s) and developing reasonable approaches to a very complex problem.  There are people trying, and our body of knowledge is growing, growing well beyond “gee, I haven’t got an obvious solution so I’ll blame it on lack of data”.  Heck, I’ve seen readers of this blog suggest Douglas Hubbard’s book in other security forums!*

I’VE GOT YOUR DATA RIGHT HERE…

But we don’t have enough data?  I have to ask, how much more do we need?  I mean crikey, JPMC just visited our ISSA chapter claiming, like, a bajillion events an hour.  There’s not one, but several companies out there that will want to tell you about how they have deep “insight” into the attacker community.  The boundaries of IT Risk losses are pretty well established by events that happen to public companies.  We have pretty mature testing/assessment tools and methodologies now that help us test our ability to resist the force an attacker can apply to us.  So what part of the Threat Landscape, Asset (Controls) Landscape, or Loss Magnitude landscape is too incomplete (and what are you doing to find the information you need)?

SO WHY DO WE FAIL?

Which brings me to a final, somewhat depressing conclusion.  Maybe there’s data, and maybe we’re starting to see the means to use it.  But in the end I do have to agree with Marcus that the vast majority of the infosec world *is* doing a really, really bad job with regards to “risk” and “risk management”.  The majority of people I know consider GRC to be a cruel, expensive joke.  Risk Assessment Methodologies tend to be built on the faulty premise that if we create a repeatable process, our measurements and conclusions will magically become accurate and wise.  Risk models tend to be factors loosely measured by ordinal scales and then somehow “multiplied” together to create a relatively meaningless qualitative value.  The State of the Union here is not good.  But after reading such a superficial treatment of an important and complex subject, I am left wondering if Bruce and Marcus were the right people to write about risk management in a mainstream publication.  As Inspector Callahan says, “A man’s got to know his limitations.”

===============================

* Speaking of which, if you want to do one cost effective thing to address your uncertainty - go find Douglas Hubbard’s book. It’s even got a nice recommendation from Peter Tippett.  The book is called “How To Measure Anything” - the title sounds rather hyperbolic, but there are good techniques in it we can use to identify useful information and refine our ability to frame that qualitative information into quantitative values. The key is how Hubbard has you deal with your uncertainty.  For those of you who are more scientific minded and want to dig deep into the subject, I have on good authority that E.T. Jaynes “Probability Theory, The Logic of Science” is a rather under appreciated work.

Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

Interop NY 2008 was up considerably in size from the show in 2007.  According to Lenny Heymann, the GM of Interop, this is a trend that they expect to continue.  My personal experience was that the size of the vendors was also up this year.  I think there were so few startups that “Startup City” was pulled from the show completely.  In any case, the show floor was full and there was plenty of attendee traffic to go around.

Definitely helping out from a traffic and draw perspective was the addition of the Web 2.0 Expo - Interop was co-located with both Mobile Business Expo and the Web 2.0 show. It seems like that buzzword still hasn’t lost most of its luster.

From the InteropNet perspective, the main feeling was one of being rushed.  With the show only lasting two days, and the InteropNet team only having a couple of days of ramp up time, everything was compressed into a much shorter period than in Las Vegas.  While this would normally be a challenge, it’s an even bigger challenge at the Javits where the InteropNet team was allowed to do almost nothing ourselves because of union rules.  You’d be surprised how frustrated you can make a network guy who’s told that he has to stand there and watch the electrician plug things in, rather than just doing it himself.  The only thing faster than the InteropNet team getting the Interop NY network up, was my pedicab ride to the InteropNet Booze Cruise.

In any case, everything came off without a hitch, and EM7 performed flawlessly catching a couple of power outages that last day and alerting everyone before the batteries on the UPSes had a chance to run down.

Over the next couple of weeks I’ll analyze the data from the show to see how many tickets were handled, amount of bandwidth consumed, etc and we’ll do a comparison to Interop Las Vegas.

We’re (both ScienceLogic and me personally) looking forward to Interop 2009.

Now I’ve been “accused” of being a quant in the past (hi rybolov!) but in reality the only dogs I have in this fight are the model and the application of scientific method - and really, ethically speaking, I have to be tied to the latter while applying the former.

And I see a false dichotomy in this whole Quant vs. Qual thing.  We, as a profession, tend to create a political divide between the two which, if it even exists, I’d say is based more on our ignorance rather than our expertise.  After all, we are the profession that regularly multiplies across ordinal scales and uses wonderful models like R=VxTxI.   As someone  learning to deal in probabilities and rationalism, I have to recognize that this discussion is really just about the act of observation using different metrics of measurement.

But how we’re going about observing does not change the fact that there is measurement based on observation.  So if I’m working with you I can easily turn your qualitative scale into a quantitative one, and vice-versa.  Yes, Shrdlu, if we had the time, even your most seemingly Qual things could be Quant! (This flexible world view, btw, is an outcome of that new-fangled Bayesian thing).

COGNITIVE BIAS A-PLENTY

But back to what Rich is saying there about information security and risk - and he isn’t/won’t be the only one saying these sorts of things - we should try to understand what’s really going on rather than get caught up in the emotional hurricane.  Our profession suffers several forms of cognitive bias.  The nature of our jobs and what we do can cause us to be focused on the outcome and not the quality of the decision at the time it was made.  We want to bring in things from other professions that are useful, but at times we do view things outside our profession with false correlation to our own (unfortunately for those who write these sorts of articles, financial risk is completely different than operational risk).  We also have the tendency to focus on negative outcomes without acknowledging the positive outcomes (For example, I hear that Alan Greenspan’s new firm is up a couple of $billion in all this mess since he joined them, short sellers are doing quite well - must be because they have qualitative models or something -grin-).  The effect of these biases are compounded by the facts that proper correlation takes more work than we usually give it, and rational thought is not that easy when there’s a witch-hunt mentality.

What also floats in water? (link to Youtube)

WHAT SHOULD WE BE THINKING ABOUT?

So as you and I read opinions that seem to be the polar opposite of irrational exuberance (and there will be plenty between now and the election) we’ll have to ask ourselves, “what really failed here?”  At the risk (pun) of over-simplification:

• Was There an Error on the part of Probability Theory?

After all, Probability Science like all other fields of knowledge is always “advancing” as they say.  So perhaps probability theory is wrong somehow?

I’m personally disinclined to put the blame here, primarily because I would think that there would be evidence from other fields (like Quantum Mechanics) that something is amiss waaaaay before it hit a field like economics.

• Was There Error In The Model Used to Determine Risk?

Some people who understand real estate valuation and complex derivatives and financial risk want to put the blame here.  It’s a little too early to tell, but one thing is for sure - Financial risk is so different from operational risk I couldn’t begin to hazard an opinion on the subject.   But it would seem that this is really somewhere we might look.

• Was There Error In The  Scale Used (Quantitative vs. Qualitative)?

Honestly?  I find it extremely difficult to understand how this could be the source of financial ruin.

• Was There Error on the part of the Decision Maker?

What if all of the above were just fine, and the decision maker chose short term gain over long term stability?  What if this was (to simplify the matter greatly) a choice of “heads” over “tails” and the coin landed on tails?  What if the model represented the right risk (probability of negative outcome vs. positive outcome), but the complex derivative was sold to someone else who had poor “risk management” (ability to make a good decisions)?

Now I have no clue about complex derivatives, and I’m oversimplifying to be sure - chances are like most things, there are several problems that helped create the primary cause. But it seems to me that as we go into incident response mode for the economy, it’s more helpful to do so in a rational, logical manner.

OTHER THINGS WE MIGHT WANT TO CONSIDER

Consider the Source
Some authors (who I think tend to exploit outcome and hindsight bias,and then combine those with indirect ad hominem attacks in order to sell their books), are actually putting forth arguments against the use of analytics.  The source of this is a current epistemic debate between those who believe that only falsification is certain, and those who maintain that neither proof nor falsification are certain, there are only probabilities.    So before you go believing any “quadrants” of usefulness on faith - I encourage you to understand what is at the heart of the discussion.

We All Have to Live In The Real World
The sun will rise tomorrow, and someone will try to find the source of the problem and do a better job.  Now chances are, they’ll be doing it in a quantitative manner.  Chances are also that at some point their models will fail and we’ll need to build new ones.  And this will happen whether the field is cosmology, economics, meteorology, information security, or professional baseball.

WHAT ABOUT YOU, ALEX?

I’m far from certain and subject to change, but these days I lean towards Robin Hanson & MIchael Lewis w/regards to placing blame.

In any case, if you do pentests, think about all the RECENT cases where you break in to a major corporation through:

• a Solaris system with Internet-exposed telnet with a guessable password OR a telnet vulnerability (circa 1994!)
• an exposed VPN appliance with a manufacturer's administrator password
• a router with default "enable" password
• or, something else entirely - but something that rivals the above example in its unparalleled, unbelievable, abysmal, deep idiocy.

Indeed, many of my pentesting friends still report plenty of such cases (one was also featured in the presentation mentioned above). Whenever I hear about it from a pentester, I always ask:

Do you think "somebody bad" had already passed through the hole you just discovered?

Maybe an hour ago, a day ago - or a year ago?!

I cannot see how the answer can be "no."

Even though pentesters usually don't focus on forensics (no time for this), it is not uncommon to notice "your predecessor's" intrusion traces while you break through systems, "plant flags", change screen backgrounds [for the admins to notice that you've been there...], etc.

Let's think what this situation really means? Here are the choices I see:

1. Nobody discovered the hole - a law of large  numbers (aka "dumb luck") have "shielded" the company from an incident. Yes, Virginia, dumb luck IS a security strategy for some companies... AND it works for them.
2. It was discovered, but not used/abused by the attacker - maybe he was busy hacking other systems, or saved this for later and never came back due to his ADD. Congratulation, you win! The immense power of dumb luck wrapped you in a protective "security" blanket ... again :-)
3. It was discovered; the attacker went in, looked around and compromised a few others systems, but found nothing of interest (no low hanging fruits)  - and he was not a bot herder. Again, you win. Next time you are in Vegas, bet on "00."
4. It was discovered; the attacker went in and deployed a bot on "your" system - given how many botnets are there, this situation is clearly acceptable to many organizations. In this case, dumb luck strategy, apparently, still work: so they use your box to spam and phish somebody else ... big deal!
5. It was discovered; the attacker went in and stole all your credit card information (it is now for sale) - even in this case, the user of "the dumb luck strategy" still "wins" (in some perverse sense)! Unless and until the stolen information IS tracked back to you OR a friendly neighborhood PCI auditor come and jams a broomstick up your ..., you can still continue to be stupid at your leisure and ignore basic security practices.
6. It was discovered; the attacker went in and stole your CEO's Inbox, including the email related to his affair (it is now on CNN) - now, in this case, you lose AND it is time to stop being stupid! Welcome to the "0wned world." Time to launch (relaunch?) your security program and get serious.

What does this teach us about RISK? The lesson here is important:

• For a security professional, an Internet-exposed system with "root/root" is an obvious HUGE risk!
• For your boss's boss's boss, it is NOT!

This is exactly why I think that the most critical problem in security today is METRICS. Metrics that a) work AND mean something to decision makers and b) can be clearly communicated to said decision makers [BTW, a) and b) are two separate problems.] Metrics that cover not only threats and vulnerabilities we face, but also the effectiveness of security countermeasures we deploy. Metrics you can act on - and ones your boss (and his boss) will act on. Metrics that lead to correct decisions about which risks to accept, which to  mitigate (all while knowing with what efficiency such mitigation occurs) and which to transfer.

Until that time, the dreaded "C-word" (compliance) will trump "the other C-word" (common sense) as a driver for security ... and we will continue to live in the "0wned world."

Possibly related posts:

• Risk vs Risk