I’m assuming the author wants us to read the title as “Things to Look Out For in Performing Risk Analysis” and not “Risk Management is Folly - Stop, Stop, Stop!” The former is fine, the latter isn’t supported by the evidence presented by the subjects of the article.
The subjects of the article are a good study from Wade Baker & Co. at Verizon, and a report from RSA’s Security for Business Innovation Council. Let’s take a look at each of these and examine why what they’re saying might contribute to poor risk management, shall we?

1.)  THE VERIZON REPORT

The Verizon report is an analysis of some 530 forensic investigations their company performed.  It is well worth your time as it’s chock full of interesting information.  As it relates to the Dark Reading piece, a coarse summary would be that “likelihood” is “different” for different people and so you can’t use the same “likelihood” across different industries.

Distilled through the lens of FAIR:

“different threat communities may be applicable based on Probability of Action factors which include: Value, Level of Effort and Risk (of Getting Caught).”

Or, even further distilled and in the words of my six year old son,

“Duh-uh”.

With regards to what I assume is the purpose of the article (What Doesn’t Work in Risk Analysis) this concept  seems just to rehash the old GIGO argument regarding risk analysis.  Great.  Can’t argue with that, nor it’s corollary QIQO (quality in, quality out).

But let me ask you -  is this really a problem common in your analysis?  Did reading this article make you go “Crap, we’ve been using data normalized across multiple industries in our analysis! They’re all wrong!”  Or have you already been accounting for the unique value proposition your company has to the specific threat community you’re worried about?  See, maybe I’m just not your average analyst, but even in my NIST/OCTAVE days, this has *never* been an issue for me.

Let me be specific, this is not a problem with Verizon’s very cool report.  It’s just that I don’t see what the big deal is.  This article is starting to feel like someone is running through the motions, trying to play the ” a crazy title gets people to read a boring article” game.

Speaking of cool reports - You know what would be cool?  I think it would be interesting to see is the quality of these companies’ “risk management process” established using good criteria,  and then correlated to the frequency and magnitude of real-world losses across the aggregate sample.  In other words, can we establish evidence that strong risk management practices not just reduce “risk” but also reduce actual incidents.

2.)  THE RSA COUNCIL “EXPLORES WHY LEGACY METHODS OF EVALUATING INFORMATION SECURITY RISK DON’T WORK IN TODAY’S CONNECTED WORLD, IN WHICH ANY NEW BUSINESS INNOVATION INHERENTLY CARRIES SOME LEVEL OF RISK TO INFORMATION.”

This report from the RSA council puts forth a seemingly obvious proposition, that risk must be balanced by reward.  Why is this news?  Now as I read the article it’s not clear if:

• The RSA Council is claiming that the CISO’s office should be the ones determining reward.  Absurd.

or

• Businesses aren’t doing a good job at determining risk and reward.

Let’s go with the latter.  So I’m pretty sure (good) businesses do a good job at estimating reward.  Businesses I’ve been a part of?  We LOVE(D) estimating reward.  We don’t tend to start projects all willy-nilly. No we tend to be careful to identify the size of the market and what it will cost to address the market.  So what could the problem be that this RSA council is trying to address?  Maybe it has to do with something like the following:

Yesterday, I got a demo of an IT-GRC application that shall remain nameless.  It seemed to be very good at the “C” bits - lots of information on regulations and expectations and even what sorts of controls would answer the regulations (which is goofy, but we’ll have to talk about that later).  It also gave you the ability to build workflow quite nicely.  But it measured NOTHING.  There really was no observable “G” and “R” was really Medium X Low X Low = High sorts of stuff.  So let’s use this relatively expensive tool as evidence of what your average CISO is armed with going into a Risk/Reward sort of meeting.  I imagine a nice board room with wood-grain paneling and glass bowls filled with little chocolate covered mints designed to give everyone involved in the meeting (CEO, CFO, CIO, CSO, VP S&M, etc…) a little sugar rush when needed and fresh breath.  The conversation goes a little something like this (apologies to Rich):

Business Guy Who Wants to Make Money Because That’s What Businesses Do: Based on market studies, we believe that initial gross revenues from the new product and technology rollout will be eleventy gazillion dollars based on a 37% market penetration in Scandinavia, alone.

CSO: Well now, we have a likelihood of “High” and a “C” impact of Medium, and an “I” impact of Low, and an “A” impact of “High” and because we are a (bank/hospital/retailer/basically any business that breathes anymore) we weight “C” by a factor of 2 - we multiplied those all together and got a “High”.

So can you guys delay the product rollout by 9 months and give me a bunch more money that’s not in the budget so that I can get this thing down to a “Medium”, please?

Again, I just don’t see the problem with Information Risk Management being that our businesses have no idea what the rewards of business might be.  Now maybe we need get a seat in that boardroom just to be able to talk about our “Mediums”, sure.  And maybe we’re infantile in our ability to describe our problem space.  But I cannot fathom that “Risk Management Doesn’t Work” because businesses haven’t been considering “reward”.

WHY RISK MANAGEMENT MAY  NOT BE WORKIN’ FOR YOU

Two meta-categories of causation:

• No skills

and/or

• No resources

Any ancillary “cause” can be mapped to one of these categories.  You could have significant resources but crappy models, and have conversations like our imaginary CSO, above.  You could have really good models and people trained and motivated to use them, but scarce time & money, so no conversation happens.

Now my question for you is - which does it make sense to acquire *first* to solve the “Why Risk Management Doesn’t Work” problems, skills or resources?

Charlie Rose:  
And so when you look at where we are going, there seems to be two issues that are apparent to me at least, risk and leverage.  We just lost sight of risk and leverage of what was appropriate?

Warren Buffett:  
Yeah.  Again, because it pays off for a while.  You know, you can lose leverage, and it's the only way a smart guy can go broke.  If you owe money, you can't pay them out.  You just pay for everything, you do smart things, you eventually get very rich.  If you do smart things and use leverage and do one wrong thing along the way, it could wipe you out, because anything times zero is zero.  But it's reinforcing when the people around you are doing it successfully, you're doing it successfully, and it's a lot like Cinderella at the ball.  I mean you know at midnight everything is going to turn to pumpkins and mice; right?  But if the evening goes along, I mean, you know, the guys look better all the time, the music sounds better, it's more and more fun, you think why the hell should I leave at quarter of 12.  I'll leave at two minutes to 12.  But the trouble is, there are no clocks on the wall.  And everybody thinks they're going to leave at two minutes to 12.

Charlie Rose:  
And should wise people have known better?

Warren Buffett:  
People should always know better.

Charlie Rose:  
Yeah.

Warren Buffett:  
I mean people -- people don't get -- they don't get smarter about things that get as basic as greed and you can't stand to see your neighbor getting rich.  You know you're smarter than he is, and he's doing these things, you know, and he's getting rich, and your spouse is getting unhappy with you because you aren't doing -- pretty soon you start doing it.  And so you get what I call the natural progression, the three Is.  The innovators, the imitators, and the idiots.  And that's what happens.  Everybody just kind of goes along.  And you look kind of silly if you disagree.  I mean, you know, you could have these crazy Internet valuations in the late 1990s, but they prove themselves out in the market.  The next day they were selling for more than they were the day before, and people said, you know, you're crazy if you don't get in on this.  So it's very human.  Now, with housing it's something even more dramatic than that, because most people aspire to own their own home.  And if you really think that houses prices are going to go up next year and the year after, you feel if I don't buy it this year, I'm going to have to buy it next year.  That's not true of an Internet stock.  But it's true of a home.  And when somebody makes it very easy for you to do it by saying you don't really have to put up my money, you can lie about your income a little, or we'll give you 100 percent mortgage, you're going to do it, because everybody that's done it has been proven right.  You have what they call social tools, and, you know, you're going to feel like an idiot if you didn't do it, because the house cost more.

Warren Buffett:  
Oh, I think confidence will come back.  I will tell you this.  This country is going -- be living better ten years from now than it is now.  It will be living better in 20 years from now than ten years from now.  The ingredients that made this country, you know, the miracle of the world -- I mean we had a seven for one improvement in the average American standard of living in the 20th century.  Now, we had the great depression, we had two world wars, we had the flu epidemic.  You know, we had oil shock.  You know, we had all these terrible things happen.  But something about the American system unleashed more and of a potential to human beings over that hundred years so that we had a seven for one improvement in -- there's never been any -- I mean, you have centuries where if you've got a 1 percent improvement, then it's something.  So we've got a great system.  And we've got more productive capacity now than we ever have.  The American worker is more productive than he's ever been.  We've got more people to do it.  We've got all the ingredients for a sensational future.  It's just that right now the athlete's on the floor.  But we -- this is a super athlete.

I’m sure you already know about the other “network management” – where ISPs and carriers get their hands publicly slapped for limiting bandwidth to high-traffic offenders. But when is this kind of “network management” a good thing? At a panel sponsored by the FCC in DC, reps from carriers and ISPs discussed what steps they’ve been taking to prepare for a pandemic or other major global crisis – that would force workers to stay at home or work from more remote locations to limit exposure.

Are people paying attention to ICANN? They’re saying that IPv4 will be fully allocated in the next two or three years. Does anyone care? In their bid to make people care, ICANN talks about the state of IPv6 adoption and touts Africa as the most rapid adopter.

SOA soon part of the ‘cloud’? No, please no.

Microsoft – The Silver Lining in Every Cloud. Joe Wilcox over at eWeek’s Microsoft Watch, has been following Steve Ballmer around and collecting some nice quotes on how the company is transitioning. “For many years, we had kind of what I would call the all-encompassing mission, vision and scorecard statement: a computer on every desk and in every home. …Well, our footprint and portfolio is broader than that. “ [In every hand and of course, in every cloud…] “So, as a vision statement we talk about creating seamless experiences that combine the magic of software, the power of the Internet across a world of devices.” The magic of software – something I haven’t thought about for a while. And:

“You need a real platform in the cloud. When we wanted to go after the PC, we built an operating system. When we wanted to go after the phone, we built an operating system. When we wanted to go after the enterprise, we built an operating system. We’ll announce a new operating system, one that runs in the cloud and has a wide variety of capabilities.”

1. Great paper that complements the whole "SIEM is dead?" saga - "Most enterprises are looking for a product that will solve all of their problems in some sort of off-the-shelf miracle, and when they find out that the currently available tools can't do it, they either postpone their deployment or put them on the back burner. "
2. "The Mess: looking for someone to blame?" is an awesome piece on Internet security and its architecture - and so is Gunnar's follow-up ("If a tree falls in someone else's silo...")
3. Mike call to "Rise up against Mediocrity."  - "Dilbert makes the risk of the lowest common denominator approach abundantly clear."; in other words, you say 'best practices', I say 'mediocrity!' Mike also remind us, in vain, to do "Security FIRST!" (and compliance second)
4. A great piece from Burton: "On Response" - I think the world needs another 10-20 million reminders that PREVENTION FAILS. This is definitely a good one for those still in the "we'll just block the threat world" - "we will not win a continuing war of escalation" and "using response can be more cost effective than installing the latest and greatest preventative tool"
5. More on metrics, including the highly-awaited ISO27004.
6. Pretty dumb paper by a person confused by why PCI DSS exists (the guy needs to read this). PCI doesn't "fall short," it helps people who will otherwise not do anything and their systems will "power" those botnets of the future...
7. While we are on this subject: a really good coverage of PCI 1.2. changes, released Oct 1st. More PCI fun here. And more here ("PCI Compliance - dispelling some common myths"). And, more PCI myths. And more good ideas on PCI from Mike R. Sorry, can't stop thinking about PCI :-)  - also this is good.
8. Adrian on behavioral monitoring; mostly in DAM, but also elsewhere in security.
9. "Premature Chasm-Crossing"  - a must-read for all security vendors and especially their marketing (and  their easily-excitable PR teams...) - "Shouldn't vendors be spending more time fighting the problems that security managers are facing today, right this minute?" (Mike R also comments on that). A related - and  just as interesting point is made here: "Security is not a solution"
10. More on compliance and security checklists, good and bad: "I think this is a dangerous trend unless the "checklist" is all inclusive." (how can a checklist include ALL? :-))
11. "SANS Top 7 New IR/Forensic Trends In 2008"
12. Read "The three approaches to computer security!"  Why? Come on, it is from Joanna! :-)
13. A fun discussion about a hot new technology: network IDS. Is IDS absolutely indispensable to ALL companies? No. Can it be incredibly useful? You bet. End of discussion.
14. On an unrelated note, are lasers the future of warfare? Some say no.
15. Finally, some security humor from Gartner (!): "Get Rich Quick With Network Security"

Enjoy!

Previous security reading.

Even thought TIBCO’s BusinessEvents does not yet support Bayesian Classifiers, Artificial Neural Networks and other advanced decision support algorithms, it is just a matter of time before TIBCO will add these advanced features “out of the box”.  On the other hand, the extensible nature of TIBCO’s BE makes it possible to add probabalistic computing functionality, however this requires quite a lot of programming and integration work.

When I see a great release like this for TIBCO, it makes me a little nostalgic for “the good old days” travelling the world in the front of the aircraft for TIBCO.   TIBCO has a rich and diverse customer base.  This customer base includes financial services companies; however, TIBCO is much less dependent on financial services than other event processing companies.   So, with TIBCO you not only get great technology, but rock-solid stability in an unstable and uncertain business world.

As a side note, an S&P analyst recently downgraded TIBCO’s stock (TIBX), primarily due to chao in the financial services sector.    Because of TIBCO’s global reach and stability, plus forward vision, advanced technologies and many years of commericial success, the S&P downgrade will create a buying opportunity for TIBCO stock.

Now I’ve been “accused” of being a quant in the past (hi rybolov!) but in reality the only dogs I have in this fight are the model and the application of scientific method - and really, ethically speaking, I have to be tied to the latter while applying the former.

And I see a false dichotomy in this whole Quant vs. Qual thing.  We, as a profession, tend to create a political divide between the two which, if it even exists, I’d say is based more on our ignorance rather than our expertise.  After all, we are the profession that regularly multiplies across ordinal scales and uses wonderful models like R=VxTxI.   As someone  learning to deal in probabilities and rationalism, I have to recognize that this discussion is really just about the act of observation using different metrics of measurement.

But how we’re going about observing does not change the fact that there is measurement based on observation.  So if I’m working with you I can easily turn your qualitative scale into a quantitative one, and vice-versa.  Yes, Shrdlu, if we had the time, even your most seemingly Qual things could be Quant! (This flexible world view, btw, is an outcome of that new-fangled Bayesian thing).

COGNITIVE BIAS A-PLENTY

But back to what Rich is saying there about information security and risk - and he isn’t/won’t be the only one saying these sorts of things - we should try to understand what’s really going on rather than get caught up in the emotional hurricane.  Our profession suffers several forms of cognitive bias.  The nature of our jobs and what we do can cause us to be focused on the outcome and not the quality of the decision at the time it was made.  We want to bring in things from other professions that are useful, but at times we do view things outside our profession with false correlation to our own (unfortunately for those who write these sorts of articles, financial risk is completely different than operational risk).  We also have the tendency to focus on negative outcomes without acknowledging the positive outcomes (For example, I hear that Alan Greenspan’s new firm is up a couple of $billion in all this mess since he joined them, short sellers are doing quite well - must be because they have qualitative models or something -grin-).  The effect of these biases are compounded by the facts that proper correlation takes more work than we usually give it, and rational thought is not that easy when there’s a witch-hunt mentality.

What also floats in water? (link to Youtube)

WHAT SHOULD WE BE THINKING ABOUT?

So as you and I read opinions that seem to be the polar opposite of irrational exuberance (and there will be plenty between now and the election) we’ll have to ask ourselves, “what really failed here?”  At the risk (pun) of over-simplification:

• Was There an Error on the part of Probability Theory?

After all, Probability Science like all other fields of knowledge is always “advancing” as they say.  So perhaps probability theory is wrong somehow?

I’m personally disinclined to put the blame here, primarily because I would think that there would be evidence from other fields (like Quantum Mechanics) that something is amiss waaaaay before it hit a field like economics.

• Was There Error In The Model Used to Determine Risk?

Some people who understand real estate valuation and complex derivatives and financial risk want to put the blame here.  It’s a little too early to tell, but one thing is for sure - Financial risk is so different from operational risk I couldn’t begin to hazard an opinion on the subject.   But it would seem that this is really somewhere we might look.

• Was There Error In The  Scale Used (Quantitative vs. Qualitative)?

Honestly?  I find it extremely difficult to understand how this could be the source of financial ruin.

• Was There Error on the part of the Decision Maker?

What if all of the above were just fine, and the decision maker chose short term gain over long term stability?  What if this was (to simplify the matter greatly) a choice of “heads” over “tails” and the coin landed on tails?  What if the model represented the right risk (probability of negative outcome vs. positive outcome), but the complex derivative was sold to someone else who had poor “risk management” (ability to make a good decisions)?

Now I have no clue about complex derivatives, and I’m oversimplifying to be sure - chances are like most things, there are several problems that helped create the primary cause. But it seems to me that as we go into incident response mode for the economy, it’s more helpful to do so in a rational, logical manner.

OTHER THINGS WE MIGHT WANT TO CONSIDER

Consider the Source
Some authors (who I think tend to exploit outcome and hindsight bias,and then combine those with indirect ad hominem attacks in order to sell their books), are actually putting forth arguments against the use of analytics.  The source of this is a current epistemic debate between those who believe that only falsification is certain, and those who maintain that neither proof nor falsification are certain, there are only probabilities.    So before you go believing any “quadrants” of usefulness on faith - I encourage you to understand what is at the heart of the discussion.

We All Have to Live In The Real World
The sun will rise tomorrow, and someone will try to find the source of the problem and do a better job.  Now chances are, they’ll be doing it in a quantitative manner.  Chances are also that at some point their models will fail and we’ll need to build new ones.  And this will happen whether the field is cosmology, economics, meteorology, information security, or professional baseball.

WHAT ABOUT YOU, ALEX?

I’m far from certain and subject to change, but these days I lean towards Robin Hanson & MIchael Lewis w/regards to placing blame.

The idea is simple — find SQL Injection vulnerabilities in high-traffic, trusted websites where the site’s content is dynamically fetched from a database (i.e. just about any content-rich site). Then use an automated tool to prepend or append malicious content to that content in the database. When the unsuspecting user visits the page to read an article, they will be treated to a barrage of <script> or other tags fetching content from sites in .ru, .cn, or who knows where else.

The guidance you give to mom and dad, “don’t visit sketchy looking sites in other countries,” is no longer good enough. If BusinessWeek can be compromised, it’s a given that USA Today, CNN, the New York Times, and other establishments are being targeted as well.

For this and similar examples, NoScript would have thwarted the attack because it wouldn’t permit the .js file to be loaded from an off-domain location. But what happens when the attackers start injecting the entire .js payload into the database instead of just a <script> tag? Now the malicious code is coming from the trusted domain, and if I’ve configured NoScript to allow scripts from businessweek.com, I’m out of luck. In fact, I have no idea why the attackers aren’t using this tactic already. Any ideas?

Two side notes: My Nmap class will be held at Ivy Tech in Sellersburg Indiana at 1PM on Sat Sept 20th 2008 in room P5. If this one goes well the next presentation will be on sniffers. Also, thanks to all of the folks who have signed up for Dreamhost using my discount code, it's really helped support the site with extra revenue.