FEAR AND LOATHING IN DAYTON, OHIO

Had a great time Sunday with Rob Newby. We solved the world’s problems over deep fried whitefish and french fries (fish & chips to him).  It was a very good time, even if my driving did make him a bit uneasy.  If I may quote myself (said in an attempt to soothe Rob’s uneasyness about being lost in the car of a complete stranger in a strange country):

If your life doesn’t imitate the surreal aspects of a Douglas Adams book at least once a day, you’re just not living right.

Aside:  Bruce Scheier already has too many awards and too much recognition, so go vote for Rob instead :)   :  http://robnewby.blogspot.com/2008/07/award-up-for-grabs.html

SEPARATION OF CHURCH AND (CURRENT) STATE

Rob and I spent some time discussing risk and security,  and our conversation circled around the (now) recurring blogo-topic concerning the State of the Practice.  It’s a favorite topic of mine, so I’ve been delighted that it has reappeared in blogodom.

Rob writes about it some here in PCI the Priest.  LonerVamp’s and Richard Bejtlich’s blogs talk about Galileo, his confrontation with his church, and lessons we can learn from history (there’s nothing wrong with them recycling the meme, IMHO - because I, for one, never got closure the first time). Jon added a nice quote from Feynman today that’s also inline with the meme.

I’m not going to belabor the analogy, the “art vs. science” misnomer, nor discuss the problems with our various canon (PCI, ISO, CoBTI, COSO, blah, blah, blah).  Rather I’d like to talk about some essential things I think our industry needs to “sort out”  before it can move on towards a more scientific view of the world.  And by “sort out” of course, I mean agree with me on

CAN’T WE ALL JUST GET ALONG?

1 - Can we agree that risk is a probability issue?
Now obviously, you can retreat in probability theory a century or so and claim that risk is a Knightian uncertainty and that we just can’t “know” it.  Have fun.  But you should know that there’s the catch - “security” is also a probability issue.  So I’m betting that you can’t know “secure” for much of the same reasons Frank Knight would argue we can’t know “risky”.

But if risk (and security) is a probability issue, however, then we’re going to have to do better than “A’s in three college courses in statistics” to address the problem.  We will have to do as Curphey (and others) suggest and bring elements of other disciplines to bear on our problem space.  Let me suggest probability theory and economics as fine, fine places to start.

2 - Can we agree to stop measuring stupidly?
We have to agree that Ordinal Scales are not measurements, and Interval Scales are not useful measurements?

I had a post titled “More Ways To Confuse Your Auditor/Assessor” but it turned out to be a pretty cruel discussion about how we tend to try to act like our calculations based on ordinal or interval scales are useful (hint:  insist that your auditor/assessor/consultant replace the label “one” with the label “zero”).

Note that if risk is a probability issue, then we’re going to have to throw out the concepts of measuring in any scale other than a ratio anyhow.

3 - Can we agree on a (good) taxonomy?
We’re going to have to do (much) better than ISO 27005 (nudge, nudge).

4 - Can we agree we need to do a better job with our data?
We’re going to have to do better with measurements, metrics, models and testing.

It’s a shame that honeypots tend to be under appreciated.

5 - Can we agree to test that data and share it with each other?
We may not need to share specific data, but we will need to share when a model falls down.

I’d like to be as idealistic as some of my fellow ‘New Schoolers’ and suggest we’ll someday all be sharing data together, but I’m skeptical.  But that doesn’t mean we can’t demonstrate where results from the models we use are not repeatable, consistent or logical.   One thing Rob and I talked about at length yesterday was the ability to disprove a model using realistic but “substitute” or sanitized data.  There’s gonna be a TON of work to be done here, and that work will take not years but careers.  Which begs a great question:

Is it the sharing of data that we need, or the sharing of models?

HELP ME OUT, HERE
That’s my list of 5 fundamental concepts I wish we could move past.  Let me ask you - what else am I missing?  What’s it going to take to get past our current malaise?  How does the New School reach critical mass?  Who is going to help us agree in a centralized manner?

Your comments or own blog posts are most welcome (please include a trackback or post here)