Today, allow me to update you on FAIR and the movement towards a formal, open standard.  There’s a couple of cool things going on in our little risk-world.

First, The Open Group Security Forum continues to move towards a formal adoption of FAIR.

WHAT DO YOU MEAN “WE” - YOU GOT A STANDARDS BODY IN YOUR POCKET OR SOMETHING?

Our meeting in Chicago a few weeks ago was great, but also slightly disturbing for me. I got pronoun-confusion syndrome.   I’m used to using the “we” pronoun to refer to RMI, or Jack and myself as we vet the models.  So without even thinking I would said “we have been looking at how loss occurs, and may want to change the model some” and The Open Group Members freaked out (rightfully so).  Adrian Seccombe gently reminded me that the “we” was now the Security Forum, and that “we” didn’t go changing things at will without vetting against each other.  Man I love this stuff.  I get to run our thoughts and ideas past some great folks now - you know, those smart people who tend to have really complex problems and are trying hard to solve them.

Formal Adoption:  Soon, Very Soon Now

Formal Adoption basically means we’ve made this document, everyone is close to saying that they generally like it, and once that finally happens then “bam”, we’re ready to move onward and upward with better things (see Cookbooks, below).  We’ve got a couple of changes to the current document that have been requested that aren’t a big deal.  For example, one request is that we make some statement about general applicability of FAIR to risk domains outside of the IT realm.   But once additions like that and others are done, this long process should be complete.

New Document Moving Towards Public Release:

We’ve got a basic document that should be public in the next few weeks on “What Makes a Good Risk Assessment Methodology” - written by yours truly and Jack.  It’s a very high-level document, and serves two purposes:

• For novices it helps parse out what is important in any undertaking to understand corporate risk (the repeated discussions on the ISO 27001 mailing list make me think it would be a place ripe for such a document).
• For those who “know” risk, it helps to re-establish some fundamental principles like the use of scales (ratio, please), the implications of dealing in probabilities, what attributes like consistency and defensibility mean, how “risk” should be reported to the business (something you know, meaningful) and so on.

When this doc is deemed ready for public consumption I’ll be sure to post on this blog here.

COOKBOOKS, EUROPEAN AGENCIES, AND, IRON CHEF “RISK” - WHOSE CUISINE WILL REIGN SUPREME?

One interesting thing that came up in the Chicago meeting was that ENISA (The European Network and Information Security Agency) developed a very nice document that reviewed something like 18 different risk assessment methodologies against their Criteria for Goodness.  FAIR was one of the ones they reviewed, and we (the royal “we” used there to include all us FAIR-Folk) did awfully well.  Things of interest:

1. They based their work on the current introduction paper which is not at all a step-by-step guide towards an organizational risk assessment (what ENISA really wanted) and we did pretty well.  Well enough that if we had developed a paper along the lines of NIST 800-30 or OCTAVE for the use of FAIR in a formal process, we could have done really, really well.  Like won-the-bake-off kind of well.
2. FAIR is actually not at all incongruous to many of the risk assessment methodologies offered, and in fact compliments many of them by letting those methodologies develop real, structured probabilities.  Think OCTAVE, where they basically say “math is (probabilities are) hard, so if you want to do them for reals, good luck!  But here’s a nonsensical way to do things if you want to believe in magic-fairy risk“.  FAIR fits right in there by stomping on the magic-fairy risk with the jack-boots of rationality.  FAIR similarly helps other risk standards that might lack structured probability development.

So The Open Group Security Forum decided that though we could create a new document and totally p0wn any future ENISA bake-off, there wasn’t much demand for the development of that documentation by the membership  - a point which was made quite apparent at the beginning of the discussion when one large European company CISO asked “What’s ENISA?”  Relevancy is everything, I suppose.

But that second item up there - the one about helping rather than competing with other “risk assessment methodologies” - really struck a chord.  So “we” (The Security Forum) are going to develop some “Cookbooks” that basically are high-level documents that say “If you want to use FAIR with (OCTAVE/COSO/CoBIT/Whatever) here’s how it fits, makes it better, and improves your life.  I’m pretty excited about these, and our first document looks like it’s going to be COSO integration.

THE OPEN GROUP SECURITY FORUM - THEY’RE A TRUSTING BUNCH (WITH QUALIFICATION, OF COURSE)

Finally, many people have asked me “Why work with The Open Group?”  There are many reasons, to be sure, but I will give you one example.  Members of the Security Forum there are not only great at vetting the model and getting consensus on risk and risk factors - but they’re quick to start applying.  So in Chicago, I thought I’d be talking about FAIR and the standard and fighting groupthink.  Nope.  Not at all.  In fact, the forum members spent more time suddenly discussing use of FAIR in a new Trust Model they’re developing.  So all of the sudden, I’m part of a new and exciting project to develop a Trust Model - how cool is that?  While formal adoption of the Trust Model will be necessarily long and deliberate - the collaboration and development is happening much faster than I can keep up with.  But if you all will allow me, it will help me get my head around it all by blogging about it later this week.  So be prepared to read about me dealing in “Trust” a little bit.

Have you ever sat in a corporate training? Some are good, some are bad, but did you ever say, “man I can’t wait for training today.” What about mandatory training? What about mandatory training in a subject that you really don’t think is your area? What if you had to do it every year, and got harassed if you didn’t do it? What if you were, say, an audio engineer and were dragged into a security class?

I ran the SDL training program at Microsoft for a long time, and developed and taught a big chunk of the training. I spent hundreds of hours in front of thousands of developers, testers, and program managers. I got some really good reviews (and a few bad ones) on the classes I offered. And I tried to do a lot of things to try to make the trainings interesting. I handed out dozens of fresh peaches in an early class on fuzz testing, for example.  The room smelled really nice after that, and there are probably still a few people around Microsoft who think of fuzz testing when they see a peach.

But even on my best day, I was under no illusion that the majority of the audience was excited to be there, and I was certain that they weren’t going to go back to their offices and spend weeks applying the lessons from the class, setting aside other things that are causing present and immediate problems in favor of something that is far off into the future.

You have to work at getting people’s attention – especially as it relates to security and privacy. From time to time, I would see people reading their mail in class, and I would point to them and ask them a question. That did not endear me to the audience as much as the peaches, but embarrassment is always fresh and in season.  J

One student wrote of one of my classes, “the basics for secure design - could be replaced by non-anonymous site-wide exam with open material.”  He was not alone, I assure you.

Is that an indication that our training, or any training, is pointless? Hardly, but training alone is not a change agent.

Richard Derwent Cooke wrote,   “It is a first principle of Change Management that people will act in what they perceive as being their best interests.”

At best, training can provide people with insight into what they need to do to solve a security problem if they believe that solving that security problem is in their best interests.

To be effective, training needs to happen in an environment:

·         Where expectations are clearly set (the SDL sets specific minimum requirements).

·         People have appropriate incentives and consequences (security is a great career path at Microsoft, and nobody wants to be the one holding up a ship schedule for failure to meet a security requirement).

·         Where tools and resources to accomplish the goals are available (we build a whole variety of tools that map to the SDL requirements).

·         Where management models the behavior (recall the original BillG TWC memo).

·         Where the environment reflects and supports the values presented in the training (apparent in everything Microsoft does).

Don’t make the mistake of thinking that a bunch of training, even really high quality training done periodically, will result in actual behavior change. It won’t. You have to build an environment where people perceive solving security problems as being in their best interests. You have to make security their problem – not in the sense of passing the buck, but in the sense of changing their behavior so they will bring security problems to you.

To illustrate further, I’ll cite two examples. First, fuzz testing. Fuzz testing has been a success story here at Microsoft. Tools arise spontaneously to solve new fuzzing challenges, written by people who believe the challenges are their challenges. There are people who feel ownership for our fuzzing strategy and on-going research and science, there are specific goals and requirements, we have training (remember the peaches?), and internally developed fuzzers have won prestigious awards within the company, handed out by members of the executive staff, and all of this gets revisited periodically as part of the SDL.

 By contrast, I’ll choose a less successful area – defect estimation. On my own volition, I created (based mostly on some excellent material from Microsoft Research) and taught a class called “Defect Estimation and Management” and added it to the SDL curriculum. Microsoft is a great place to work in that regard. It was pretty close to the best-reviewed class I taught. But, we have not yet been able to establish a set of tools to estimate security defect density effectively, and establish a fair set of expectations, incentives, and consequences, or even  decide what we should do if we had the data. We discovered some things, though. For example, based on what I observed (which should not be construed as rigorous research), it does not appear as if the density of general defects correlates closely with the density of security defects.  And Microsoft Research found higher code coverage in testing correlates with higher bug rates in the field.

And so even though people like the idea of defect estimation, and we’ve got some interesting and surprising data, we’ve not yet been successful in changing people’s behavior.  Generally speaking, an individual test manager does not feel that establishing a high quality estimate of their defect density is in his or her best interests, as compared to, say, improving the time in which an established series of tests can be performed .  

We need to build an environment that has the tools, training, rewards and incentives, and expectations and consequences to change people’s behavior. Not that we’re not trying. But training won’t solve it alone, nor would tools, trophies, rants, testing, code review, or some edict from on high. The SDL is as much about changing the culture and influencing the behavior of individual engineers as it is anything else.

I’m convinced that Microsoft’s SDL process works because it addresses the end-to-end problem - from training through servicing, and provides a complete environment where people feel ownership of their part of the security problem and have the resources to solve it.

So the next time you find yourself sitting in some mandatory training, remember the lessons of the SDL (and most of the research on human performance management): training alone won’t cut it. If you want real behavior change, there have to be things outside the lecture room to influence people to change their behavior.