First, I’d like to thank Steve McCalmont for including FAIR in his excellent article in the May 2008 ISSA Journal, “Streamlining the Risk Management Process”. Three quick things to anyone who has read it and is visiting our blog for the first time:
- We don’t believe that the goal of Quantitative Risk Analysis is to be precise. We believe the goal is to be accurate. Subtle but important difference.
- FAIR can be used both Quantitatively and Qualitatively. The decision on which method to be used depends on various factors that Steve lays out nicely in the article there.
- We believe that Risk Management is more than looking at specific vulnerabilities, their likelihood and impact. It must encompass all aspects of the organizations ability to effect the probable frequency and magnitude of loss on an aggregate level, not just within the context of a discreet technical or policy issue.
That last point is important. And it’s related to my post today.
WHAT DO YOU MANAGE TOWARDS?
This blog is blessed to have some very smart people be part of it. There are security managers from all sorts of industries that read and comment and contribute. And so today’s blog is more of an open-ended question for you all. It’s a question that, if I have a comfortable relationship with the organization I like to first ask the senior manager, and then subsequently ask the direct reports.
When you think about it, Sales & Marketing managers have goals they manage towards. CFO’s have goals that they manage towards. COO’s have goals and measurement that they manage towards (cost management, production, etc…). So what does the CSO manage towards? I’m guessing if we took a national poll, we’d get all sorts of very nice sounding answers to that question. I thought I’d list some of the answers I’ve heard and talk about them with you today.
1.) Being Secure or “Managing to Security”
Generally, this concept of being secure is the most common answer. And when I’m given that answer, it generally means that management focuses on Vulnerability Management, Patch Management, and to some degree, log analysis from various sources. These are very basic core security functions, and the belief is that if we do these well, we will be “secure”. Ok, well… what does this “secure” mean, and how can we talk to management about whether we are meeting this goal? If you examine that question, you actually find out what a “Being Secure” organization is really managing towards, another answer I hear often:
2.) Being Incident-Free or “Managing to Perfection”
Security Person: “Alex, our goal is not to have any incidents.” Alex: “Good luck with that.”
OK, that’s not what I really say, but here’s the problem I see with this common answer and the one above both of these common answers: How do you know if you’re good or just lucky?
Well, are you, punk? (youtube link)
In my six years of working with a Penetration Testing team, nobody ever really “passed” with a perfect score*. Some did better than others, some folks looked really, really good - but the degree of good/bad was really more dependent on scope than the actual state of controls or the ability of the team to overcome them. That is to say, when pressed, the mature security professional must admit that, given a strong, capable threat community - there is no secure. And therefore any state of “incidentlessness” deals with some combination of amount of control strength, and some lack of attacks (frequency!) by someone with enough skills and resources to overcome those controls. If that last sentence sounds very FAIR-Like, that’s because it is. If FAIR really accounts for those things that create Risk, then Managing to security or lack of incident means that you’re primarily concerned with FAIR Vulnerability, and ignoring other critical aspects of risk (like frequency of attacks, controls that reduce the probable impact of an event due to an ability to respond well to external stakeholders, etc…).
3.) Being Compliant or “Managing to Compliance” (External Compliance Pressures)
Because that’s what business buy, right? They buy compliance! Or so I’m told. So let’s say that you go out and actually twist senior managements arm to get them to cough up enough dough so that you can be as compliant as Large Accounting Firm says you need to be. Good on you!
But what I always wonder is, what happens when you want to manage something beyond compliance? What happens when the checklist you’re managing towards is run by a bureaucracy that can’t keep up with a changing threat landscape? For many people, the answer is “GOTO 1″ and try to sell upper management using FUD (hey, it used to work, maybe it’ll work again). An alternative is to get to the next step:
4.) Being Measured or “Managing to Metrics”
Say what you will, but “quants” have one thing right. What gets measured gets done. And a few mature organizations have spent a ton of time and effort on being able to create dashboards of KPI’s that attempt to measure security. Problem is, that we don’t know if a 98% on patch levels is good or bad or just right. We don’t know what value, if any, does creating metrics around the number and severity of vulnerabilities found in a monthly scan actually have. So we’ve come up with this thing called “GRC” that’s supposed to make sense of those things we can measure empirically and help you find out if/when you’ve fixed them. And while GRC tools can tell you some good information about systems out of compliance, they tend to give you fantastic information like how your “risk = 57“.
Wha….?
Risk = 57 means very little to someone who doesn’t spend their life in the machinations of the GRC indicies. So again, measurement without a (good) model still falls down when faced with that ultimate business decision. Or, as Shurdlu so eloquently puts it in her post on GRC:
“By contract, risk is personal. It’s variable as hell. It “governs” what you spend your money on, and therefore, with or without a dashboard, your CEO is already doing risk assessment every time she decides what your security budget is going to be. Will you really be able to change her mind by showing her the dashboard and saying, “But—but—the needle is pointing to RED!” when you’re sitting there with your line items in your fiscal shopping cart? “
5.) Using Risk or “Risk Management”
Which brings us to my favorite, using risk (as defined as the probable frequency & probable magnitude of loss event(s)) as a means to manage. Now many industry veterans will tell you how jaded we all are on the term “Risk Management”. And we have every right to be, as Risk Management has been horribly abused by vendors, committees and standards bodies alike.
These days, the term has been narrowly defined to mean an extension of vulnerability management. This is small, small thinking, IMHO. To me, Risk Management isn’t the management of individual issues deemed as “risky” as much as it is measuring (see 4) our ability to make decisions through the lens of risk. Maybe I should start saying “Risk-Based Management” instead of “Risk Management”.
This Risk-Based Management approach provides meaning to metrics. We can know what we’re measuring and why we care about it. And why we care about it needs to match what management cares about. If your approach to Risk Management results in some metric or KPI that non-IT (or non-security) management doesn’t understand or speak to them in an evident language, it’s time to find a new model. This is why “Quants will win” and where risk = 57 is wrong. Risk, expressed as “expect a once in 5 year chance to lose $875,000 if we don’t spend $90,000 now” actually gives executives something beyond an arbitrary ordinal number or color to work with. And what’s interesting is, if your model does the right things in getting you to that expression - then metrics and KPIs - those “why/when/where” questions we have a tough time answering about metrics - they become easier to discover.
DISPROVING RISK MANAGEMENT
As a side note, originally I was going to write today a completely different post on how we can disprove whether or not OCTAVE or 800-30 or ISO 27001 risk management efforts are really “Risk Management” - and one significant point was “Does your non-IT management really care about the deliverable?” This thought came to me after seeing a few too many emails into the ISO27001 mailing list asking “How can I get management to fund ISO 27001 certification?” Of course, the value of implementing the ISMS and the value of certification are two separate business propositions, but if you can’t sell the first, then are those efforts really good risk management? You know, the kind of effort that we can use to make meaningful reporting?
=============================
* I can tell you that at times we were asked to test products out for clients before they made a significant investment. One biometric device stands out in memory as not being “hacked” in the time alloted for the engagement by a defense contractor. After it passed the “Gummi Finger” test - we were going to try using a recently severed finger, but oddly enough nobody on the team volunteered their digit for the sake of security. Bunch of slackers.
