“How I can communicate the value of an ISO implementation to non-security management?”

This question came to me after one of the posters on the ISO Google Group asked about KPIs for ISO implementation.  Got great responses in email, blog comments, and on Twitter from current/former CISO folks and consultants and analysts.  Some really great thought and effort, by the way - thank you.  It’s really great to be able to have these sorts of conversations online.

First, I have to point out some resources Brian Honan linked to from Gary Hinson, just because they’re so cool.  Gary has invested gobs of time and effort to become one of the defacto resources on the ISO (you might also want to read or re-read Gary’s web post on the 7 myths of metrics).   Brian links to an implementation guidance document(pdf) and a metrics example(pdf) document.

As full of awesomeness as they are, though, these are simply metrics “mapped” to the ISO (i.e. the ISO isn’t a pre-requisite for generating this information).  They are not KPI’s that express the value of ISO implementation.  Problem is the metrics created here still require some level of “translation” in order to create some value statement that data owners can understand.  As Myrcurial twittered me “27001 is orthoganal to process” meaning (I hope) that metrics have their foundation in events that are generated by processes.  27001 by itself was never meant to create metrics (see above), and so we’re asking a question the ISO can’t answer.  But the desire, the need to measure still exists.  To that extent we can google “ISO compliance” (whatever that means) and if something can be certifiable or deemed “compliant” we can and are “measuring”.  But does that have value? Rybolov (my favorite Guerilla CISO) wrote:

“Whatever you do, don’t start measuring percentage of compliance. Eventually, that’s what all metrics efforts around a framework devolve into.”

I have to agree.  Being ISO “compliant/certified” has little expressive business value prima facia. I find that one KPI that absolutely asserts value when expressed properly is risk - and similarly  Shrdlu wrote:

“I really have no idea. I personally wouldn’t try to justify an ISO implementation by itself. If I could show traceability on how it affected our overall security risk, then that’s what I’d do.”

And that’s a delightful answer.  That “traceability” (geeze-louise Shrdlu - what a word!) is absolutely what I’m after here.  How do I get that?  

If you’re going to do something with corporate budget (time, money - and goodness knows an ISO implementation is time & money) you better be able to communicate the value.  And while the zealotry for ISO implementation differs from person to person, I have yet to come across someone who says that ISO adoption is totally without value.  It’s just not apparent what that value of adoption is and how we can measure (metrics) and express it (KPIs).

Jenean Paschalidis wrote what he thought that value was in a very nice email in which he puts a qualitative name on the value of adoption:

“Transparency and accountability-this is what all executive/senior management (the company) is on the hook for. ISO provides that. If you want to understand and have confidence in your operations as supported by security (because you will know the who, what, where, when, why and how of a system (human, technical etc.) and you want to be able to trace back why a decision (risk-vetted) had been made - then adoption of this best international practice will assist in providing these answers.”

So working with our above thoughts a little here - if we agree with Shrdlu that the only value of an ISO implementation can only be expressed if we can say how said implementation affected our overall security risk - and we agree with Jenean that the primary benefit is an ability to have confidence in operations as supported by security, then….

The value of the ISO should be expressed as a KPI or set of KPIs that cleary explain how the confidence it generates helps us understand (and then reduce) our risk.

If risk is a probability issue,  ISO adoption helps generate confidence in our predictive analytics.  The dollar value the ISO generates (the ultimate KPI) is part of the cost of being able to make wise risk decisions.

So what is that (making wise risk decisions) worth to you?

SOME CONCLUDING THOUGHTS

First, it occurs to me that this is a real shame.  In a sense, an inability to generate a quantitative value statement for ISO use is simply more witch-doctory (“use it because we, the wise men of the tribe say you should”).  In some future version, the ISO should include some mechanism for measuring and expressing the worth of adoption to the organization (a better reason to use the ISO than “because we said so”).

Second, It should be noted that of Jack Jones’ 3 true value statements from which all metrics/KPIs should point to - we’re only talking about one of those value statements - the ability to reduce risk.  Using the ISO in an organization most certainly could create operational efficiencies (help us do more with less) - but the ISO isn’t a standard that creates operational efficiencies as a primary goal, nor does it give implicit direction on how to create operational efficincies.    The ISO folks do, however, play fast and loose with the idea of “risk” and “risk management” so it’s within this context that I interpreted our conversation.

Finally if you’re going to hire someone to help you with ISO adoption in your organization, the deliverables you ask for in your RFP/SOW/what-have-you should include quantitative (probability) statments about risk reduction and the creation of operational efficiencies.  If the firms answering can’t tell you what value their work will be to your company, then drop me a note and I’ll gladly point you to some friends of RMI’s that know FAIR & all our Risk Management frameworks and also do great ISO work.

• Low-tech is very effective. Movie-plot threats -- terrorists with crop dusters, terrorists with biological agents, terrorists targeting our water supplies -- might be what people worry about, but a bunch of trained (we don't really know yet what sort of training they had, but it's clear that they had some) men with guns and grenades is all they needed.

• At the same time, the attacks were surprisingly ineffective. I can't find exact numbers, but it seems there were about 18 terrorists. The latest toll is 195 dead, 235 wounded. That's 11 dead, 13 wounded, per terrorist. As horrible as the reality is, that's much less than you might have thought if you imagined the movie in your head. Reality is different from the movies.

• Even so, terrorism is rare. If a bunch of men with guns and grenades is all they really need, then why isn't this sort of terrorism more common? Why not in the U.S., where it's easy to get hold of weapons? It's because terrorism is very, very rare.

• Specific countermeasures don't help against these attacks. None of the high-priced countermeasures that defend against specific tactics and specific targets made, or would have made, any difference: photo ID checks, confiscating liquids at airports, fingerprinting foreigners at the border, bag screening on public transportation, anything. Evenmetal detectors and threat warnings didn't do any good:

  "If I look at what we had, which all of us complained about, it could not have stopped what took place," he told CNN. "It's ironic that we did have such a warning, and we did have some measures."

  He said people were told to park away from the entrance and had to go through a metal detector. But he said the attackers came through a back entrance.

  "They knew what they were doing, and they did not go through the front. All of our arrangements are in the front," he said.

If there's any lesson in these attacks, it's not to focus too much on the specifics of the attacks. Of course, that's not the way we're programmed to think. We respond to stories and not analysis. I don't mean to be sympathetic; this tendency is human and these deaths are really tragic. But eighteen armed people intent on killing lots of innocents will be able to do just that, and last-line-of-defense countermeasures won't be able to stop them. Intelligence, investigation, and emergency response. We have to find and stop the terrorists before they attack, and deal with the aftermath of the attacks we don't stop. There really is no other way, and I hope that we don't let the tragedy lead us into unwise decisions about how to deal with terrorism.

First, I enjoyed DeepSec conference and I am grateful for the invitation to speak there. I love European conferences – and not only for having infinitely (with that being an under-statement of the year) superior coffee during breaks :-)  In particular, I liked the audience for my presentation (slides will be posted here soon) and I think the audience liked my material and myself too :-)

What also impressed me a lot was Ivan Ristic speech, which was the second day keynote. He started by simply stating that ‘security industry has failed’ and that ‘a desktop is lost.’ His proof was in typical numbers like “75% of corporate systems are infected with at least 1 malware piece per system”, “1 million of malware types” and “25,000 unique malware samples a day seen.”  However, he then broadened the subject and talked about how not only “a trusted desktop” is gone, but the entire world of “trust everything [on a system], all the time” is gone (his ideas were similar to what I planned to present in my HITB 2008 presentation about “the 0wned world”)

I also like how he positioned all those “security user prompts” (in Vista and even before) as a proof that security technologies have failed and now we have to rely on the user to make security decisions (which will obviously fail as well since users are now fully conditioned to “see a chunk of technical mumbo-jumbo, then click OK”)

It was also interesting how he connected a lot of security failures to his “#1 reason: all programs run with all privileges of the user that runs them.”  In fact, he illustrated it by reminding the audience that “everybody runs untrusted code every day today [web browser + Javascript, etc] while nobody did this 30 years ago.”  He also beat up blackisting as an approach to security (but then again, everybody does it today :-)) - what was interesting that he opined that “we will spend the next 10 years proving that whitelisting will fail just as we spent previous 10 years proving that blacklisting fail.” His main point was that global “onslaught” of whitelisting and code signing will kill all sorts of useful things AND provide little security.

He then called for everybody to think about solving the hard, possibly non-sexy problems. This is the part where I could have used more details :-)

So, a fun speech (even though my telling of it is a bit jumbled… check out his slides whenever they are posted) – and a fun conference overall. Worth a 12 hour flight :-)

UPDATE:  @MYRCURIAL from the great site Liquidmatrix says that I need to post the following warning:

YOU MAY NOT WANT TO PROCESS THIS PRIOR TO YOUR 11TH CUP OF COFFEE

==================================

Carrying on from yesterday’s post a bit, I’m happy to admit that Chris’ poem is right: we don’t have nearly the information we need now when we’re supposed to have “control” over our assets, putting things in a hosted/asp/cloud/buzzword model ain’t going to help our quest for visibility. My intention was/is to show that you need visibility (in part one) and then today explain that unfortunately, that’s only half the picture.

Today’s follow-on is about the fact that whatever visibility we can contractually enforce (be it in the “cloud” or in our own perimeter) has to be informative (Amrit, this is why I was plugging you with those variance questions on Twitter yesterday).  That is, we can ask whatever IT department (ours, theirs, whomever) for all sorts of information, and maybe they’ll even give it to us.  But we’re not really ready to:

• Know what to ask for
• Use it to create wisdom

A really salient example of this from outside IT hit my browser this morning.  Now it’s not at all my intention to be political or endorse one candidate over another.  Those who know me know I’m fiercely independent.  But this morning there’s a headline on a well-read news website about how one candidate is now “+2″ over another in a Gallup poll of “likely voters”. The source is here.

That is a screen grab from Gallup’s website that shows the “+2″.   I have to ask - how informative is this information?  Part of the problem is that Gallup’s methods are hidden as some sort of “secret sauce” (their FAQ section doesn’t help much, either).  But regardless of the quality of the measurement, this “+2″ has no context - we don’t really know what this information means with regards to an actual election.  Nor is there any predictive element (I hate the using the word predictive, but it’s common nomenclature - so there you go).  We don’t have what we need from this Gallup poll to create wisdom about the ability of either candidate to be elected.

Allow me show you what I mean by way of contrast.  Take a look at Nate Silver’s work at http://www.fivethirtyeight.com/.  Now I’ve been long familiar with Nate due to his work in baseball.  He’s been at these sorts of ‘predictive’ analytics around our shared passion: creating wisdom from baseball statistics.

What Nate is doing at 538 is applying that acumen from his baseball work to the political process.  He’s breaking down the vote not just on popularity among likely voters, but in the context of the electoral college, accounting for variance and uncertainty, running Monte Carlo simulations and taking into account all sorts of polling information.  The result is really quite amazing. Here’s just one graph he presents - it’s the most similar to the Gallup one above, but you should really visit the site to understand the difference in quality of information and to check out the predictive elements he creates.

NOT ALL INFORMATION IS CREATED EQUAL, AND NOT ALL  JUDGMENTS ARE CREATED EQUALLY

And take a look at the contrast, here:

On one hand you have Gallup giving us a “+2″ advantage to a particular candidate.  Now Gallup themselves draws no conclusion but, as digested, how many readers do you think take this as evidence that the election is *really* close?

On the other hand, 538’s predictions show a 348/189 electoral college split, and one candidate winning 96% of the time in simulated elections.  That doesn’t seem close at all!

RISK MANAGEMENT

It is these predictive elements that we need in order to make better strategy and decisions.  I’ve been talking in the past about risk management’s inability to link current state to systemic causes, and this “context” is what predictive analytics provide.  We might have all sorts of visibility into our environment, and measurement of various amounts of variability that visibility gives us. But unless we have context to create wisdom, it’s all just, as Chris says, “machinations”.  We have to move beyond “+2″.

So Cloud/Grid/Utility/ASP/TimeShare/Whatever you want to call it - security will have to clean up our own mess first before we can do a good job with or without a perimeter.  Once we can start moving beyond “+2″ statements, then we can know what sort of visibility we require into an ability to Prevent, Detect, and Respond.

The best security program is at the business with the happiest customers.

There's a fine line between happy customers and playing piano in a bordello.

Sees a lot
Can tell the king he has no clothes
Can tell the king he really is ugly
Does not get killed by the king
Nice to have around but…how much security improvement comes from this ?

Changes happened faster that he was able to move
Did not read the signs
Good intentions went unfulfilled
A brutal way to ending a promising career
Sad to have around but…how much security improvement comes from this ?

Obviously these models of CISOs are not solving our information security problems. Instead Dr. Garigue points us to Charlemagne as a better model

King of the Franks and Holy Roman Emperor; conqueror of the Lombards and Saxons (742-814) - reunited much of Europe after the Dark Ages.

He set up other schools, opening them to peasant boys as well as nobles. Charlemagne never stopped studying. He brought an English monk, Alcuin, and other scholars to his court - encouraging the development of a standard script.

He set up money standards to encourage commerce, tried to build a Rhine-Danube canal, and urged better farming methods. He especially worked to spread education and Christianity in every class of people.

He relied on Counts, Margraves and Missi Domini to help him.

Margraves - Guard the frontier districts of the empire. Margraves retained, within their own jurisdictions, the authority of dukes in the feudal arm of the empire.

Missi Domini - Messengers of the King.

This is the way forward! Find software security champions in the architecture and development groups,help them understand the real security issues. They will find solutions you have not thought of. Same for DBAs, same for business analysts even. Its all about beating the bushes, education, and decentralizing security services. Specifically, he points out this important mandate for IT security

Knowledge of risky things is of strategic value

How to know today tomorrow’s unknown ?
How to structure information security processes in an organization so as to identify and address the NEXT categories of risks ?

To me this is our mandate and measure of effectiveness. Empower our customers, educate, and create business value. If I am a CISO  I don't want 20 people reporting to me who do firewall ruleset changes. I want one champion in 20 different groups - development teams, architects, DBAs, business analysts.

A concrete example, infosec can continue to go along with the herd and follow the "what everyone else is doing architecture" meanwhile developers are connecting every single thing in your business to the Web. I have been doing integration and new technology projects for a long time, and let me tell you - Change does not always create happy customers in the short run. But the chart below shows that information security is maybe more concerned with not causing waves rather than adapting.

The best security program is at the business with sustainable competitive advantage.