“How I can communicate the value of an ISO implementation to non-security management?”

This question came to me after one of the posters on the ISO Google Group asked about KPIs for ISO implementation.  Got great responses in email, blog comments, and on Twitter from current/former CISO folks and consultants and analysts.  Some really great thought and effort, by the way - thank you.  It’s really great to be able to have these sorts of conversations online.

First, I have to point out some resources Brian Honan linked to from Gary Hinson, just because they’re so cool.  Gary has invested gobs of time and effort to become one of the defacto resources on the ISO (you might also want to read or re-read Gary’s web post on the 7 myths of metrics).   Brian links to an implementation guidance document(pdf) and a metrics example(pdf) document.

As full of awesomeness as they are, though, these are simply metrics “mapped” to the ISO (i.e. the ISO isn’t a pre-requisite for generating this information).  They are not KPI’s that express the value of ISO implementation.  Problem is the metrics created here still require some level of “translation” in order to create some value statement that data owners can understand.  As Myrcurial twittered me “27001 is orthoganal to process” meaning (I hope) that metrics have their foundation in events that are generated by processes.  27001 by itself was never meant to create metrics (see above), and so we’re asking a question the ISO can’t answer.  But the desire, the need to measure still exists.  To that extent we can google “ISO compliance” (whatever that means) and if something can be certifiable or deemed “compliant” we can and are “measuring”.  But does that have value? Rybolov (my favorite Guerilla CISO) wrote:

“Whatever you do, don’t start measuring percentage of compliance. Eventually, that’s what all metrics efforts around a framework devolve into.”

I have to agree.  Being ISO “compliant/certified” has little expressive business value prima facia. I find that one KPI that absolutely asserts value when expressed properly is risk - and similarly  Shrdlu wrote:

“I really have no idea. I personally wouldn’t try to justify an ISO implementation by itself. If I could show traceability on how it affected our overall security risk, then that’s what I’d do.”

And that’s a delightful answer.  That “traceability” (geeze-louise Shrdlu - what a word!) is absolutely what I’m after here.  How do I get that?  

If you’re going to do something with corporate budget (time, money - and goodness knows an ISO implementation is time & money) you better be able to communicate the value.  And while the zealotry for ISO implementation differs from person to person, I have yet to come across someone who says that ISO adoption is totally without value.  It’s just not apparent what that value of adoption is and how we can measure (metrics) and express it (KPIs).

Jenean Paschalidis wrote what he thought that value was in a very nice email in which he puts a qualitative name on the value of adoption:

“Transparency and accountability-this is what all executive/senior management (the company) is on the hook for. ISO provides that. If you want to understand and have confidence in your operations as supported by security (because you will know the who, what, where, when, why and how of a system (human, technical etc.) and you want to be able to trace back why a decision (risk-vetted) had been made - then adoption of this best international practice will assist in providing these answers.”

So working with our above thoughts a little here - if we agree with Shrdlu that the only value of an ISO implementation can only be expressed if we can say how said implementation affected our overall security risk - and we agree with Jenean that the primary benefit is an ability to have confidence in operations as supported by security, then….

The value of the ISO should be expressed as a KPI or set of KPIs that cleary explain how the confidence it generates helps us understand (and then reduce) our risk.

If risk is a probability issue,  ISO adoption helps generate confidence in our predictive analytics.  The dollar value the ISO generates (the ultimate KPI) is part of the cost of being able to make wise risk decisions.

So what is that (making wise risk decisions) worth to you?

SOME CONCLUDING THOUGHTS

First, it occurs to me that this is a real shame.  In a sense, an inability to generate a quantitative value statement for ISO use is simply more witch-doctory (“use it because we, the wise men of the tribe say you should”).  In some future version, the ISO should include some mechanism for measuring and expressing the worth of adoption to the organization (a better reason to use the ISO than “because we said so”).

Second, It should be noted that of Jack Jones’ 3 true value statements from which all metrics/KPIs should point to - we’re only talking about one of those value statements - the ability to reduce risk.  Using the ISO in an organization most certainly could create operational efficiencies (help us do more with less) - but the ISO isn’t a standard that creates operational efficiencies as a primary goal, nor does it give implicit direction on how to create operational efficincies.    The ISO folks do, however, play fast and loose with the idea of “risk” and “risk management” so it’s within this context that I interpreted our conversation.

Finally if you’re going to hire someone to help you with ISO adoption in your organization, the deliverables you ask for in your RFP/SOW/what-have-you should include quantitative (probability) statments about risk reduction and the creation of operational efficiencies.  If the firms answering can’t tell you what value their work will be to your company, then drop me a note and I’ll gladly point you to some friends of RMI’s that know FAIR & all our Risk Management frameworks and also do great ISO work.

In the paper, we propose a simple solution: take-down companies should share their raw, unverified feeds of phishing URLs with their competitors. Each company can examine the raw feed, pick out the websites impersonating their clients, and focus on removing these sites.

Since we presented our findings to the Anti-Phishing Working Group eCrime Researchers Summit, we have received considerable feedback from take-down companies. Take-down companies attending the APWG meeting understood that sharing would help speed up response times, but expressed reservations at sharing their feeds unless they were duly compensated. Eric Olsen of Cyveillance (another company offering take-down services) has written a comprehensive rebuttal of our recommendations. He argues that competition between take-down companies drives investment in efforts to detect more websites. Mandated sharing of phishing URL feeds, in his view, would undermine these detection efforts and cause take-down companies such as Cyveillance to exit the business.

I do have some sympathy for the objections raised by the take-down companies. As we state in the paper, free-riding (where one company relies on another to invest in detection so they don’t have to) is a concern for any sharing regime. Academic research studying other areas of information security (e.g., here and here), however, has shown that free-riding is unlikely to be so rampant as to drive all the best take-down companies out of offering service, as Mr. Olsen suggests.

While we can quibble over the extent of the threat from free free-riding, it should not detract from the conclusions we draw over the need for greater sharing. In our view, it would be unwise and irresponsible to accept the current status quo of keeping phishing URL feeds completely private. After all, competition without sharing has approximately doubled the lifetimes of phishing websites! The solution, then, is to devise a sharing mechanism that gives take-down companies the incentive to keep detecting more phishing URLs.

Here is our stab at devising a suitable sharing mechanism. We propose the creation of a members-only sharing club with compensation for net contributors paid for by net receivers. Take-down companies submit real-time copies of their entire feeds to a trusted third party (for the sake of argument, let’s assume that the APWG takes on this role). The APWG collates the individual feeds, marks the source of each submission (i.e., which take-down company) along with a timestamp. The APWG makes the amalgamated feed available immediately to all members. The members pick out phishing URLs impersonating their own clients, while ignoring the rest. Crucially, the expensive task of verifying phishing URLs and initiating take-down continues to be performed by the take-down company.

Periodically, the combined feed is audited to determine the reciprocity of contributions. Take-down companies provide a list of their clients to the auditor. The auditor then computes the number of phishing websites impersonating each take-down company’s clients that are missed by the takedown company but identified by others. The auditor also tallies the time difference for phishing websites that are identified by others first.

For example, suppose bank A1 has hired take-down company A to remove phishing sites on its behalf, and bank B1 has hired take-down company B. Suppose 500 phishing sites impersonate A1, and that A identifies 400 while B identifies an additional 100 sites missed by A. Likewise, suppose another 500 phishing sites impersonate bank B1, and that B identifies 300 while A identifies an additional 200 sites missed by B. B has received a net of 100 useful phishing sites more from A than B has given to A. Consequently, B should pay A a previously-agreed ‘finder’s fee’ for identifying these extra 100 websites.

The ‘finder’s fee’ provides additional incentive for take-down companies to invest in better phishing website detection. Designed properly, such a sharing club can overcome the potential for free-riding that companies such as Cyveillance fret about, while increasing sharing to shorten phishing website lifetimes.

Some subtleties must be mentioned, however. If the finder’s fee is big enough, some companies may be tempted to cheat to minimize their payout. For instance, underperforming take-down companies could claim to have independently discovered missing data from their feed shortly after collecting it from the shared feed. This can be mitigated by adding a credible threat of detection — inserting a few dubious fake phishing URLs that only appear in the shared feed. If the company claims to have ‘independently’ rediscovered these URLs, then they will be caught cheating. Another issue is that the auditing system does incur some overhead, which could be avoided if sharing was made unconditional.

To sum up, we recognize that many take-down companies will be reticent to share. However, we feel that sharing is too important to the goal of tackling phishing to brush aside because of a few inevitable complications. For the good of protecting consumers, the anti-phishing industry should learn to co-operate!

Partial Disclosure - The Good

The responsible disclosure process tends to break down in rare occasions where the vendor doesn’t want to fix the issue. When this occurs, the researcher is put into a difficult position whereby full disclosure could put users’ systems at high risk of compromise. The other case where partial disclosure becomes an alternative is when the researcher has discovered a design flaw in a protocol or underlying multiple vendor component. Examples of this case include the DNS flaws published this past summer by Dan Kaminsky and the TCP denial of service condition discovered by Robert E. Lee and Jack Louis that is currently in the disclosure process. When the flaw affects a very large number of vendors and the actual problem is located within the underlying protocols that support the communications of the Internet as a whole, one possible solution is to follow a partial disclosure model where phasing the details to the general public can be used to encourage adoption and creation of patches throughout the enormous target audience.

Partial Disclosure - The Bad

What is driving the fear surrounding partial disclosure is the potential for abuse. When a major flaw is partially disclosed, a number of potential issues may occur. First and foremost, the further along the partial disclosure path we are, the more details will be released to the public, and the higher the probability that someone (either good or bad intentioned) will figure out the exploit and disclose the details. Second, when partially disclosing, the vendor’s hand is being forced into a situation that could speed up fixes, reduce testing, and cause ripple problems elsewhere within the infrastructure. It is difficult enough to dance the fine time line when doing responsible disclosure, but if we are escalated to the point of partial disclosure, additional fuel is added to the fire.

The Ugly

The real ugly part of partial disclosure is when we add to the equation the ability to spread fear, uncertainty, and doubt into the normal user community. It is generally well accepted that FUD can be used to drive additional revenue. If it is possible to increase the perceived magnitude of the “problem” that your product or service solves, it is possible to directly impact the demand for that product or service. That is the major fear imposed by the growing trend of partial disclosure. By releasing just enough information to trigger wide scale speculation into the flaw, it is possible to create buzz and garner media attention resulting in a lot of speculation and very little hard facts around the issue. The potential for abuse by the security industry at large is enormous.

The Fix

Some have suggested a group of security researchers be convened to vet the requirement of partial disclosure and to allow for independent peer review of any security research that requires the partial disclosure process. This suggestion leaves questions regarding who would stand on this group and who would be impartial enough to ensure that the right thing was always done regardless of profit potential. It also leaves open the opportunity for member researchers to utilize the information gathered during the vetting process to position themselves to profit from the data upon release. It might be wiser to rely on a higher level authority or government entity to manage this process and use the services of security researchers as required for subject matter expertise. While a group of this type wouldn’t ensure that all partial disclosure is appropriate, it would hopefully limit the potential for abuse and the ever present chance that people try to profit from the FUD that surrounds the current partial disclosure process.

You can read all the details on their website (except for the part about the certification not being a measure of practical skills). From what I can tell, the CSSLP is just the CISSP with different CBKs, or Common Bodies of Knowledge. As with the CISSP, they are going for broad knowledge, not depth. Starting in June 2009, you can get certified by taking a paper exam, likely a multiple choice test similar to the CISSP. Why June? Because the test isn’t even written yet — I’ve heard from several sources that they are actively soliciting their existing pool of CISSPs to help write test questions.

Ah, but what if you can’t wait that long and want to get certified right away? You’re in luck. If you act before March 31, 2009, you can get grandfathered in without even having to take the exam! That’s right, they call it the CSSLP Experience Assessment, and here are the requirements:

• Upload a resume showing three years of experience related to software security, or four years if you don’t have a college degree
• Write short essays (500 words maximum) discussing four CBKs of your choice
• Get a CISSP to vouch for you
• Pay $650

Let’s examine these requirements one at a time.

Three years of experience. (ISC)2 doesn’t provide any requirements on depth of experience, other than citing the broadly-defined CBKs. Considering they are targeting everyone from software developers to security assessors to business analysts (yes, really), chances are they are going to accept any experience that is even tangential to the SDLC or software security.

Short essays on four of the CBKs. I asked the (ISC)2 exhibitors specifically what they are looking for to satisfy this requirement, and they said the essays should be a general discussion of the CBK topic, optionally citing your personal experience in that area if you have any. This messaging is not quite aligned with the website guidance, which states that the essays should be “Accomplishment Records” which are self-reported descriptions of experience. Either way, with a maximum essay length of 500 words, it’s pretty obvious that substance is not (ISC)2’s first priority. Here’s one data point for you: I spoke to someone who has already submitted the CSSLP Experience Assessment, and he said it took about an hour to write the essays.

Get a CISSP to vouch for you. Actually this can be any (ISC)2 certified person, not just CISSPs. Contrary to what you’d expect, though, the person isn’t vouching for your skillset so much as they are confirming that the attestations on your resume are accurate.

Pay $650. You knew it was coming. After all, there is money to be made. How is it that qualifying for the CSSLP through professional experience should cost $650? If you’re taking the written exam, fair enough, (ISC)2 does incur the cost of administering and grading that exam (even though the Scantron machine is probably paid off by now). But $650 for the submitted-online Experience Assessment? If we assume that the person reading these essay submissions makes a rather generous $100k per year, then $650 accounts for roughly a day and a half. Will it really take that long to read a maximum of 2,000 words and pass judgment? Of course not. (ISC)2 wants to get as many people as possible to qualify based on “experience”, seeding the initial pool of CSSLPs and netting them $650 per head for doing next to nothing.

As Lee Kushner stated during his OWASP AppSec presentation (7 Habits of Highly Effective Career Managers), “the more people who own a cert, the less relevant it becomes.” Irrelevant — that’s exactly what the CISSP has become, and it’s exactly where the CSSLP is headed. Meanwhile, (ISC)2 will sit back and watch while you and your employers continue to fill their coffers.

In closing, let me acknowledge that this blog entry probably comes across as judgmental. I accept that. I’m not ranting against the idea of certifications, though admittedly I’m not a fan of them either. I am disappointed that (ISC)2, an organization with tremendous influence, could have created something more meaningful but chose not to. Why bother when people will just fork over the cash anyway?

So, what are we hoping to gain by creating a network of security consulting and training experts to work with customers who want to implement the SDL?  Generally speaking, this question has a two-part answer:  First, Microsoft is, and always will be a partner-driven company - we rely on the skills and capabilities of our partners to provide specialized services and broad geographic coverage for Microsoft products and services.  Second, even though there are talented folks in the Microsoft Services organization, it's clear that we will need help from our partners to scale to meet the demand.  I can't tell you how many times the folks on the SDL team have been approached by people - after an executive briefing, or a session at TechEd - asking for guidance in implementing SDL in their own organizations.  When we look at the demand and pair it with the geographic diversity of our customer base, it's clear that a partner approach is the right answer.

Now a few words about the partners who will be participating in the pilot phase...

After the decision was made to work with partners on SDL delivery, we had two primary criteria that we had to address; partner quality, and manageability of the SDL Pro Network pilot. We have all seen instances where individuals or consulting organizations have represented themselves to the IT community as having security expertise when in reality the "experts for hire" were simply reading a page or two ahead of the customer in whatever security tome was "in vogue" at the time. 

Based on those observations, it was clear that partner "quality" was a critical criterion.  Fortunately for us, we didn't have to look far to satisfy our quality bar - many of the companies in the SDL Pro Network pilot have direct experience with executing portions of the SDL on our products, or have delivered services to Microsoft in a security context. Design reviews, code reviews, penetration testing, training and other tasks critical to SDL implementation were (and are) common fare for these folks.

Despite the customer demand for SDL that I alluded to above, starting with a small pilot was the right thing to do; a small group of trusted consultancies supports our imperative for quality and it allows us to pragmatically grow the SDL Pro Network as the market matures.  As we continue to evolve and innovate with the SDL, we'll have a strong core of partners to help drive the software security message.

Will we grow the SDL Pro Network?  The qualified answer is: "When the market demands it..." - there are a number of talented potential partners who meet the quality bar - and clearly, the need for security in software development will grow to demand additional talented specialists. However, it's our plan to begin with a small set of partners of known expertise, and then respond to growing demand as it materializes.

So there you have it - the nuanced beginning and bright future of the SDL Pro Network...  I invite your comments, and encourage you to check in at the SDL Portal as we continue to build out the program

HANSEI - WHAT IS “RELENTLESS REFLECTION?” - And why we’re talking about it in the context of Risk Analysis.

Recall from yesterday’s post about how I got to thinking about the concept of Hansei-Kaizen, “relentless reflection” and “continuous improvement” and how we might apply that to risk management.  It’s a concept born of Toyota and is, in some way, the foundation for “Lean” production.

Call me biased, but I think that Hansei - the act of ‘relentless reflection’ made structured is the analytical function.  And I hate to debate (post-mortem) the father of Toyota quality success when he says that Hansei is the “check” in Plan/Do/Check/Act, but I think that Hansei also applies to the “Plan” of the P/D/C/A or Deming cycle.

You’ll recall the P/D/C/A cycle can be thought of even as an implementation of Scientific Method, in that it is Observation & Hypothesis Creation (P), Experiment (D), Analysis (Check), and Act (Revise/New Hypothesis, etc…).  Well then as such, the Hypothesis creation involves creating a model or creating an expected outcome for data using the currently accepted model.

So in our industry there is an opportunity for Relentless Reflection in both the Observation and Hypothesis (Plan) creation steps, and the Check step.  We create an estimate for control strength, or probable losses in the context of risk- then we go to Experiment step.  That hypothesis can be put it into production, have an audit, have a penetration test, whatever, in the context of the Do step.  BTW - using Hansei/Analytics in Plan is one way that strong analytical functions can really make penetration testing more useful - as a means to test the estimates and inputs into a model.  It’s Penetration Testing 2.0!  (<- tongue fully in cheek, yes)

Those who are versed in the reasons to merge Six Sigma and Lean together are probably already seeing where I’m going with this today.  But before you think that a simple DMAIC function is all that is needed to create proper “Hansei”, let me encourage you to keep reading.

Now if the analytical function can said to be “reflection”, why must it be relentless?

One word.  Change. There are essentially four separate “landscapes” or sources of change that we face (more on those tomorrow).  But anyone who has tried to manage system compliance, log management or policy exceptions knows that change is possibly the most difficult thing we security professionals must manage.  And when you think about it, there aren’t too many other business functions like information security where significant visibility and insight about the environment is needed for “complete” information (get bullish on Log Management is my recommendation).

HANSEI STEPS ADAPTED TO INFORMATION SECURITY

This is one of those quality control concepts that we can mangle adopt.  At Toyota, Hansei-Kaizen includes the following basic steps:

1. Initial problem perception
2. Clarify the problem
3. Locate area/point of cause
4. Investigate root cause (using an ask why 5 times approach)
5. Countermeasure
6. Evaluate
7. Standardize

Now it’s important to note that part of this includes the concept of Go See For Yourself, called “Gemba“.  Gemba can be translated as “the actual place” or “the place where virtue or truth is found.” At Toyota this might mean going to the shop floor to see the issue at hand in the production line.  But for us, that’s a problem because we live in the virtual world.  There’s usually not much use in hanging out in the wiring closets to try to see the problems.

But if you combine the concept of Gemba with the concept of “Nemawashi” –the process of discussing problems and potential solutions with all those affected- we can forge a similar concept using risk analysis.  That is discussing the issue and the risk associated with an issue (what some people would call “risk management”) with the business/LOB/data owner and let them accept authority and the risk decision.  We, the risk analyst, our goal is simply to perform items 1-5 (presenting countermeasure options that include transferring or accepting risk).  By going to the line of business and involving them, responsibility is shared.  Also, if you structure organizational behavior right, personal risk is transferred!

This sort of approach is also in harmony with concepts like “mutual ownership of problems,” or “genchi genbutsu,” (solving problems at the source instead of behind desks), and the “kaizen mind,” (an unending sense of crisis behind the company’s constant drive to improve).

One of the criticisms I have with the way most people try to implement DMAIC into “Lean”

REQUIREMENTS

Now to get this done, I really see three significant requirements.

1.)  A change in political structure.

2.)  Models that provide consistent, defensible analysis.

3.)  A Quantitative approach.  This means using actual units of measurement (not just amorphous percents, ordinal scales, etc.)  for risk and it’s subsequent factors.  Sure there are times when Q&D qualitative approaches are acceptable, but policy should be to have quantitative analysis whenever and wherever possible.

That last item - the quantitative approach - is really quite important.  And the reasons why will be discussed further in tomorrow’s post:

“What should we be reflecting about? & What is needed for reflection?”

P.S.  Your comments and suggestions, as always, are welcome.

P.P.S  Those who may be familiar with Lean/SixSigma/Kaizen sorts of mashups may be thinking - “hey, an Analytical step is built into SixSigma”.  Well, yes there is some prevision for analytical functions based on statistics, but I find SixSigma geared towards creating a State of Knowledge about operational processes, not towards creating a State of Wisdom for CISO’s around security & risks “big questions”.  In otherwords, the analytical function in DMAIC is in the context of Kaizen, and a different step than “reflective” analytics.

1. DNS + Karma = Boom! Enuf said. Also, hear Pete Linstrom squeal.
2. Fun essay on "blocking" and risk. Is it our job to stop'em from using Facebook?
3. MS Exploitability Index. Smart ... or misguidedly focused on "vulnerability release" (and not creation)
4. Chip-n-PIN, a PCI killer? I don't think so!
5. Mike R revisits "good enough security" - read it, then review your IR plans (...for you will be 0wned)
6. Very fun RSA survey here; data leakage beats malware again, people still not report incidents (to whom???)
7. More and more and more people point at idiocies of academic security research... Read the whole w00t 08 thread here. Weep. Laugh.
8. Neosploit has a bad quarter... breaks support "contracts" ... shuts down? Ah, the economy :-)
9. Awesome stuff from  Richard Bejtlich: CAER.
10. "The Network Firewall is a Consensual Hallucination" :-)
11. More GRC-ball-kicking: here, here ("IT-GRC "vendors" are not IT-GRC vendors") - both are pretty insightful for GRC-lovers and GRC-haters)
12. More SIEM-ball-kicking: here ("underwhelming","ridiculous", "missing the point"), here ("dead ...unless","cripple")
13. Fun DLP portal launches.
14. Final word (?) on TerryChilds-gate here. "When management starts controlling the actions of admins, things start to fall apart." Huh? When management loses control of the business, it dies. Folks, IT vs IT security gap IS real. I never quite believed it, but this taught me a lesson. Some common security sense for a change (also here).

Enjoy.