I’d love to have another blog even more specific - “Ok, that Assurance is Evidence Of What, Exactly?

Today he has a great article called:

What’s the matter with Risk Management?

And “in short, it’s everything.” It pretty much sums up why I had to grow to re-evaluate how our industry does risk, risk management, approaches controls & vulnerability and find a new way.   A couple of things jump out at me in reading Chris’ article:

1.)  Just because that Deming cycle sucks and is full of unknowns doesn’t mean “risk” doesn’t exist, nor that it isn’t of primary importance. Nor does it mean that in the absence of model & methodology, we won’t be “doing” risk analysis anyway - just in an ad hoc method and completely from “the gut”.

Our industry calls these unstructured risk analysis “Best Practices”, as it’s an easy and convenient way of sweeping the unknowns under the rug of bureaucracy and enforcing it via peer pressure.

2.)  What this “suckiness” does mean is that your model and methodology aren’t helping you. As Chris intimates, there is too much uncertainty in the inputs for his model (they are, in the language of Bayesians - too subjective to be useful priors).

Take for example how we might be approaching the “controls” part of our analysis.  Chris writes:

“2. What are the controls that we have to employ?
800-53, ISO 27001, PCI, etc.

Still kinda good, but we basically know that ISO is relatively voluntary and NIST supplies a control catalog and not policies. So here we have to take the control catalog, and mash our policies into it.”

I wouldn’t call this “kinda good” at all :)  These control catalogs only provide a hierarchy within which to look for evidence of  our ability to resist an attacker.  They are incapable of making any claim about the effectiveness of the controls when they are operated at 100% efficiency, or more importantly, what % efficiency our specific organization operates at.

Let’s use Chris Hayes’ Initech as our fictional example.

Initech has a control (a back door on a loading dock).  Now the locks on the door are 100% capable of locking the door.  This is different than saying that they are capable of frustrating all but the top 5% of lockpicking burgalars.  It is also diffferent than saying that in a sample of several “walk around audits” the doors are left open 20% of the time (they are not in compliance with policy 100% of the time).  Even worse, that 80% of the time the door is not propped open?  Yeah, tailgating is a known issue.

So we have several different variables here that we need to account for (and it’s just a door).  But the analogy stands that most “risk management” methodologies are “We have a door, yes/no?” And most GRC platforms, when asked for their “opinion” will simply say “door is needed” or, even worse, “a door policy is needed”.

3.)  Criticality and the Source of Value is all messed up in these Risk Management models.

Chris writes:

Someone wants me to tell them which boxes are more critical than others. This is mainly because of budgetary or operational reasons. To which I usually say “All of them, it is a system after all”.

This literally made me laugh out loud.  And this sort of “rate the firewall as Risk = 500 but rate the actual business application as Risk = 157″ thing is also endemic.  Now Chris is very smart here.  He correctly identifies that the value is tied to the business process the systems support, and not to a specific box.  Oh, we scan at the specific box level - but because of the nature of systemic failures - all the boxes in the process are inexorably interrelated.

One of the reasons I really like FAIR is that the losses are quantified (or qualified) based not on some amorphous value of the box or the process itself, but losses are linked to the actions that the threat will take. Take systems in a highly regulated industries as an example.  Usually the most probable losses aren’t due to system compromise per se, but in the disclosure the compromise causes (regulators are a threat source, after all).  But many “risk management” methodologies will say “online banking is worth $2 billion, the value of the systems is therefore $2 billion”.  And suddenly we’re telling executive management that there’s a 60% probability that they’ll lose $2 billion.

4.)  If the primary source of prior information for your “risk management” methodology is a vulnerability scanner - you’re doing it wrong.  Chris writes:

So we ran a scan and now we have a report. A snapshot in time to make all decisions. Where did these vulnerability ratings come from? Do I even know if my system is at risk? What if I spend my time on vulnerabilities that have no threat?

So first, my thoughts are that actual “vulnerability” must be a comparison of the force a threat can apply, and our ability to resist that force (this is a probability statement, btw).

Changing your thinking about vulnerability now helps us understand the problem in several new ways.  First, you can start to divorce yourself from the scanner.  After all, the scanner is simply providing you with current state information that is usually just relevant variance from policy. It doesn’t really tell you about real “weakness in a system” because the system is an interrelated mess of people, processes and IT assets.

5.)  Finally, most “risk management” approaches just *don’t* do a good job of helping us understand the how’s and why’s of managing risk. In the past, I’ve referred to these standards as really being “issue management” because they are at their heart, an act of discovery - a formal process around gathering prior information.  They are not, in and of themselves, capable of linking the issues discovered to the root cause.  And these root causes?  Yeah, they’re the things that create “risk”.  Not a threat, not a vulnerability, not the existence of an asset - the amount of risk that we have stems from our capability to manage it.

So Chris, I completely agree - but I wouldn’t give up yet.  There actually are a few of us who are focused on what you suggest:

Where to go from here: A fundamental revamp of how to deal with Risk. Where risk professionals focus on the treating the sickness and not the symptoms, and come up with some new success/actionable metrics.

Chris, there’s nothing I want to do more than that.

Blogger: Ramon Krikken

A seemingly innocent question on a mailing list - which I paraphrased for brevity - set in motion a series of events with dire consequences. The specific code, which was generating error messages in a certain software quality assurance tool, happened to be a critical part of the random number generator in a cryptographic library package. By removing this code, the strength of the cryptographic key material was reduced to a point where cracking the key would take minutes instead of decades. The unfortunate thing about cryptography and randomness is that good and bad can be virtually indistinguishable, and in this case the result still looked so random that the problem went unnoticed for about two years. The impact - needing to regenerate two years worth of key material, and casting doubt on encrypted communication and access performed with those keys - has understandably led to some vigorous discussion and finger pointing. Search Google for "debian openssl" for more discussions than I can link to.

The action - making a change without following a standardized process  - is certainly not unique to this situation, and "the system was slow so I turned off this feature", or "I just fiddled around with it and it just started working" are phrases all too commonly heard in many aspects of IT. Some might argue that a commercial development process would likely have prevented this occurrence, but to simply turn this into a comparison of open source and commercial development ignores some very important aspects. There are important lessons to be learned that could benefit any software development process, particularly when process parts are being adapted to encompass ever changing development and security landscapes. In the ideal world, source code would be based on well-documented requirements, consistently structured, well commented, and maintained by easy-to-reach teams that understand the code inside and out. The reality of dealing with the pressure of delivery deadlines, distributed development teams, and code written either long ago or by a third party can make coding a daunting task ... and quality assurance next to impossible, especially if breakdowns in process or communication occur. The myriad of testing tools, sometimes producing output that can run in the hundreds of pages, coupled with a lack of understanding about their testing coverage, doesn't make the task any easier.

Looking at how this specific event unfolded can lead us down many paths of analysis, all of which will provide valuable information in attempting to determine a root cause. Unfortunately - and this is something that is also not unique to any specific kind of environment - not all parties involved are neutral, and there can also be a tendency to fixate on symptoms rather than the cause. One reason for this may be the assumption that it's possible to fix specific process parts without necessarily re-evaluating the process as a whole; another is that risks and the resulting need for assurance, including process assurance, may be underestimated. Looking at the failures in the flaw finding process purportedly followed in the Therac 25 accidents it's easy to see how this can result in unacceptable consequences. And while likely not resulting in loss of life, the potential economic loss associated with a failure of a cryptographic module suggests that a critical security component can't be treated like just any other piece of software.

How ever unfortunate, this event presents a good opportunity to take a moment and look at our own development processes. Particularly as we start to embrace service orientation, where we loosely couple different business functions while relying on centralized, and often externally developed, security and reliability services, we increase the possibility of creating situations such as this. Using a risk-based process, and testing and revisiting the process itself to ensure it stays current, will be vital in providing appropriate levels of software, system, and information assurance. Building a high-assurance component using a low-assurance process just isn't worth the risk.

Blogger: Ramon Krikken

A seemingly innocent question on a mailing list - which I paraphrased for brevity - set in motion a series of events with dire consequences. The specific code, which was generating error messages in a certain software quality assurance tool, happened to be a critical part of the random number generator in a cryptographic library package. By removing this code, the strength of the cryptographic key material was reduced to a point where cracking the key would take minutes instead of decades. The unfortunate thing about cryptography and randomness is that good and bad can be virtually indistinguishable, and in this case the result still looked so random that the problem went unnoticed for about two years. The impact - needing to regenerate two years worth of key material, and casting doubt on encrypted communication and access performed with those keys - has understandably led to some vigorous discussion and finger pointing. Search Google for "debian openssl" for more discussions than I can link to.

The action - making a change without following a standardized process  - is certainly not unique to this situation, and "the system was slow so I turned off this feature", or "I just fiddled around with it and it just started working" are phrases all too commonly heard in many aspects of IT. Some might argue that a commercial development process would likely have prevented this occurrence, but to simply turn this into a comparison of open source and commercial development ignores some very important aspects. There are important lessons to be learned that could benefit any software development process, particularly when process parts are being adapted to encompass ever changing development and security landscapes. In the ideal world, source code would be based on well-documented requirements, consistently structured, well commented, and maintained by easy-to-reach teams that understand the code inside and out. The reality of dealing with the pressure of delivery deadlines, distributed development teams, and code written either long ago or by a third party can make coding a daunting task ... and quality assurance next to impossible, especially if breakdowns in process or communication occur. The myriad of testing tools, sometimes producing output that can run in the hundreds of pages, coupled with a lack of understanding about their testing coverage, doesn't make the task any easier.

Looking at how this specific event unfolded can lead us down many paths of analysis, all of which will provide valuable information in attempting to determine a root cause. Unfortunately - and this is something that is also not unique to any specific kind of environment - not all parties involved are neutral, and there can also be a tendency to fixate on symptoms rather than the cause. One reason for this may be the assumption that it's possible to fix specific process parts without necessarily re-evaluating the process as a whole; another is that risks and the resulting need for assurance, including process assurance, may be underestimated. Looking at the failures in the flaw finding process purportedly followed in the Therac 25 accidents it's easy to see how this can result in unacceptable consequences. And while likely not resulting in loss of life, the potential economic loss associated with a failure of a cryptographic module suggests that a critical security component can't be treated like just any other piece of software.

How ever unfortunate, this event presents a good opportunity to take a moment and look at our own development processes. Particularly as we start to embrace service orientation, where we loosely couple different business functions while relying on centralized, and often externally developed, security and reliability services, we increase the possibility of creating situations such as this. Using a risk-based process, and testing and revisiting the process itself to ensure it stays current, will be vital in providing appropriate levels of software, system, and information assurance. Building a high-assurance component using a low-assurance process just isn't worth the risk.

Extremely fair article on Sebastopol Wi-Fi networking health debate: The local paper manages to push the camel through the eye of the needle in presenting various aspects of the vote by the local council to rescind the gift of a local ISP to provide city-wide Wi-Fi. It neither ridicules the symptoms of people who describe themselves as electrosensitive, nor ignores the clinical research that shows such sensitivity to be unprovable, even as the symptoms are clearly manifest (just not correlated with EMF). The article notes that one radio host who speaks on health has his words carried by a station that is bumping more signal out across Sebastopol than any Wi-Fi network would. In a true Sonoma moment, however, the leading opponent to the city-wide network and the owner of the ISP cross paths in front of Whole Foods where high school students in favor of the network were gather signatures for a petition--and hugged. That kind of behavior is more of what we need: civility, understanding, and mutual working forward to improve everyone's health. More research? Sure. And more kindness, too.

Wired's Wi-Fi map: now, useful! My friend and colleague Cyrus Farivar spent weeks researching what municipal projects were proceeding, on hold, or dead across the U.S., and I wasn't very impressed by the way in which Wired presented this material in their print issue. But never fear! Online, paired with Google Maps, his research is tremendously accessible. It's now a few weeks out of date, but still useful for the scope and locations of projects. It makes me want to build an ongoing effort of the same kind!

Complimentary essay on Boston's pace: By not building fast, OpenAirBoston avoids the mistakes of other municipal networks. True. But in the end, they need to build something; they are only "behind" in the sense of not having put their neck out too far.