I’d love to have another blog even more specific - “Ok, that Assurance is Evidence Of What, Exactly?

Today he has a great article called:

What’s the matter with Risk Management?

And “in short, it’s everything.” It pretty much sums up why I had to grow to re-evaluate how our industry does risk, risk management, approaches controls & vulnerability and find a new way.   A couple of things jump out at me in reading Chris’ article:

1.)  Just because that Deming cycle sucks and is full of unknowns doesn’t mean “risk” doesn’t exist, nor that it isn’t of primary importance. Nor does it mean that in the absence of model & methodology, we won’t be “doing” risk analysis anyway - just in an ad hoc method and completely from “the gut”.

Our industry calls these unstructured risk analysis “Best Practices”, as it’s an easy and convenient way of sweeping the unknowns under the rug of bureaucracy and enforcing it via peer pressure.

2.)  What this “suckiness” does mean is that your model and methodology aren’t helping you. As Chris intimates, there is too much uncertainty in the inputs for his model (they are, in the language of Bayesians - too subjective to be useful priors).

Take for example how we might be approaching the “controls” part of our analysis.  Chris writes:

“2. What are the controls that we have to employ?
800-53, ISO 27001, PCI, etc.

Still kinda good, but we basically know that ISO is relatively voluntary and NIST supplies a control catalog and not policies. So here we have to take the control catalog, and mash our policies into it.”

I wouldn’t call this “kinda good” at all :)  These control catalogs only provide a hierarchy within which to look for evidence of  our ability to resist an attacker.  They are incapable of making any claim about the effectiveness of the controls when they are operated at 100% efficiency, or more importantly, what % efficiency our specific organization operates at.

Let’s use Chris Hayes’ Initech as our fictional example.

Initech has a control (a back door on a loading dock).  Now the locks on the door are 100% capable of locking the door.  This is different than saying that they are capable of frustrating all but the top 5% of lockpicking burgalars.  It is also diffferent than saying that in a sample of several “walk around audits” the doors are left open 20% of the time (they are not in compliance with policy 100% of the time).  Even worse, that 80% of the time the door is not propped open?  Yeah, tailgating is a known issue.

So we have several different variables here that we need to account for (and it’s just a door).  But the analogy stands that most “risk management” methodologies are “We have a door, yes/no?” And most GRC platforms, when asked for their “opinion” will simply say “door is needed” or, even worse, “a door policy is needed”.

3.)  Criticality and the Source of Value is all messed up in these Risk Management models.

Chris writes:

Someone wants me to tell them which boxes are more critical than others. This is mainly because of budgetary or operational reasons. To which I usually say “All of them, it is a system after all”.

This literally made me laugh out loud.  And this sort of “rate the firewall as Risk = 500 but rate the actual business application as Risk = 157″ thing is also endemic.  Now Chris is very smart here.  He correctly identifies that the value is tied to the business process the systems support, and not to a specific box.  Oh, we scan at the specific box level - but because of the nature of systemic failures - all the boxes in the process are inexorably interrelated.

One of the reasons I really like FAIR is that the losses are quantified (or qualified) based not on some amorphous value of the box or the process itself, but losses are linked to the actions that the threat will take. Take systems in a highly regulated industries as an example.  Usually the most probable losses aren’t due to system compromise per se, but in the disclosure the compromise causes (regulators are a threat source, after all).  But many “risk management” methodologies will say “online banking is worth $2 billion, the value of the systems is therefore $2 billion”.  And suddenly we’re telling executive management that there’s a 60% probability that they’ll lose $2 billion.

4.)  If the primary source of prior information for your “risk management” methodology is a vulnerability scanner - you’re doing it wrong.  Chris writes:

So we ran a scan and now we have a report. A snapshot in time to make all decisions. Where did these vulnerability ratings come from? Do I even know if my system is at risk? What if I spend my time on vulnerabilities that have no threat?

So first, my thoughts are that actual “vulnerability” must be a comparison of the force a threat can apply, and our ability to resist that force (this is a probability statement, btw).

Changing your thinking about vulnerability now helps us understand the problem in several new ways.  First, you can start to divorce yourself from the scanner.  After all, the scanner is simply providing you with current state information that is usually just relevant variance from policy. It doesn’t really tell you about real “weakness in a system” because the system is an interrelated mess of people, processes and IT assets.

5.)  Finally, most “risk management” approaches just *don’t* do a good job of helping us understand the how’s and why’s of managing risk. In the past, I’ve referred to these standards as really being “issue management” because they are at their heart, an act of discovery - a formal process around gathering prior information.  They are not, in and of themselves, capable of linking the issues discovered to the root cause.  And these root causes?  Yeah, they’re the things that create “risk”.  Not a threat, not a vulnerability, not the existence of an asset - the amount of risk that we have stems from our capability to manage it.

So Chris, I completely agree - but I wouldn’t give up yet.  There actually are a few of us who are focused on what you suggest:

Where to go from here: A fundamental revamp of how to deal with Risk. Where risk professionals focus on the treating the sickness and not the symptoms, and come up with some new success/actionable metrics.

Chris, there’s nothing I want to do more than that.

the balaclava "could be used to conceal someone's identity or could be used in the course of a criminal act".

Don't they realize that balaclavas are for sale everywhere in the UK? Or that scarves, hoods, handkerchiefs, and dark glasses could also be used to conceal someone's identity?

The game sounds like it could be fun, though:

Each player starts as an empire filled with good intentions and a determination to liberate the world from terrorists and from each other.

Then the reality of world politics kicks and terrorist states emerge.

Andrew said: "The terrorists can win and quite often do and it's global anarchy. It sums up the randomness of geo-politics pretty well."

In their cardboard version of realpolitik George Bush's "Axis of Evil" is reduced to a spinner in the middle of the board, which determines which player is designated a terrorist state.

That person then has to wear a balaclava (included in the box set) with the word "Evil" stitched on to it.

Buy yours here; I first blogged about it in 2006.

Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software.  He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.”  I think that sums up the problem pretty well.  Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think.  The same thinking is used by people who install social networking applets and give them access to their personal data.  Someone, somewhere is taking care of the software security so I don’t have to.  It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives.  I’m safe in a plane or car because the government is looking out for me with standards and testing requirements.  I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space.  I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process.  I ask them if they have made security quality part of the requirements of the project and they say no.  Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers.  They have no evidence.  Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for.  So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems.  The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source.  Surely the developers writing the software are trained in secure software engineering.  Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers.  The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog.  He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment.  Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software. He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.” I think that sums up the problem pretty well. Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think. The same thinking is used by people who install social networking applets and give them access to their personal data. Someone, somewhere is taking care of the software security so I don’t have to. It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives. I’m safe in a plane or car because the government is looking out for me with standards and testing requirements. I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space. I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process. I ask them if they have made security quality part of the requirements of the project and they say no. Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers. They have no evidence. Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for. So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems. The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source. Surely the developers writing the software are trained in secure software engineering. Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers. The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog. He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment. Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

Dear network vendor,

As someone that is forced to configure and implement security on your hardware, I would greatly appreciate stable code and properly functioning features. Unfortunately, I cannot always choose the hardware my customers are using in their infrastructure. However, if you would like for me to recommend they continue purchasing and using it, then the product must demonstrate to me that it is: capable, reliable, predictable and well-documented. If your product is not meeting these requirements, I’m forced to recommend other solutions to your (current) customer.

Stable Code. If I have to spend 2-6 hours per implementation working through your product’s bugs, and then must either spend time on a support call or spend time getting packet captures to prove to you it’s not working, I am not a happy camper because you’re slowing down my progress. Your customer is not happy because they’re paying for that time and I’m not cheap.

Features. Don’t publish in technical documentation that your product, or code can do something, only for me to find out later that it cannot. On-site in the middle of an implementation is not the time to architect Plan B. Let me know before, either through technical docs, white papers, best practices or release notes. I do read those. If you want to bend the truth, do it the marketing fluff, not my technical documents.

Documentation. If your product does do what you say it does, then please do document and explain the concepts and procedures. Examples are good, but explanations are mandatory. A correct CLI reference is always lovely as well. If there are got’chas or tricks, please also document those. Again, white papers or release notes are fine. Having to track down the one security engineer from your company that holds the magic key is not practical, nor scalable. Plus, he may be on vacation during my install, which would make me irate.

Support. If your product is not functioning or performing as expected, do NOT expect your customers to have a current maintenance contract to address a known issue or bug (or an un-known issue or bug for that matter). If they found a bug for you, you should probably give them a maintenance contract for a year… or two. If you don’t let us call support, I will find one of your pre-sales engineers and we will use him or her for post-sales support, which is not what you want them to do. But that’s your problem, not mine.

I believe that sums up the major issues. Specifically, I am interested in security, RADIUS, SSH, SNMP, DHCP and 802.1X functions. Before you add another bell or tweak another whistle, please make what you have works… consistently. That should be first, so it’s my Feature Request #1.

Respectfully,

jj

# # #