Some basic statements, the rest of this post will explain:

• C&A is a commodity market
• Security controls assessment is a commodity market
• PCI assessment is a commodity market
• Most MSSP (or rather, Security Device Management Service Providers) services are commodity markets

Now my boss said the first one to me about 4 months ago and it really needed some time for me to grasp the implications.  What we mean by “commodity market” is that since there isn’t really much of a difference between vendors, the vendors have to compete on having the lower price.

Now what the smart people will try to do is to take the commodity service and try to make it more of a boutique service by increasing the value.  Problem is that it only works if the customers play along and figure out how your service is different–usually what happens is you lose in the market simply because now you’re “too expensive”.

Where Boutique Sits by miss_rogue.

Since the security assessment world is a services business, the only way to compete in a commodity market is to pay your people less and try to charge more. But oh yeah, we compete on price, so that only leaves the paychecks as the way to keep the margin up.

Some ways that vendors will try to keep the assessment costs down:

• Hire cheaper people (yes, paper CISSPs)
• Try to reduce the engegement to a formula/methodlogy (ack, a checklist)
• It’s all about billability:  what percentage of your people’s time is not billable to clients? 
• Put people on assessments who have tangential skills just to keep them billable
• Use Cost-Plus-Margin or Time-Plus-Materials so that you can work more hours
• Use Firm-Fixed-Price contracts with highly reduced services ($150 PCI assessments)

Now inside Government contracting, there’s a fact that’s not known outside of the beltway:  your margins are fixed by the Government.  In other words, they only allow you to have around a 13-15% margin.  The way to make money is that the pie is a much bigger pie, even though you only get a small piece of it.  And yes, they do look at your accounting records and yes, there are loopholes, but for the most part, you can only collect this little margin.  If you stop and think about it, the Government almost forces the majority of its contractors into a commodity market.

Then we wonder why C&A engagements go so haywire…

The problem with commodity markets and vulnerability/risk/pen-test assessments is that your results, and by extension your ability to secure your data, are only as good as the skills and creativity of the people that the vendor sends.  Sounds like a problem?  It is.

So knowing this, how can you as the client get the most out of your service providers? This is a quick list:

• Every year (or every other), get an assessment from somebody who has a good reputation for being thorough (ie, a boutique)
• Be willing to pay more for services than the bottom of the market but be sure that you get quality people to go along with it, otherwise you’ve just added to the vendor’s margin with no real improvements to yourself
• Get assessments from multiple vendors across the span of a year or two–more eyes means different checklists
• Provide the assessors with your own checklists so you can steer them (tip from Dave Mortman)
• Self-identify vulnerabilities when appropriate (especially with vulnerabilities from previous assessments)
• Typical contracting fixes such as scope management, reviewing resumes of key personnel, etc
• Get lucky when the vendor hires really good people who don’t know how much they’re really worth (that was me 5 years ago)
• More than I’m sure will end up in the comments to this post  =)

And the final technique is that it’s all about what you do with the assessment results.  If you feed them into a mitigation plan (goviespeak: POA&M) and improve your security, it’s a win.

Bookmark to:

• State of Nature. State of Nature just means what the current state is.  There are two ISSA Journals on my desk right now - State of Nature statement.

• State of Knowledge:  Analysis derived from examination of State of Nature.  “One of these ISSA Journals has an article co-authored Donn Parker on ROI.  I’ve read it, and it makes some statements he regards as truth.  Looking at those, well, I know that risk is quantifiable, best practices have significant issues, and there are many, many other statements of authority in the article that I can refute on evidence.” - Analysis or State of Knowledge.

• State of Wisdom:  Synthesis from the analysis.  The “So” moment.  “So since there are many statements of authority made in the article that I can refute on evidence, I should be open but skeptical about whether the conclusions of this article are likely to have much value to me in my quest to understand the value of risk reducing investments.”  What I’ve synthesized from the quality of the article - State of Wisdom.

(Just a clue for our readers, anytime you read someone talk about risk and mention the term “actuarial” - be skeptical about the conclusions they have you draw from the statement using that word. 9 times out of 10 what I’ve read after someone says actuarial is made as authoritative but shows a level of ignorance on the subject.  If you really want to mess with them - say “Really! Well, tell me how you feel about the use of non-parametric Bayesian Methods” and wait… )

MMMMM-MMMMMMM CHECKLISTS!

So what about Checklists?  They’re worth discussing because we’re swamped by them!  Heck, we’ve got people in love with the idea of checklists of checklists and claiming GRC nirvana is not in the checklist itself, but in the mapping of checklists.

Here ya go:  Checklists have one of two uses -

First they can give us a path to accomplish something.  I make a checklist every morning I call a “Todo List”.   Useful Checklists could be as Curphey mentions - steps for operating machinery or performing a certain task (heck, scientific method could be said to be a checklist of steps in analysis).  Checklists are useful in this way because, well, we’re fallible, absent minded, and novices.  They serve to reduce some level of variability in a process.

Second, they can help us develop a State of Nature.  PCI or the ISO are very nice checklists that, once you’re done, certifies that you have the existence of a certain amount of control.  Again, this serves to reduce some level of variability, comparing you to a “best practice”.

And so…..

They are both useful in each use - as long as the limitations therein are understood!   And that’s where we get into trouble.  Too many times we believe that checklists are a State of Knowledge.  Checklists allow for some limited analysis, just like the use of ordinal numbers to describe “risk” - they only serve to identify some level of variability, nothing more.

But outside of that they usually offer us no analytical function at all, they cannot provide a State of Knowledge and therefore, more succinctly, Checklists are dumb.

As slightly paranoid, skeptical and jaded risk management professionals, we know this to be true.  A PCI compliant company may or may not be at all “secure” or “risk-free” or even “risk-reduced”.  That’s an aspect of analysis that the checklist is some prior information for, but not nearly all the information we need for an analysis of risk or even a statement about the ability to control or resist.  We know an ISO certified organization did what they claim they do enough to at least fool an auditor once, but cannot arrive at any other State of Knowledge without more effort.

Make no mistake, the checklists we commonly deal with provide a very, very limited State of Knowledge.  Only analysis (with rigor and testing) will provide that.  And note that a State of Wisdom (what we’re really after, after all) is predicated on a strong State of Knowledge.

WHAT ARE YOU MANAGING TOWARDS, REDUX
So if checklists only provide a State of Nature, and are incapable of really giving us Knowledge or Wisdom - then let me encourage you to think about the amount of time you spend just getting a certain State of Nature and the relative return on that investment vs. the amount of time you spend in analysis and synthesis.  Is your time best spent mapping checklist to checklist - or is it better spent developing the analytics that allow us to synthesize wisdom?

AMAZE AND CONFUSE YOUR FRIENDS AUDITORS
Let me finish by encouraging you to have a frank discussion with those who perform your audit function.  You must really pin them down if they are out to give you any analysis at all - and when/if they do provide analysis - press them on what rigor they use to create a State of Nature, and then the means by which they create a State of Knowledge (that belief statement based on the State of Nature they see).

Well, today it’s the ISC2 blog talking about FISMA.

So why is it that nobody addresses the huge pink and chartreuse elephant in the room?  The problem is not the metrics, as flawed as they might be.  The problem is not identifying a security baseline, even though that makes sense to have.  The problem is not demonstrating Return on Security Investment (as flawed as  the concept is, and no, I don’t want to debate whether it’s a valid concept, even though we all know it’s not) even though good CISOs try to do that as internal marketing to their management.

This is the primary problem for the Government when it comes to security:  due to the scale of the Federal Government, we do not have enough skilled security people to go around.  Almost all of our governance models are designed around this flaw:

• Catalog of controls to standardize
• Checklists so that less-skilled assessors can
• Varying degrees of automation
• Prioritization of security practitioners’ time

This is why I’m adding “Fast Food Franchises” to the list of models that large-scale security can draw from.  =)  More to come on this topic once I sort out the ideas.

McDonald’s Checklist photo by myuibe

Bookmark to:

1.) Risk (and Risk Management) deals in open-ended problems.  Checklists and Best Practices, while they have some value, aren’t an end in and of themselves.  They will always fail because they are closed-ended.

2.)  Because we deal in open-ended problems,  if we are to be successful we must become subject matter experts and develop critical thinking skills.

It's useful if we take a moment and consider the definition of the auditing function. Here's mine:

Audits help us ensure that we are following our own policies. Audits measure the current state, compare the results against what the state should be, and show where we are out of compliance. Essentially, audits help us know that we are indeed doing what we say we're doing.

Audits are the natural outcomes of implementing good policies and following effective procedures. It makes no sense to spend time developing policies and without having some mechanism to measure compliance. That's the role of the auditing function -- to measure compliance. If we all agree that policies are good, then we should all agree that checking up on ourselves is also good.

So, then, who should conduct the audits? For comparison, let's examine a typical software development department. Here at Microsoft, such departments are composed of four over-arching roles:

• program management
• product management
• software development
• software test

Why this way? Consider the first two. We don't have "project managers" at Microsoft because project management incorporates two conflicting goals: managing people, schedules, and budgets (program management) versus incorporating customer requirements and creating new markets (product management). Program management optimizes resources while product management optimizes features. Rather than shoulder that inherent conflict onto a single person and expect them to deal with it without going completely bonkers, we have two roles, with different people. People skilled in each area negotiate with each other and come to an agreement about what's best both for Microsoft and for our customers.

Similar thinking exists for the second pair of roles. Developers strive to write high-quality code, and even do some testing along the way. But because no one's perfect, all code has some mistakes; it's valuable to have other people bang hard on the code, abuse it almost, to find and squash more bugs. Often, even the best developers are embedded so deeply in their own code that some bugs escape them. Developers rightly concern themselves with creating code that works and provides proper output. Testers figure out how to purposefully break software and discover code vulnerabilities. These are different skill sets, and using different people results in higher quality software.

We can apply the same logic to the information security department. How about these four roles:

• security standards
• security alignment
• security operations
• security auditing

The security standards group defines an organization's security architecture, creates policies and procedures, and ultimately takes responsibility for stewarding the integrity of the organization's information assets. The security alignment group spends time understanding the needs and drivers of the various business units, and advocates the business units' positions in meetings with the security standards group. Like in the software development model, having different folks negotiate together about standards and alignment helps ensure that business needs are met while also ensuring that the business is able to rely on information that's kept secure.

Remember: the primary purpose of information security is risk management. The standards folk know all about the bad guys and their techniques, and build up knowledge about which threats create risk for the organization. The alignment folk understand, through their constant interaction with people in the business units, all about business risk and get a feel for the business's risk tolerance -- that is, the level and kinds of risk that matter or don't matter. Together, the security standards and the security alignment folk can develop a security posture that allows the business to remain agile while also addressing the risks that make sense.

(Notice that I haven't indicated where, exactly, the alignment folk sit within the organization. They might be part of the security department, or they might be part of the individual business units. A case could be made for either choice; however, except for very large organizations, the alignment role probably isn't full-time. This leans the role toward sitting in the business units.)

Day-to-day work becomes the responsibility of those in security operations. They create standard configurations, perform installs and updates, monitor traffic, and respond to incidents. Ideally, policies and procedures guide all of these activities. But having policies and procedures isn't enough: we must also have a way to measure conformance. And that's the role of security auditing. Security auditors compare a system's current configuration to what it should be, based on the policy. Where systems are out of compliance, the auditor works with operations folk to understand the reasons, without engaging in blame-storming or launching personal attacks (this goes for operations folk, too). Most of the time, it's simply a mistake; here, auditors are like software testers, uncovering configuration vulnerabilities (bugs) that otherwise might be overlooked by operations and thus exploited by attackers.

Now you auditors out there, this doesn't mean that your role is simply that of checklist slave. Especially if your checklist is something you downloaded from the Internet. Remember: these checklists are only guidance, good ideas written by a person (or a committee) based on that person's risk tolerance. Effective auditors develop relationships with people in the other three groups: standards, alignment, and operations. Effective auditors take the time to learn the security landscape, how attackers operate, where vulnerabilities lie, and which threats matter. Really effective auditors learn how to do penetration testing, thus uncovering not only code and configuration vulnerabilities but also circumvention vulnerabilities through social engineering. By doing this, effective auditors remove the "us versus them" stigma often associated with auditing and truly become part of the security team, all working together to protect the organization's information assets.

(Notice that, as with the alignment group, I haven't indicated organizationally where the audit group should sit. I do, however, have a strong opinion on this: the management chains of the audit group and the operations group must be different. The people conducting audits shouldn't work for those who have a stake in an audit's outcome. To do so would create unavoidable and unrecoverable conflicts of interest.)

I'm sure there's more to the topic of organizing a security department. What do you think of this approach? Do you like the idea of dividing conflicting roles into different groups, then structuring them to work together to achieve realistic and useful outcomes? I don't suspect I've necessarily invented anything new here, but maybe just used a few new words -- such as "security alignment" -- and thought out loud about some of the tension that exists within the standards/alignment and operations/audit pairs. (Oh, and I got to write about my code/configuration/circumvention vulnerability triple again, heh.) Please tell me your thoughts. Maybe there's an entire white paper here, possibly even a TechEd presentation. Maybe someday we should offer a "TechManagementEd" conference!

• Document how you handle authentication. if different from standard X, get a security reviews.
• Document how you're handing input filtering. If not the standard library with declarative syntax, document and get a security review.....

1. Metrics
2. Processes

"At the current rate, it will never happen,” he said, as monitors beeped in the background. “The fundamental problem with the quality of American medicine is that we’ve failed to view delivery of health care as a science. The tasks of medical science fall into three buckets. One is understanding disease biology. One is finding effective therapies. And one is insuring those therapies are delivered effectively. That third bucket has been almost totally ignored by research funders, government, and academia. It’s viewed as the art of medicine. That’s a mistake, a huge mistake. And from a taxpayer’s perspective it’s outrageous.” We have a thirty-billion-dollar-a-year National Institutes of Health, he pointed out, which has been a remarkable powerhouse of discovery. But we have no billion-dollar National Institute of Health Care Delivery studying how best to incorporate those discoveries into daily practice.