In On EP and Analytics, good friend and respected colleague Opher Etzion applies the well known metaphor of the big elephant to describe how, if you are observing certain specific domains of a subject, like fraud detection, then your view of the whole elephant is biased by your lack of perspective of entire big elephant.

I am pleased that dear Opher continues to use this metaphor in counterpoint because the same metaphor can be used to describe the carefully selected group of vendors that have banded together to called themselves CEP Vendors.  This group, many founding members of the EPTS, have formed a merry band of well-intended event processing “specialists” and the same lovely elephant causes this group of bonded colleagues to make elephant-blinded statements, as Opher has made in his quoted post:

“Currently most CEP applications do not require analytics.” 

The reason, I believe, that Opher makes the statement above is because the group of software vendors calling themselves “CEP vendors” represent a very small part of the overall event processing elephant;  and hence, since these self-described CEP applications appear to require very little or no analytics, then, by the same logic, CEP requires no analytics. 

(I should outline the boolean logic in a future post!)

For example, one friend and colleague in Thailand is the CTO of True Internet, a leading telecommunications, voice, Video and Internet service provider in Thailand.   True processes myriad events on their network using a dynamic, self-learning neural networking technology.    The US company providing this very clever and highly recommended event processing application do not call themselves a “CEP vendor”; however, they process complex events better and more interesting than the band of merry self-described “CEP players”.

Again,  visualize the gentle giant elephant metaphor that Opher likes to use as a basis for his comments in CEP counterpoint.

When folks define the term “complex event processing” to match a technology marketing campaign that is primarily driven by software running rules against time-series data streaming in a sliding-time windows, and then go on to take the same software capabilities and apply these capabilities to problems that are suitable for that domain, then you match Opher’s elegant description of “a small view of the overall elephant”.

The fact of the matter is that the overall domain of event processing is at least three orders of magnitude larger than the combined annual revenue of the self-described companies marketing what they call “CEP engines.”  The very large “rest of the big elephant” is doing what is also “complex event processing” in everyday operations that are somehow overlooked in ”other” analysis and counterplay.

Therefore,  I kindly remain unmoved of my view  that the self-described CEP community, as currently organized, is not immune to counterpoint using the same gentle giant elephant metaphor.  I like this metaphor and hope well-respected colleagues will continue to use this metaphor; because we can easily apply this elegant manner of discussion to explain why the current group of self-described CEP vendors are, in a manner of speaking, selling Capital Market Snake Oil because they are making outrageous claims about the capabilities of their products, as if they can solve the entire ”elephant” of event processing problems.   Recently, in this article, CEP was positioned as a technology to mitigate against corporate megadisasters like the subprime meltdown.

Advice:  Tone down the hype.

Furthermore, the noise in the counter arguments marginalize most of the real event processing challenges faced by customers.

In consistant and well respected rebuttal, Opher likes to use the “glass half-full, half-empty” metaphor.   Opher’s point is a valid attempt to paint my operational realism as “half empty” negativism; while at the same time positioning the promotion of the (narrow) event processing capabilities of the self-described CEP rules community as “half-full” thinking. 

For the record, I do see my worldview as “half full” or “half empty”; but an unbiased pragmatic view based on day-to-day interaction with customers with what they would call “complex event processing” problems. 

These same customers would fall over laughing if we tried to bolt one of these rule-based, time-series streaming data processing engines on their network and told them they can detect anything other than trival business events, business opportunities and threats, in near real-time. 

Is it “half empty” thinking to caution people that a “glass” of software that is being touted as the answer to a wide range of complex (even going so far in a recent news article to imply CEP would have magically stopped the subprime crisis!) tangible business problems is not really as that it is hyped to be?  

If so, then I plead guilty to honesty and realism, with the added offense of a sense of fiscal responsibility to customers and end users.

I am pleased that dear Opher continues to use this metaphor in counterpoint because the same metaphor can be used to describe the carefully selected group of vendors that have banded together to called themselves CEP Vendors.  This group, many founding members of the EPTS, have formed a merry band of well-intended event processing “specialists” and the same lovely elephant causes this group of bonded colleagues to make elephant-blinded statements, as Opher has made in his quoted post:

“Currently most CEP applications do not require analytics.” 

The reason, I believe, that Opher makes the statement above is because the group of software vendors calling themselves “CEP vendors” represent a very small part of the overall event processing elephant;  and hence, since these self-described CEP applications appear to require very little or no analytics, then, by the same logic, CEP requires no analytics. 

(I should outline the boolean logic in a future post!)

For example, one friend and colleague in Thailand is the CTO of True Internet, a leading telecommunications, voice, Video and Internet service provider in Thailand.   True processes myriad events on their network using a dynamic, self-learning neural networking technology.    The US company providing this very clever and highly recommended event processing application does not call themselves a “CEP vendor”; however, they process complex events better and more interesting than the band of merry self-described “CEP players”.

Again,  visualize the gentle giant elephant metaphor that Opher likes to use as a basis for his comments in CEP counterpoint.

When folks define the term “complex event processing” to match a technology marketing campaign that is primarily driven by software running rules against time-series data streaming in a sliding-time windows, and then go on to take the same software capabilities and apply these capabilities to problems that are suitable for that domain, then you match Opher’s elegant description of “a small view of the overall elephant”.

The fact of the matter is that the overall domain of event processing is at least two orders of magnitude larger (maybe more) than the combined annual revenue of the self-described companies marketing what they call “CEP engines.”  The very large “rest of the big elephant” is doing what is also “complex event processing” in everyday operations that are somehow overlooked in ”other” analysis and counterplay.

Therefore,  I kindly remain unmoved from my view  that the self-described CEP community, as currently organized, is not immune to counterpoint using the same gentle giant elephant metaphor.  I like this metaphor and hope well-respected colleagues will continue to use this metaphor; because we can easily apply this elegant manner of discussion to explain why the current group of self-described CEP vendors are, in a manner of speaking, selling Capital Market Snake Oil because they are making outrageous claims about the capabilities of their products, as if they can solve the entire ”elephant” of event processing problems.   Recently, in this article, CEP was positioned as a technology to mitigate against corporate megadisasters like the subprime meltdown.

Advice:  Tone down the hype.

Furthermore, the noise in the counter arguments marginalize most of the real event processing challenges faced by customers.

In consistant and well respected rebuttal, Opher likes to use the “glass half-full, half-empty” metaphor.   Opher’s point is a valid attempt to paint my operational realism as “half empty” negativism; while at the same time positioning the promotion of the (narrow) event processing capabilities of the self-described CEP rules community as “half-full” thinking. 

For the record, I do see my worldview as “half full” or “half empty”; but an unbiased pragmatic view based on day-to-day interaction with customers with what they would call “complex event processing” problems. 

These same customers would fall over laughing if we tried to bolt one of these rule-based, time-series streaming data processing engines on their network and told them they can detect anything other than trival business events, business opportunities and threats, in near real-time. 

Is it “half empty” thinking to caution people that a “glass” of software that is being touted as the answer to a wide range of complex (even going so far in a recent news article to imply CEP would have magically stopped the subprime crisis!) tangible business problems is not really as that it is hyped to be?  

If so, then I plead guilty to honesty and realism, with the added offense of a sense of fiscal responsibility to customers and end users.

Don’t get me wrong.  I greatly admire the work Mike does and wish he and his book had been around when I started out as a CISO.  Would have saved me significant pain and suffering.  On the other hand, if I’d had Mike’s P-CSO I might have become complacent and ended up believing that’s all there was to being a CISO.  Not that I think Mike is advocating complacency — he’s not.  I also don’t think he discounts risk analysis concepts.  He’s simply focused on helping that component of our profession who’s just getting started or who faces other practical constraints in dealing with our very complex problem space.  His is a necessary and highly valuable contribution, and he provides it in an entertaining way that’s too rare.

Let me set this discussion in a medical analogy context.  If I was in the middle of nowhere or didn’t have the resources for a physician, then a medic who’s skilled in lifesaving basics would do just fine.  However, if the situation called for a deeper understanding of the complex, sometime subtle health considerations, then I’d prefer a physician.  Someone who didn’t say;  “Boy, this anatomy and physiology stuff is complicated.  I’m just going to stick with ‘The hip bone is connected to the back bone…’”   My physician may, of course, choose to follow a pragmatic, commonly-used course of treatment, but they’d be able to do so with a deeper understanding of the problem space, greater (but not perfect) certainty that the course of treatment would work, and a better ability to explain to me, the patient, why I had to swallow this bitter pill, undergo the knife, or have this long tube snaked into one of my orifices.  

Yes, I realize that physicians sometimes get it wrong, sometimes get wrapped up in fancy and even unnecessary procedures, and can drive up costs.  That’s just as true as what can happen at the other end of the spectrum — the shaman who operates entirely by superstition, faith, FUD, and intuition.  The point is, there’s absolutely a need for both medics and physicians (and levels in between).  We, as professionals, can choose where we want to be within that continuum.  With this in mind, a few things to consider are:

• In the heat of battle, when resources are limited, or when it just makes sense, physicians always have the option of behaving as medics and sticking with the bare essentials (the reverse isn’t true).  In fact, the best physicians I’ve encountered are pragmatic in their approach but have the deeper knowledge to leverage when need arises
• Medics might effectively deal with 80+% of our problems, but that remaining ~20% can be critical 
• A person can start out as a medic and then become a physician later, as need and resources dictate  
• Physicians tend to be paid more

Bottom line — knowledge and understanding are never a bad thing, but it requires extra effort to acquire them.  And, as Mike points out, the simple approach is often good enough and may be all we can hope for given our individual circumstances.  For myself though, I prefer a deeper understanding of our complex problem space.  I want to be able to answer the hard questions about why and how.  But that’s just me.

BTW - I was amused at Mike’s characterization of risk analysis as Black Magic, as this phrase would also have been used in the past to describe medical and scientific concepts/practices we take for granted today.  

People pretty quickly realized that plain HTML was not enough, so developers invented CGI/PERL for more dynamic sites. Once they wanted to scale and pool they built out ASP and JSP, then to deliver middle tier components they developed EJB, J2EE, and DCOM. After that there were a lot of heterogeneous systems that needed to talk to each other so SOAP and XML came along to address that. This path diverged into ultra-simple (REST) and more powerful but baroque (SOA), and finally, the user side got some love with Web 2.0 technologies. That's a heck of a lot of engineering and innovation by the software development community for plus or minus 8 years.

Now lets' check in with the developer's brethren over in information security. Well, once the web came along the information security community quickly realized that network address translation was going to be important, and further that encrypting the communication channel between the browser and the web server was also crucial. And then, they addressed all the security issues ASP, JSP, EJB, J2EE, DCOM, SOAP, XML, REST, SOA, and Web 2.0 with....umm...more of the same!

That's a pretty poor showing for innovation considering the enterprise investment into information security. Sure the software developers' have a bigger budget, but come on infosec - show some pride!

Infosec types like to throw developers under the bus for security issues, but its a collective failure. Sure developers need to learn more about secure coding, but as the table above shows - security is not keeping pace, and the gap is getting bigger.

Here is another dimension to the problem - attackers *do* evolve. The new technologies provide far greater attack surface (data, method and channels) for the attacker's to exploit and/or launch attacks from.

Because the defenses have not evolved its a simple evolutionary adaptation for attackers to go around or through the 1995 defenses. Its not about SOAP going through the firewall, its about never bothering to secure the apps and the data. Its like saying to your opponent, remember the how the Detroit Lions played defense in a certain game in 1995, we were just going to do that.

So with the software developer's latest evolution we get Mr. O'Reilly's famous Web 2.0 meme map

but where is the co-evolution in infosec? there is non. There is co-evolution in the attacker space. here is a sample web 2.0 attacker meme map

So the firewall offers great protection if your adversary is using Visio, but otherwise its mostly useless.

So we would want to see two things happen - developers start writing more high assurance code and second - infosec needs to evolve its security services to form fit to that which they are protecting. Hint - it ain't a Visio diagram.

The thing is - we are getting getter tools. Static analysis is a very powerful tool to improve your software security from a bottom up perspective and it can scale. These tools continue to get better. We are are getting better standards - WS-Security, WS-Trust, and company enable fundamentally new security architectures. And we're getting better primitives, especially in the identity space - SAML, Cardspace, and friends will one day let us live in a world where users are not typing username and password into a web browser to do online banking.

So maybe the innovation tide is turning, but there is a lot of ground to catch up, infosec about a decade behind the developers and probably close to that far behind the attackers. Its going to take something special to catch up, but is there any other way? I think a big part of catching up is putting together a realistic pragmatic blueprint to evolve your security architecture - a roadmap that addresses your people, processes, and technology. There are standards, primitives, and tools to leverage, but by themselves they are just pieces, they have to be brought together into a cohesive design. Its not an overnight thing to realize this, but the point is for infosec to *begin* the evolutionary process. Now. For real use cases. Using the security protocols, mechanisms, and skills we have available now.

The Road goes ever on and on,

Down from the door where it began.

Now far ahead the Road has gone,

And I must follow, if I can,

Pursuing it with eager feet,

Until it joins some larger way

Where many paths and errands meet.

And whither then? I cannot say.

-J.R.R. Tolkien,The Hobbit

As I touched on in my December posting on Common Criteria, and as Michael Howard discussed in his post on security metrics, trying to objectively quantify and measure “How secure is secure” is far more difficult than one might think. I’d like to share my perspective that there are two “dimensions” useful to consider when characterizing software security metrics: security functional requirements and security engineering quality requirements. While the SDL is focused primarily (but not exclusively) on the latter, both are ultimately important when assessing the security of a given bit of software. However, for reasons I’ll elaborate on below, the SDL does focus on trying to prevent the most common causes of vulnerabilities today and hence looking at the ways in which Microsoft tracks and measures individual products teams’ compliance with SDL requirements offers some interesting fodder for the security metrics debate. I’m not offering a complete solution, but am sharing our experience at Microsoft with measuring how development teams actually follow the SDL. It’s helped us deliver more secure software, and sharing this will hopefully help others as well as putting more data on the table for consideration when discussing security metrics.

Putting aside computer security for just a moment, it’s interesting to look at other ways in which we attempt to measure security in our society. Padlocks offer security protections, and organizations such as the American Standard for Testing and Materials (ASTM) provide standards like F883-04 Standard Performance Specification for Padlocks that characterize padlock security ratings. Prisons provide security protections as well. Prisoners reside in different facilities that vary by security level. The US Bureau of Prisons uses a numbered scale from one to six to represent the security level. Both of these examples are similar in that the threats and risks each of them must protect against are reasonably well understood and relatively static (meaning the threats don’t change much over time). Computer security is still evolving with new classes of attacks still being discovered, and while hackers understand how to exploit known types of vulnerabilities – software developers are still catching up in learning how to modify engineering practices to be resilient against both new and old types of attacks. Hence, metrics are more challenging for computer security.

Several attempts have been made by governments to come up with a security rating system similar to the examples listed above. In the 1980’s, the US Department of Defense created the “Trusted Computer System Evaluation Criteria (TCSEC)” that tried to establish a standard for measure operating system security. The “Orange Book” offered a relatively simple system for assigning “score” summarized below:

D (Minimal Protection)

C (Discretionary Protection)

C1: Discretionary Security Protection

C2: Controlled Access Protection

B (Mandatory Protection)

B1: Labeled Security Protection

B2: Structured Protection

B3: Security Domains

A (Verified Protection)

A1: Verified Design

In the 1990’s, the US and other nations combined their efforts to create an international security standard for software known as the Common Criteria (ISO 15408). Common Criteria also has a rating system that scores products with “evaluation assurance levels” (EALs):

EAL 1: Functionally Tested

EAL 2: Structurally Tested 

EAL 3: Methodically Tested and Checked 

EAL 4: Methodically Designed, Tested, and Reviewed 

EAL 5: Semi-formally Designed and Tested 

EAL 6: Semi-formally Verified Design and Tested 

EAL 7: Formally Verified Design and Tested 

Both TCSEC and Common Criteria (CC) are primarily focused on “security functional requirements” (as called out earlier, distinct from “security engineering quality requirements”). The EALs reflect the amount of rigor and attention to claimed security functional requirements a developer applied while creating a product. Furthermore, the EALs also reflect increasing levels of effort and resources necessary by anyone reviewing a product in order to evaluate the product’s claimed security functional requirements. However, EAL ratings for commercial products have historically not correlated with the number of vulnerabilities found in commercial products after release. As I discussed in my December posting on Common Criteria, this is because CC is primarily focused on “security functional requirements” and fails to adequately address “security engineering quality requirements”.  This leads a question on how to measure those aspects of software security that earlier efforts have been unable to successfully address. 

Microsoft has been releasing security bulletins since 1999. Based on some informal analysis that members of our organization have done, we believe well over 50% of *all* security bulletins have resulted from implementation vulnerabilities and by some estimates as high as 70-80%. (Some cases are questionable and we debate if they are truly “implementation issues” vs. “design issues” – hence this metric isn’t precise, but still useful). I have also heard similar ratios described in casual discussions with other software developers. In other words, most vulnerabilities can be addressed by the “security engineering quality requirements” described via SDL. This is not to say that “security functional requirements” are unimportant or that SDL ignores secure design (as Adam has described in his threat modeling series), but rather that it is not where vulnerabilities are being most frequently encountered. With SDL, we adopt a pragmatic approach in looking at identifying the root causes of security vulnerabilities, and trying to prevent those root causes from reoccurring. The challenge lies in how we actually validate that development teams are indeed adopting and executing whatever changes SDL requires in engineering (either in terms of process or tools). Process changes are often difficult to quantify, as we must rely upon development teams truthfully attesting they have followed the process. As long as development teams believe the process results in better code, they generally will adopt and follow such practices. Tool usage becomes more interesting and valuable in that using tools becomes a vehicle for objectively and independently verifying if code satisfies requirements or not. But that is just the tip of the iceberg…

As I said above in my comments on EALs, the amount of time required by anyone reviewing a product to assess “security” is relevant since security review can be a very time and resource intensive activity. However, running static code analysis tools, verifying build tools and switches, searching for banned APIs, and recording the output of other tools that inspect code and/or binaries for potential implementation vulnerabilities is a key element in how we approach the challenge of trying to measure compliance with SDL requirements from product groups at Microsoft today. While not every technique required by SDL has a corresponding tool, we try to provide both tools and automation if and wherever possible. There is still much work to be done in terms of standardizing tool output formats and creating automation to assess tool output. However, these “grass roots” metrics derive from practical experience of changing engineering requirements based on actual vulnerabilities. We look objectively at what is causing vulnerabilities, and target solutions to address the root causes of those issues. As the saying goes, “If it hurts when you do that, stop doing that”. If what we have done in the past has hurt our customers by creating vulnerabilities requiring security bulletins, we want to stop doing that. J

The challenge in using a plethora of individual detailed metrics such as I describe above (that we do internally at Microsoft for measuring SDL compliance), is that they don’t roll up into a nice aggregate score that customers can easily understand.  However, they have translated into reduced numbers of vulnerabilities as Michael Howard wrote a few weeks ago. Coupling these types of scores with assessment of compliance with “security functional requirements” might be the basis for coming up with a metric that is useful to customers, both in the government and private sector.

What do you think?

1. What is the application made of? - Complexity.

2. How was the application built? - Methodology.

3. Where does the application run? - Environment.

#1. Complexity - Applications are developed using one or more of open source software, third party libraries, re-used libraries (from the past), middleware, database and the run-time environment. In order to develop a truly secure application we need to ensure security in all of these components that go into building the application.

#2. Methodology - The development methodology that is employed to build the application. This brings up several issues: customization work, secure coding practice, outsourced development, offshore development, peer review, development tools, security requirements as a part of the design, source code scanning, threat modelling and penetration testing.

#3. Environment - Application exist in an environment. This brings up several considerations such as operating system, virual operating system(such as VMware), other applications that co-exist with this application, CPU hardware, storage, network and lastly whether the application runs behind the firewall or in the DMZ.

It is overstatement to say that the application built using secure development methodology is secure. All the three factors Complexity, Methodology and Environment should be considered to make a judgement call about application security. The pragmatic approach is to build application that is secure enough that poses risks that are acceptable to business (customer) this is what I would like to call "Application Due Care".