Complex event processing (CEP) is a new technology. It can be applied to extracting and analyzing information from any kind of distributed message-based system. It is developed from the Rapide concepts of (1) causal event modeling, (2) event patterns and pattern matching, and (3) event pattern maps and constraints. Complex event processing can be applied to a wide variety of Enterprise monitoring and management problems, from low level network management to high level enterprise intelligence gathering.

Applications of Complex Event Processing:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer. (coming soon)
  • Analysing business processes (paper in pdf format)
• Network Level Monitoring and Management (Powerpoint presentation)
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management (coming soon)
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring. (coming soon)
• Analysis and Debugging of Distributed Systems (coming soon)

Presentations:

• “Complex Event Processing: An Essential Technology for Instant Insight into the Operation of Enterprise Information Systems,” lecture at the Stanford University Computer Systems Laborary EE380 Colloquium series. Video of the lecture (duration: 60 minutes).

Publications:

• Complex Event Processing in Distributed Systems. David C. Luckham and Brian Frasca, Stanford University Technical Report CSL-TR-98-754, March 1998, 28 pages.Abstract: Complex event processing is a new technology for extracting information from distributed message-based systems. This technology allows users of a system to specify the information that is of interest to them. It can be low level network processing data or high level enterprise management intelligence, depending upon the role and viewpoint of individual users. And it can be changed from moment to moment while the target system is in operation. This paper presents an overview of Complex Event Processing applied to a particular example of a distributed message-based system, a fabrication process management system. The concepts of causal event histories, event patterns, event filtering, and event aggregation are introduced and their application to the process management system is illustrated by simple examples. This paper gives the reader an overview of Complex Event Processing concepts and illustrates how they can be applied using the Rapide toolset to one specific kind of system.
   
• Event Mining with Event Processing Networks. Louis Perrochon and Walter Mann and Stephane Kasriel and David C. Luckham, The Third Pacific-Asia Conference on Knowledge Discovery and Data Mining. April 26-28, 1999. Beijing, China, 5 pages.Abstract: Event Mining discovers and delivers information and knowledge in a real-time stream of data, or events. We show that the process of delivering knowledge by searching patterns in data and subsequent abstraction of found patterns can be applied in real-time to a complex, asynchronous system. Our event processing engine consists of a network of event processing agents (EPAs) running in parallel that interact using a dedicated event processing infrastructure. The agents can be configured at run-time using a formal pattern language. The underlying infrastructure (1) provides an abstract communication mechanism and thus allows dynamic reconfiguration of the communication topology between agents at run-time and (2) provides transparent, location-independent access to all data. These features allow dynamic allocation of EPAs to different threads and processes on different machines at run time.
   
• eJava - Extending Java with Causality. Alexandre Santoro and Walter Mann and Neel Madhav and David Luckham, Proceedings of the 10th International Conference on Software Engineering and Knowledge Engineering, June 1998, 10 pages.Abstract: Programming languages like Java provide designers with a variety of classes that simplify the process of program development. Some of these classes allow one to easily build multithreaded programs. Though useful, especially in the creation of reactive systems, multithreaded programs present challenging problems such as race conditions and synchronization issues. Validating these programs against a specification is not trivial since Java does not clearly indicate thread interaction. These problems can be solved by modifying Java so that it produces computations, collections of events with both causal and temporal ordering relations defined for them. Specifically, the causal ordering is ideal for identifying thread interaction. This paper presents eJava, an extension to Java that is both event based and causally aware, and shows how it simplifies the process of understanding and debugging multithreaded programs.
   
• Event-Based Execution Architectures for Dynamic Software Systems. James Vera, Louis Perrochon, David C. Luckham.
  Proceedings of the First Working IFIP Conf. on Software Architecture. 1999. San Antonio, Texas.Abstract: Distributed systems’ runtime behavior can be difficult to understand. Concurrent, distributed activity make notions of global state difficult to grasp. We focus on the runtime structure of a system, its execution architecture, and propose representing its evolution as a partially ordered set of predefined architectural event types. This representation allows a system’s topology to be visualized, analyzed and con-strained. The use of a predefined event types allows the execution architectures of different systems to be readily compared.
   
• Using Context-Based Correlation in Network Operations and Management. Louis Perrochon (work in progress, mail author for newest version)Abstract: Network operation consists to a large degree of reaction to activities happening in the network. Better knowledge of the network at any time allows more appropriate reactions. On the example of intrusion detection, we show how context-based correlation of such activities can provide a more detailed view of the network in shorter time. We first present how we model context and then describe the architecture of the Stanford University CEP context-based correlator. Correlation is specified as event patterns in a declarative language that allows us to specify what needs to be detected, instead of specifying how it should be detected. CEP introduces the concept of causal context to intrusion detection. The correlator is able to process events on-line, as they are generated and it can be reconfigured at dynamically. We then show how it increases detection rate, reduce false alarms, and detect large-scale attack patterns at an early stage.

As an Enterprise Architect, I understand the importance of the ability for a security professional to articulate risk to IT and business executives, yet I am also equally passionate that security professionals should also have the capability to sit down at a keyboard and actually do something as opposed to just talking about [it].

I agree wholeheartedly with this sentiment, and I believe the project goals are noble. So I went to read the latest OPCP draft proposal to see how they planned to tackle this admittedly difficult problem. What did I find? It’s just another test, with questions in a dozen or so broad categories. Far more specialized that CISSP, with topics that are more relevant to application security, but ultimately, still just a test.

The comment I once made about security educators/trainers is relevant here. Whatever questions end up on the OPCP test, these educators could probably answer most of them correctly without even studying. They lecture day in and day out about these topics. They have heard obscure questions and are prepared to answer them. And yet, many of them do not have any practical field experience.

A client chastised me once for making a statement that penetration testing is a mixture of art and science. He wanted to believe that it was completely scientific and could be distilled down to a checklist type approach. I explained that while much of it can be done methodically, there is a certain amount of skill and intuition that only comes from practical experience. You learn to recognize that “gut feel” when something is amiss. He became rather incensed and, in effect, told me I was full of it. This customer went on to institute a rigid, mechanical internal process for web app pen testing that was highly inefficient and, ultimately, still relied mostly on a couple bright people on the team who were in tune with both the art and the science.

Certifications only test the science.

Have you ever sat in a corporate training? Some are good, some are bad, but did you ever say, “man I can’t wait for training today.” What about mandatory training? What about mandatory training in a subject that you really don’t think is your area? What if you had to do it every year, and got harassed if you didn’t do it? What if you were, say, an audio engineer and were dragged into a security class?

I ran the SDL training program at Microsoft for a long time, and developed and taught a big chunk of the training. I spent hundreds of hours in front of thousands of developers, testers, and program managers. I got some really good reviews (and a few bad ones) on the classes I offered. And I tried to do a lot of things to try to make the trainings interesting. I handed out dozens of fresh peaches in an early class on fuzz testing, for example.  The room smelled really nice after that, and there are probably still a few people around Microsoft who think of fuzz testing when they see a peach.

But even on my best day, I was under no illusion that the majority of the audience was excited to be there, and I was certain that they weren’t going to go back to their offices and spend weeks applying the lessons from the class, setting aside other things that are causing present and immediate problems in favor of something that is far off into the future.

You have to work at getting people’s attention – especially as it relates to security and privacy. From time to time, I would see people reading their mail in class, and I would point to them and ask them a question. That did not endear me to the audience as much as the peaches, but embarrassment is always fresh and in season.  J

One student wrote of one of my classes, “the basics for secure design - could be replaced by non-anonymous site-wide exam with open material.”  He was not alone, I assure you.

Is that an indication that our training, or any training, is pointless? Hardly, but training alone is not a change agent.

Richard Derwent Cooke wrote,   “It is a first principle of Change Management that people will act in what they perceive as being their best interests.”

At best, training can provide people with insight into what they need to do to solve a security problem if they believe that solving that security problem is in their best interests.

To be effective, training needs to happen in an environment:

·         Where expectations are clearly set (the SDL sets specific minimum requirements).

·         People have appropriate incentives and consequences (security is a great career path at Microsoft, and nobody wants to be the one holding up a ship schedule for failure to meet a security requirement).

·         Where tools and resources to accomplish the goals are available (we build a whole variety of tools that map to the SDL requirements).

·         Where management models the behavior (recall the original BillG TWC memo).

·         Where the environment reflects and supports the values presented in the training (apparent in everything Microsoft does).

Don’t make the mistake of thinking that a bunch of training, even really high quality training done periodically, will result in actual behavior change. It won’t. You have to build an environment where people perceive solving security problems as being in their best interests. You have to make security their problem – not in the sense of passing the buck, but in the sense of changing their behavior so they will bring security problems to you.

To illustrate further, I’ll cite two examples. First, fuzz testing. Fuzz testing has been a success story here at Microsoft. Tools arise spontaneously to solve new fuzzing challenges, written by people who believe the challenges are their challenges. There are people who feel ownership for our fuzzing strategy and on-going research and science, there are specific goals and requirements, we have training (remember the peaches?), and internally developed fuzzers have won prestigious awards within the company, handed out by members of the executive staff, and all of this gets revisited periodically as part of the SDL.

 By contrast, I’ll choose a less successful area – defect estimation. On my own volition, I created (based mostly on some excellent material from Microsoft Research) and taught a class called “Defect Estimation and Management” and added it to the SDL curriculum. Microsoft is a great place to work in that regard. It was pretty close to the best-reviewed class I taught. But, we have not yet been able to establish a set of tools to estimate security defect density effectively, and establish a fair set of expectations, incentives, and consequences, or even  decide what we should do if we had the data. We discovered some things, though. For example, based on what I observed (which should not be construed as rigorous research), it does not appear as if the density of general defects correlates closely with the density of security defects.  And Microsoft Research found higher code coverage in testing correlates with higher bug rates in the field.

And so even though people like the idea of defect estimation, and we’ve got some interesting and surprising data, we’ve not yet been successful in changing people’s behavior.  Generally speaking, an individual test manager does not feel that establishing a high quality estimate of their defect density is in his or her best interests, as compared to, say, improving the time in which an established series of tests can be performed .  

We need to build an environment that has the tools, training, rewards and incentives, and expectations and consequences to change people’s behavior. Not that we’re not trying. But training won’t solve it alone, nor would tools, trophies, rants, testing, code review, or some edict from on high. The SDL is as much about changing the culture and influencing the behavior of individual engineers as it is anything else.

I’m convinced that Microsoft’s SDL process works because it addresses the end-to-end problem - from training through servicing, and provides a complete environment where people feel ownership of their part of the security problem and have the resources to solve it.

So the next time you find yourself sitting in some mandatory training, remember the lessons of the SDL (and most of the research on human performance management): training alone won’t cut it. If you want real behavior change, there have to be things outside the lecture room to influence people to change their behavior.

(It is certainly colossal: 1,089 pounds and 26 feet long.)

There's live video. There's also a lecture series. Video will be available on the Web.