Complex event processing (CEP) is a new technology. It can be applied to extracting and analyzing information from any kind of distributed message-based system. It is developed from the Rapide concepts of (1) causal event modeling, (2) event patterns and pattern matching, and (3) event pattern maps and constraints. Complex event processing can be applied to a wide variety of Enterprise monitoring and management problems, from low level network management to high level enterprise intelligence gathering.

Applications of Complex Event Processing:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer. (coming soon)
  • Analysing business processes (paper in pdf format)
• Network Level Monitoring and Management (Powerpoint presentation)
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management (coming soon)
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring. (coming soon)
• Analysis and Debugging of Distributed Systems (coming soon)

Presentations:

• “Complex Event Processing: An Essential Technology for Instant Insight into the Operation of Enterprise Information Systems,” lecture at the Stanford University Computer Systems Laborary EE380 Colloquium series. Video of the lecture (duration: 60 minutes).

Publications:

• Complex Event Processing in Distributed Systems. David C. Luckham and Brian Frasca, Stanford University Technical Report CSL-TR-98-754, March 1998, 28 pages.Abstract: Complex event processing is a new technology for extracting information from distributed message-based systems. This technology allows users of a system to specify the information that is of interest to them. It can be low level network processing data or high level enterprise management intelligence, depending upon the role and viewpoint of individual users. And it can be changed from moment to moment while the target system is in operation. This paper presents an overview of Complex Event Processing applied to a particular example of a distributed message-based system, a fabrication process management system. The concepts of causal event histories, event patterns, event filtering, and event aggregation are introduced and their application to the process management system is illustrated by simple examples. This paper gives the reader an overview of Complex Event Processing concepts and illustrates how they can be applied using the Rapide toolset to one specific kind of system.
   
• Event Mining with Event Processing Networks. Louis Perrochon and Walter Mann and Stephane Kasriel and David C. Luckham, The Third Pacific-Asia Conference on Knowledge Discovery and Data Mining. April 26-28, 1999. Beijing, China, 5 pages.Abstract: Event Mining discovers and delivers information and knowledge in a real-time stream of data, or events. We show that the process of delivering knowledge by searching patterns in data and subsequent abstraction of found patterns can be applied in real-time to a complex, asynchronous system. Our event processing engine consists of a network of event processing agents (EPAs) running in parallel that interact using a dedicated event processing infrastructure. The agents can be configured at run-time using a formal pattern language. The underlying infrastructure (1) provides an abstract communication mechanism and thus allows dynamic reconfiguration of the communication topology between agents at run-time and (2) provides transparent, location-independent access to all data. These features allow dynamic allocation of EPAs to different threads and processes on different machines at run time.
   
• eJava - Extending Java with Causality. Alexandre Santoro and Walter Mann and Neel Madhav and David Luckham, Proceedings of the 10th International Conference on Software Engineering and Knowledge Engineering, June 1998, 10 pages.Abstract: Programming languages like Java provide designers with a variety of classes that simplify the process of program development. Some of these classes allow one to easily build multithreaded programs. Though useful, especially in the creation of reactive systems, multithreaded programs present challenging problems such as race conditions and synchronization issues. Validating these programs against a specification is not trivial since Java does not clearly indicate thread interaction. These problems can be solved by modifying Java so that it produces computations, collections of events with both causal and temporal ordering relations defined for them. Specifically, the causal ordering is ideal for identifying thread interaction. This paper presents eJava, an extension to Java that is both event based and causally aware, and shows how it simplifies the process of understanding and debugging multithreaded programs.
   
• Event-Based Execution Architectures for Dynamic Software Systems. James Vera, Louis Perrochon, David C. Luckham.
  Proceedings of the First Working IFIP Conf. on Software Architecture. 1999. San Antonio, Texas.Abstract: Distributed systems’ runtime behavior can be difficult to understand. Concurrent, distributed activity make notions of global state difficult to grasp. We focus on the runtime structure of a system, its execution architecture, and propose representing its evolution as a partially ordered set of predefined architectural event types. This representation allows a system’s topology to be visualized, analyzed and con-strained. The use of a predefined event types allows the execution architectures of different systems to be readily compared.
   
• Using Context-Based Correlation in Network Operations and Management. Louis Perrochon (work in progress, mail author for newest version)Abstract: Network operation consists to a large degree of reaction to activities happening in the network. Better knowledge of the network at any time allows more appropriate reactions. On the example of intrusion detection, we show how context-based correlation of such activities can provide a more detailed view of the network in shorter time. We first present how we model context and then describe the architecture of the Stanford University CEP context-based correlator. Correlation is specified as event patterns in a declarative language that allows us to specify what needs to be detected, instead of specifying how it should be detected. CEP introduces the concept of causal context to intrusion detection. The correlator is able to process events on-line, as they are generated and it can be reconfigured at dynamically. We then show how it increases detection rate, reduce false alarms, and detect large-scale attack patterns at an early stage.

Image via Wikipedia

When I find a new product or service that I think is good I am only too happy to let the world know it on my blog. For the past almost 2 years in the notes of every episode of our podcast, I mention and thank ClickCaster for hosting our podcast.

I originally was turned on to ClickCaster by Scott Converse out in Boulder, Co who was the founder of ClickCaster.  When Scott realized that a free model was not going to pay the bills, he instituted a pay model for podcast hosting. I was only too happy to pay for the great service and stats I was receiving. Well a few months ago Scott and team sold ClickCaster to focus on their new project, Medioh!.

The new owners, nexplore promised no changes and same great service.  Since then the stats stopped working, it became harder and harder to post new content and the site was down more than it was up.  Finally after getting no satisfaction from ClickCaster I had no choice but to look for another host.  Mitchell and I have chosen Pod-o-matic to host the podcast going forward.

Of course we don't have all of the episodes moved over yet because ClickCaster isn't even up enough for us to grab all the episodes.  But most of them are up at pod-o-matic and we have already repointed the feedburner/iTunes feed.  So from here on you can hear us at pod-o-matic. 

If you are looking to host your podcast, you don't have to use pod-o-matic, but don't use ClickCaster!

The various components (sub functions) of security organization should align with the business objectives of the IT and the whole organization. There needs to be a cohesive security strategy in order to align the various comoponents. One good way of understanding the business objective is why is the business parting with money for deploying a specific security component. Why is business giving me money for Compliance? Why is business giving me money to implement IDP? Constitutive questions such as these will help you to understand the fundamental concerns for the business and based on these we can come up with a strategy suitably aligned with the business.

One good example is the area of compliance. Attempting to make each every units of your business complaint with certain standards/legal regulations and so on would be a tall order. First define the scope, draw a circle around the units that need to be compliant, then come up with a strategy to make it compliant by formulating your objective - derived from the business objective of why the business gave you money.

Any security implementation effort should have a well defined focus (scope), business objective and strategy to bind the various components cohesively that aligns with the ultimate business objective. By this business will view security organization with dignity else security organization will end up being a spoke in the wheel of business.

In the past, I was involved in discussion about the ROI of information security and security is insurance and so on. After eating the forbidden apple from the tree of paradise, I realize security has neither ROI nor akin to insurance. Information security is way of doing business with due care. Security is way of enhancing the trust of a business among customers and thus enhancing the identity (or brand image of the company). Few years down the line people won't even question why you do security, it will become a part of  your background conversation. Nobody questions why we buy hybrid vehicles anymore right?

If components of security function is not cohesively aligned with business objective it is spoke in the wheel of business else it is a brand enhancer of business.

One of the projects leaders uses terms that should warm every New Yorker's heart, if he or she knew what they meant. IT head Paul Cosgrave says the system will overcome silos, an often disparaging term for the separation of resources across groups that can only expensively be overcome. It's the government and business equivalent of the academic problem of a lack of cross-discipline focus.

One of the first applications allows sanitation workforce managers a frighteningly precise amount of knowledge about routes, activities, and behavior of trucks in their territory. Let's hope that's not misused! Efficiency is one thing; micro-management is another.

Another project is testing wireless water-meter reading. The city hopes to spend $90 per meter for the upgrade and shed part of a $12.2m contract with Con Edison that covers 850,000 units. What should be useful about this is that problems can be detected by monitoring waterflow patterns, which in turn allows the often huge problems that take months to notice (occurring underground or in basements where rivers formerly flowed) to be stopped before they turn into multi-million-dollar problems for property owners or the city. Anytime anything happens in Manhattan, it's a multi-million dollar problem.

First up is Laura.

I have been a Security Program Manager for the last 3 years, working as a security advisor for a variety of products across Microsoft and the last seven months as a member of the SDL policy team.

 It's been a few years since I've been to TechEd, and this was my first time attending as a member of the security team. TechEd is now a two week conference, with one week dedicated to developers and  the other to IT professionals.  I think that breaking down the conference into a Developer week and an ITPro week was a good idea, and it allowed us to have good conversations with people who wanted more information about the SDL. I did two main things at TechEd:, I presented on threat modeling, and I spent a lot of time talking to customers at the SDL booth. At the SDL booth, we heard questions ranging from "What does the SDL stand for?" to "Our Web site was hacked; how do I stop it from happening again?" It was encouraging hearing people interested to hear more specifics about how we implement the SDL at Microsoft, and thinking through how they can apply it in their own companies.  My understanding from other TechEd veterans in our booth is that interest in the SDL seemed higher, which is great.

During my Threat Modeling session, , most of the feedback and follow-up questions were similar to the ones in the booth: how to expand the threat modeling processes to their own companies, and how to get started.

My typical response to both questions is to start small and do what makes sense for your organization. At  Microsoft, for example, when we introduce new SDL requirements, we usually start with a few teams so we can refine the requirement and supporting tools before expanding the requirements to a broader group. Similarly, while we have a core set of requirements that all teams have to meet, there are other requirements that are specific to a platform, scenario, or functionality. For example, there are some requirements that make sense for desktop-oriented products, but do not make sense for mobile devices.  You may very likely have to make changes to our policies to make them relevant to your organization, your scenarios, and functionality.

Now over to Michael.

Hi, Michael here.

One of the joys of presenting at TechEd each year is hearing from real people about the issues they face using our products in the real world; rarely are the issues pure philosophical security geekness. This year I gave two talks and one "chalk talk." The talks were "Top Ten Strategies
To Secure Your Code" and "How To Review Your Code
and Test For Security Bugs", and the chalk talk, which was a lot of fun, was simply answering numerous developer questions.

It's interesting to gauge overall security awareness from our customers, and there is no doubt that over the years, the level of security knowledge and maturity has risen. I think it's possible to evaluate overall security maturity by the questions posed. Some years ago, security was never really a topic of discussion other than those that relate to security technologies, such as how to use and manage X.509 certificates. About four years ago the tide really changed and people started asking more questions about "secure" application deployment and management, and developers wanted to learn more about securing their code; especially C and C++ code. Even then there was still a reliance on exterior defenses like firewalls. All too often I would hear people claim that they don't need to focus on securing their apps because a firewall was in the way. Heck, David and I documented this excuse in the original version of Writing Secure Code (Appendix D, "Lame Excuses We've Heard, #6, ‘We're Secure-we use a Firewall'") way back in 2002.

Fast forward to 2008.

Things have obviously changed. I don't know if finally the security message is getting through because many people asked me highly specific questions about securing their apps and how best to use the defenses we offer in Windows Vista and Windows Server 2008.

I still hear the firewall excuse a little, but not too much!

Perhaps the most telling trend I saw this year was a great deal of interest in the SDL. Not cursory, "that looks interesting" interest, but, "how can I implement this in my company" interest. After answering specific questions, I pointed most folks to  Jeremy's "Crawling Toward SDL" post on the subject.

In my opinion, getting to a point where you want to change your development process shows you really understand there's an issue that needs fixing.

(Photo credit: The Tech Stop)

ScienceLogic: How long have you been involved with Interop?

Geoff Horne: Since about 1996.

ScienceLogic: How has it been changing? Does the show get more complex with new technologies or because of the constantly changing size of the show?

Geoff Horne: The technologies have changed. Every year there’s a different market environment. Since we build on customer needs, things change every year. Things like ScienceLogic for Network Monitoring, for how long have Network Management tools been completely web based? In general, it doesn’t really get any better or worse because every year we’re building it again. You don’t get the stability of a standard environment. The upside is that we’re always doing a full upgrade, a full technology refresh and not using old code.

ScienceLogic: Do those kinds of changes influence the types of vendors you look for for InteropNet?

Geoff Horne: The base categories don’t change. You always need to forward packets. You always need switches, you always need routers. We’ve tried to open it up to everyone that has products involved with networks to see if we have the time or space for it.

ScienceLogic: The kind of cooperation that you get between the vendors is what seems to be an unachievable nirvana for Enterprises. What’s the secret to getting 17 vendors to work together in such a short time? Enterprises would kill for that.

Geoff Horne: The honest answer is don’t trust the vendors. If they try and build something the way they want to, its not going to interoperate. You have to pull them out of their safety zone, make them do things that you think the product can/should do to ensure interoperability.

ScienceLogic: In a blog post prior to Interop Vegas 2008 you stated three major goals for InteropNet. They were Education, Monitoring and Statistics. How did you do against these goals?

Geoff Horne: I think we did pretty well. They’re 3 things we really didn’t have before. They’re things that just weren’t focused on the right way. For the first round of changing the focus, changing the way people look at the network (statistics rather than packets), it worked quite well, it gave people a much better idea as to what’s going on.

ScienceLogic: If we look at NY as take two for Interop 2008, are there things you are going to do differently based on lessons learned in Vegas?

Geoff Horne: We’re building more physical redundancy in the core network, geographic distribution of the infrastructure within the show. This will allow us to bring up chunks of the network independently. It isn’t something that we really thought of before. This helps us take the single point of failure (the NOC) out of the equation.

ScienceLogic: Are there any lessons learned from Interop that you think would help enterprises?

Geoff Horne: Visibility is key. Your network is significantly more functional when more people can see what’s going on. If the only guy that can see what’s going on is the guy with his fingers on the terminal, no one can make good decisions. You have to make people loosen up their control so that everyone can see and therefore make educated decisions.

ShareThis