• Network Level Monitoring and Management,
  
• Cyber Security: Network Intrusion Detection,
  
• Enterprise Monitoring and Management,
• Modeling and Simulation of Collaborative Business Processes,
• Business Policy Monitoring, and
• Analysis and Debugging of Distributed Systems.

In each of the CEP application examples above, the amount of event information available to software developers can be staggering; however, despite all the available information, the capability to sense-and-respond to threats and opportunities is crude, at best.

Folks who work in network and security management, for example, are bombarded with event information.  However, this deluge of event information is, for the most part, “noise” that is difficult to understand.   In network management one of the most difficult things to accomplish is to find the root cause of an outage or performance problem.   This is why researchers at Stanford were funded to focused on research topics such as (above), the Analysis and Debugging of Distributed Systems.

These are the classes of asymmetric event processing problems that define complex event processing, or CEP.   Processing events by mediating events, routing events, or running a rule-set against events and making a processing decision are all perfectly valid event processing applications.   However, the core reason to have “complex event processing” is to solve event processing problems where there exists a significant asymmetry between the deluge of “event noise”  (Professor Luckham called this phenomena the “event cloud”) and detecting business-relevant, actionable complex events in an climate of uncertainty and noise.

In my next post on this topic I will briefly the review motivation behind my 1999 ACM paper, Intrusion Detection Systems and Multisensor Data Fusion, where we were working on solving complex distributed security challenges based on real-world experiences with the problems of asymmetric processing capabiilities.  I will discuss why we evolved from an early rule-based expert system model to a more advanced inference model that was not dependent solely on rule-based thinking.   I will also explain why other researchers and developers experienced in complex event detection applications have come to the same conclusion.

Abstract

Complex Event Detection (CED) is emerging as a key capability for many monitoring applications such as intrusion detection, sensorbased activity & phenomena tracking, and network monitoring. Existing CED solutions commonly assume centralized availability and processing of all relevant events, and thus incur significant overhead in distributed settings. In this paper, we present and evaluate communication efficient techniques that can efficiently perform CED across distributed event sources.

Our techniques are plan-based: we generate multi-step event acquisition and processing plans that leverage temporal relationships among events and event occurrence statistics to minimize event transmission costs, while meeting application-specific latency expectations. We present an optimal but exponential-time dynamic programming algorithm and two polynomial-time heuristic algorithms, as well as their extensions for detecting multiple complex events with common sub-expressions. We characterize the behavior and performance of our solutions via extensive experimentation on synthetic and real-world data sets using our prototype implementation.

“Experienced end users are very intelligent.

These end users know the complex event processing problems they need to solve; and they know the limitations of the current COTS approaches marketed by the CEP community.  Even in Thailand, a country many of you might mistakenly think is not very advanced technologically, there are experts in telecommunications (who run large networks) who are working on very difficult fraud detection applications, and they use neural networks and say the results are very good.   However, there is not one CEP vendor, that I know of, who offers true CEP capability in the form of neural nets.

Almost every major bank, telco, etc. has the same opinion, and the same problem. They need much more capability than streaming joins, selects and rules to solve their complex event processing problems that Dr. Luckham outlined in his book.   The software vendors are attempting to define the CEP market to match their capability; unfortunately, their capabilities do not meet the requirements of the vast majority of end users who have CEP problems to solve.

If the current CEP platforms were truely solving complex event processing problems, annual sales would be orders of magnitudes higher.  Hence, the users have already voted.   The problem is that the CEP community is not listening.”

Not to be overly repetitive,  but the last part of this quote from a year ago is worth highlighting:

“If the current CEP platforms were truely solving complex event processing problems, annual sales would be orders of magnitudes higher.  Hence, the users have already voted.   The problem is that the CEP community is not listening.”

Frankly speaking, nothing in the “CEP world” has changed, technologically speaking, since this September 2007 post was written.  From a sales perspective, we have seen less CEP-related sales in 2008 than in prior years.   If these so called CEP products were actually capability of detecting “real” complex network-centric situations (threats) in real-time, they would be selling faster than a cup of ice water in the blazing hot Sahara desert.

Don’t shoot the messenger.  Build better detection engines!

On the other hand, maybe complex detection is too hard for most of these companies and that is why they focus on routing, mediation and relatively simple rule-based scenarios, versus complex event processing?

Marc’s initial shoplifting model in his post is based on John Colapinto’s concepts of matching a pattern of customer movements in the store with their estimated patterns of shoplifting behavioral patterns.    Marc’s asks how Coral8 might address this.   We are not ready to seek a vendor solution.  We do not yet have a workable detection model.

As indicated above, I don’t think the example situation cited by John and Marc is a viable model for automated processing.    Tracking the behavior of customer’s movements, by machine, would require some very sophisticated image processing technology that would be too expensive compared to any possible loss at most retails stores.    This type of behavioral pattern recognition. in retail stores, is performed by people (security personnel), not machines, observing people.  

To develop a machine pattern recognition application to detect retail shoplifting we need to build detection models that are economically feasible.  If we are going to use a model of shoplifting pattern recognition versus anomaly detection, we need to define the objects we must track.  

In the most simple model, we have merchandise-objects.   Stores normally (physically) track merchandise-objects only at the exit/entry points of the store using some electromagnetic proximity detection technology.   In this model, the detection configuration is a combination of simple alerting with humans watching the store (”minding the store”).    This is not complex event processing.

However, if we added another object to our model, the customer-object, then we start to get more “complex,” but we have not defined “complexity” yet because we have not defined the object properties, the possible states of the objects, and the relationships between the objects that are the basis for estimated situations.

Hence, model building is constrained by available resources, simple economics and risk (cost-benefit).  If we are detecting shoplifting in Walmart the cost-benefit model for implementing an automated shoplifting detection system would be different than at a top diamond store on 5th Avenue in NYC.   Protecting loss at a weapons-grade uranium respository follows a different model than protecting loss at a handicraft shop, naturally.

Like Marc, I find models to automatically detect shoplifting interesting, so permit me to close with a general discussion of shoplifting in the context of our CEP/EP reference model.

One approach would be do determine what objects will be represented in our model.   For example, if we are going to track merchandise, we need to model the ”merchandise-object”.  If we are going to track people, we need to define the properties of this “person object.”  If we are going to represent the store layout, we need to define all these objects (store-object, table-object, shelf-object, entry-object and so forth).  The model can get “complex” quite quickly. 

Editorial Note:  An object-oriented approach greatly assists complex model building because we can benefit from OO properties such as encapsulation and polymorphism.  For example, we can define a basic “person object class” and then create superclasses of this object for “customer-object”, “manager-object”, “or criminal-object.”

Generally speaking, each object we define will require a state-model, for example, in Marc’s example of a customer moving around the store, we would need to model the possible states (customer at the entrance, at table 1, at table 2, at shelf 1, in the bathroom, at the cashier, etc.)  Indeed Marc, this is complex event processing if we have modelled multiple objects and defined object-object relationships that indicate situations of interest.   For example, customer-object at table2 where merchandise-object has the property of  ”very expensive, high risk” and then customer-object changes state to “in bathroom”.  Of course, we need more key indicators, but you get the idea.

Right now, I am typing from the Taste from Heaven Vegetarian Restaurant in Chiang Mai and my battery is running low.  The owner of this excellent restaurant also runs the Elephant Nature Park, a non-profit organization advocating and acting on behalf of the rights of the mighty elephants in Thailand.  Would be great if we could also automatically detect the situation of “elephant abuse” by poachers and other crimes against nature.   Time to get back to my delicious mushroom salad, Northeastern Thai style.

As always, thanks for reading, time for me to get back to eating!