Around the same time that Professor Luckham and his team was working on CEP applications in network management and security management, I was leading efforts to build network and security management control centers for the United States Air Force.  In the beginning, dating back to 1994, my Internet-related work was for Air Combat Command (ACC), working out of ACC headquarters at Langley Air Force Base.

In 1997, I lead a technical team that developed countermeasures against an actual distributed Internet-based attack on the Langley AFB SMTP email infrastructure.  This attack was documented in a technical paper, E-Mail Bombs and Countermeasures: Cyber Attacks on Availability and Brand Integrity, IEEE Network Magazine, Vol. 12, No. 2, pp. 10-17, March/April 1998.  In addition, this attackand countermeasures I designed was featured in Popular Science Magazine in an 1998 article, War.Com and other news channels.  I also published a number of related papers on this topic.

Our team used a rule-based approach for countermeasures against massive email bombs attacks on the Langley Air Force Base email infrastructure.   We called this rule-based system, BombShelter. and it was written in PERL.  I developed both the original software architecture and the original working prototype for BombShelter (in two days) and then we turned the software over to our team who used the rule-based approach for daily attack countermeasures.

I watched for days, and then weeks, as my team designed rules, and the attackers wrote new attacks that circumvented the rules.  Some folks in the Pentagon used to say that I “lead the effort to fight the first war in cyberspace”.   It might have have been the first cyberwar, I am not sure, but it was certainly the first publicly documented cyberwar.  There is no doubt about this.

Without getting into all the historical footnotes and significance of this cyberwar that was fought with experts and rule-based systems, I would like to jump to an important conclusion.

Rule-based systems are useful, but have limited functionality and scaleability in most complex event processing applications.

Rule-based systems are human resource intensive because rule-based systems cannot learn and adapt on their own, humans learn and then write new rules.  This is how rule-based systems work.

This is the motivation behind why I spend a lot of time to search for new, more efficient and adaptive methods as alternatives to rule-based systems.   After extensive research, I published a series of papers on the future of intrusion detection in the Internet.  Intrusion Detection Systems & Multisensor Data Fusion - Creating Cyberspace Situational Awareness [1], helped lead an evolution in Internet security, particularly in the area of network-based intrusion detection systems (IDS).

In my published research work, motivated by limitations with rule-based approaches, I used the same mature functional model that is used to process missile attacks, control global air traffic, and other complex event processing applications in physical space; but I applied these concepts to cyberspace.

Around the same time, Professor Luckham and others were working on similar problems, all related to real-time detection and response to threats in cyberspace.  They were also funded by the US government.

Sidebar: Stream processing of transaction- based systems (databases), another area of interest, was focused on a totally different problem, which was the low latency processing of straight-thru processing in databased-oriented systems.   These stream processing systems were, and remain however,  rule-based systems.  The problems we were trying to solve in cyberspace, however, cannot be efficiently and pragmatically solved by rule-based systems alone.  Only relatively simple scenarios can be efficiently detected by rule-based stream processing systems.

The vast majority of complex event processing classes of problems require rules plus advanced algorithms that can learn and adapt in real-time.    I know this, not from reading papers or taking university classes on rule-bases systems, but from working on some very challenging operational problems in real-time.    This is why I remain interested in complex event processing and why I continue to elaborate on why rule-based systems have limitations.

• Network Level Monitoring and Management,
  
• Cyber Security: Network Intrusion Detection,
  
• Enterprise Monitoring and Management,
• Modeling and Simulation of Collaborative Business Processes,
• Business Policy Monitoring, and
• Analysis and Debugging of Distributed Systems.

In each of the CEP application examples above, the amount of event information available to software developers can be staggering; however, despite all the available information, the capability to sense-and-respond to threats and opportunities is crude, at best.

Folks who work in network and security management, for example, are bombarded with event information.  However, this deluge of event information is, for the most part, “noise” that is difficult to understand.   In network management one of the most difficult things to accomplish is to find the root cause of an outage or performance problem.   This is why researchers at Stanford were funded to focused on research topics such as (above), the Analysis and Debugging of Distributed Systems.

These are the classes of asymmetric event processing problems that define complex event processing, or CEP.   Processing events by mediating events, routing events, or running a rule-set against events and making a processing decision are all perfectly valid event processing applications.   However, the core reason to have “complex event processing” is to solve event processing problems where there exists a significant asymmetry between the deluge of “event noise”  (Professor Luckham called this phenomena the “event cloud”) and detecting business-relevant, actionable complex events in an climate of uncertainty and noise.

In my next post on this topic I will briefly the review motivation behind my 1999 ACM paper, Intrusion Detection Systems and Multisensor Data Fusion, where we were working on solving complex distributed security challenges based on real-world experiences with the problems of asymmetric processing capabiilities.  I will discuss why we evolved from an early rule-based expert system model to a more advanced inference model that was not dependent solely on rule-based thinking.   I will also explain why other researchers and developers experienced in complex event detection applications have come to the same conclusion.

“Complex Event Processing (CEP) is a technology which has been used for many years in the Aerospace and Defence Industry for Situational Awareness and Data Fusion modules in Command, Control, Communications, Computing and Intelligence Systems (aka C4I).

Currently CEP is being rediscovered as a foundation for new class of extremely effective Business Intelligence, Security and System/Network/SCADA Monitoring solutions in industries like Financial Services, Telecommunications, Oil and Gas, Manufacturing, Logistics etc. The increasing connectivity and processing power of the modern IT and Telecom technologies lead to increasing speed and volume of the dataflow available to the organisations. By using CEP solutions companies can gain competitive advantage by achieving real-time situational awareness and tapping the information value that is hidden within the streams of real-time event data that are coming from a variety of sources such as enterprise applications, financial transactions, sensor networks and supply chains.”

Unfortunately, the author does not cite references in the paper.

Special Session: Situation Management I

Paper: Complex Event Processing approach for strategic intelligence

Authors: Nicolas Museux, Juliette Mattioli, Claire Laudy and Helene Soubaras

Abstract: One of the key issues of strategic intelligence within a crisis situation is to build an early assessment of the situation, based on a context sensitive information interpretation and through a well constructed situation representation. Our proposal is based on the conjunction of a conceptual modelling to represent situations out of document analysis and a reactive rule-based modelling to analyse them according to a domain knowledge and a goal. This paper focuses on this Situation Analysis process. But we present our global approach and sum-up the Situation Representation and its objectives. We introduce the Complex Event Processing formalism used for the analysis and dynamic recognition of such situations. We illustrate our approach through a case study taken from what happened during the energy crisis in California in 2001.

Presenter Biography: Dr. Nicolas Museux is a research scientist in the PLATON lab, at THALES Research and Technology. He had his engineering diploma in computer science in 1998. Then he started his Ph.D. in Applied Mathematics, Computer Science Systems and Control at the Computer Science Center of e’Ecole des Mines de Paris, and THALES Research and Technology. His Ph.D. focused on the application of constraint programming in distributing low-level digital signal processing programs onto multiprocessors architectures, to optimize data management and computing duration. After he obtained his Ph.D. in 2001, he worked until the end of 2004 on several projects in the PLATON lab linked with combinatorial optimization. Since 2005, Dr. Nicolas MUSEUX works on the Situation understanding research program. Its objectives are to identify, to specify and to design tools for situation model based reasoning in order to address situation analysis, risk assessment and situation projection.

ATC is an excellent working example of complex event processing.   Radar and GPS provide the basic sensory information to accurately track and trace the position of each aircraft in the area of responsibility (AOR) of a particular control tower/zone.     Naturally,  sensory information is preprocessed and formatted in such a way that the data can be processed upstream by multiple real-time applications.

Before we look at complex ATC scenarios, such as “potential collision” or “aircraft off approach vector” we must trace and trace individual objects, aircraft-objects, accurately with very high confidence.    In addition to tracking aircraft-objects, there is a database of information about the aircraft (ideally), such as make, model, age, range, passengers and other properties about the aircraft-object.      In addition, there is a state-model for each aircraft, for example the aircraft might be “on the ground”, “approaching the runway”, “cleared for takeoff”, “cruising altitude”, “approaching runway”, “final decent” etc.  

Tracking and tracing individual aircraft is what is generally referred to as “object refinement” in our CEP/EP reference architecture.   The reason we call this function “object refinement” is that system engineers are focused on optimizing the situational knowledge about individual objects.     Sometimes we refer to this function as “track and trace” because that is what we are doing to  each object in the model.  In Marc Adler’s recent shoplifting scenario, Marc was interested in tracking and tracing people in a store using imaging processing techniques to estimate their behavioral patterns.  In the same way, before we can process for scenarios such as “potential shoplifter” or “suspicious criminal gang activity” we must be able to accurately process (track and trace) individual object, such as people or merchandise.

Back to aircraft and ATC, the “complex event processing” begins when we are looking about object-object relationships, in this model, aircraft-to-aircraft, but this is an overly simplistic model, as we have not yet added (to our model) ground features (towers, buildings, power lines), weather (storm cells, wind) and other flying objects (known migratory bird paths, swarms of insects) to our simple model.  

Complex event processing occurs when we are processing multiple objects in our model looking for threats in real-time.     Practically speaking, all ATC applications are CEP applications.  This means that vendors and integrators who build ATC applications are also CEP vendors.   

Editorial Note: CEP/EP has been around for a long time and was not recently invented in the past decade as some “inventors” would like for us to believe. 

As you can imagine, there is considerable “complex event processing” that goes on “behind the scenes” to provide air traffic controllers and pilots situational knowledge into the “friendly skies”.   As you might further imagine, the situation is more complex when the skies are “not so friendly”, for example, in air combat situations.   

Processing myriad objects is not the end of the processing “chain”.  For example, decisions are being made constantly about potential damage, alternative airports, and more.    In our reference model, we refer to this, generally speaking, as “impact assessment” because we must take an estimated detected complex event, for example “aircraft collision,” and estimate potential damage based on numerous factors such as, the amount of jet fuel in the aircrafts and the location of the aircrafts (over a large city or rural area, near a hospital and emergency services).   Regardless of the scenario, an impact assessment is normally required before optimal decisions can be made.

This is true, by the way, for our shoplifting example (the impact is different if a piece of gum is stolen versus a $1,000,000 diamond necklace or weapons-grade nuclear material) and other scenarios and models.  Static data (information about objects) is required for accurate decision processing.  

Impact assessment is not the end of the “knowledge chain”.    Decisions are constantly being made that effect resources.  For example, suggestion an alternative route for an aircraft is a resource management decision.    Turning on and off radar or switching to alternative tracking devices is a resource management function.  In our CEP/EP reference model (based on the JDL data fusion model), we call this “resource management”.   This function includes contacting emergency services and directing them to a potential crash location or sending out a message to instruct all aircraft to stay off a certain radio frequency.  Resource management is critical.

Our simple ATC model today is by no means complete, it just scratches the surface.  In fact, I have a very close friend, Mark Secrist, who is a former Marine fighter pilot and currently a senior captain for American Airlines.   I have asked Mark to read this post and help me further refine this crude “laymans” ATC model (Thanks Mark!).

My goal at that time was to show everyone that there was a very mature (functional) reference architecture with decades of maturity that applies to (complex) event processing, adapted from the JDL model for information fusion.  Kum Bai Ya.

There is plenty of room for everyone in this model.  Kum Bai Ya.

The model is inclusine not exclusive. Kum Bai Ya.

The JDL model is based on years of operational maturity.  Kum Bai Ya.

The model is functionally and technically accurate.  Kum Bai Ya.

Everyone at the first event processing symposium (March 2006) seemed to agree with this model, at least publicly, because there was no “push back” at the symposium.  Kum Bai Ya.

Professor David Luckham did not discuss architecture in his book, The Power of Events.  Kum Bai Ya.

David’s research at Stanford, some CEP related, was funded by DARPA, who also support the JDL information fusion model.  Kum Bai Ya.

TIBCO Software adopted the JDL model (Note: I worked for TIBCO the time.) Kum Bai Ya.

We built a functional reference architecture around this mature model.  Kum Bai Ya.

We did not claim we invented it.  Kum Bai Ya.

We did not patent the model, only shared it. Kum Bai Ya.

The model is free and open for everyone to use.  Kum Bai Ya.

The folks in the military and government totally agree with this model for CEP/EP.  Kum Bai Ya.

Complex operational problems are addressed every day with this model. Kum Bai Ya.

Air traffic control uses this model.  Kum Bai Ya.

Missile defense uses this model to protect us from harm everyday.  Kum Bai Ya.

Intrusion detection and network management now use this model (Note: I published an ACM paper on adapting this model for cybersecurity 10 years ago). Kum Bai Ya.

Oh, blog-o-sphere.  Kum Bai Ya.

“I have quite a few other concerns the with EPTS Member Agreement.   Basically, the agreement needs to be written with an eye toward a more flexible, open and inclusive process that puts the future of the EPTS square into the hands of the event processing community, not a small group of well intended folks who represent a small part of the overall event processing community and worldview.”

Opher’s reply was to just dismiss these comments, a bit surprising since I served the CEP/EP community on the EPTS steering committee; worked quite hard as a matter of fact, for a number of years.   Opher’s appreciation for the years of work is to just off-handly dismiss my comments.

Then in On faithfull representation and other comments and On Top Down and Bottom Up Opher does the same thing, he simply dismisses my comments, defensively, adding humor, sarcasm and fallacy.

I am sorry Opher is so defensive of his narrow society; however I will not yield, because I do not need to resort to sarcasm, fallacy and ad hominums; the facts obviously support my view.  For proof that Opher has a narrow view of event processing, go no further than look at the companies he hand-picked for his EPTS Steering Committee; most startups (or with startup products) in the event processing space, working on common messages to distinguish themselves in a market with much more mature players excluded - classic “not invented here,” isn’t it?

Opher’s claims the EPTS view on event processing is quite general, but the  majority of vendors on the EPTS Steering Committee members are selling similar platforms, a very narrow segment of the CEP/EP space.    Opher claims that he agrees that other domains (like sensor fusion) are significant to CEP/EP, but he simply dismisses my advice to create a true, general EPTS, inclusive of the prior-art and science of CEP/EP (before the marketing folks took over).  He insists on having the EPTS “reinvent the wheel” and develop their own vocabulary, as if event processing did not exist prior to one book on CEP.

Opher’s fun-to-read blog counterpoints to my concerns are evolving to a mixture of ad hominums and sarcasm, sometime wrapped in a defensive tone.   I think we can do better and we must be more inclusive of the other prior-art.  I say we, because I am also a founding member of the EPTS, althought I suspect Opher will banish my name from the membership for trying to diminish the “not invented here” attitude that seems to dominate the EPTS since inception.

The truth of the matter is that the EPTS has a relatively narrow view of event processing, evident by the makeup of the steering committee and the focus of their discussions.    It is not a technical society about event processing, per se; it is a marketing society with a narrowly focused membership that discounts most of the prior-art in the event processing space, it is really, an Event Processing Marketing Society (EPMS) for a narrow group of niche players.

The event processing domain is much, much larger.   The art-and-science of event processing is deep and mature, much more mature (and inclusive) than what we see in the EPTS. 

I think Opher (and the EPTS committee) should take these comments seriously and not discount them with sarcasm and subtle ad hominum replies.

As a reference, you may have seen this briefing, one of many where I show these functional relationships, Mythbusters: Event Stream Processing Versus Complex Event Processing, from DEBS2007.  For example slide 23 shows the functional relationship between events, pre-processing, event tracking, situational detection, historical patterns (the output of BI tools, for example), visualization and business process management.

In Faithful Representation, Richard Veryard reminds his readers that the most challenging part is in the situation models (not the system integration).  Unfortunately, by accident, Richard incorrectly attributes Opher Etzion’s “first order situation model approximation” to both Opher and I in this quote from Richard’s post, “a simple situation model of complex events, in which events (including derived, composite and complex events) represent the “situation”.   

Actually, that simple situation model above is Opher’s, not mine.  I have offered a more general and comprehensive (first draft) situation model, in A Simple Situation Model for Complex Events based on a cognitive situation model used by researchers at the University of Notre Dame.  I do not believe that complex events and situations can be modelled accurately using Opher’s simple model of derived, composite and complex events.   This model is overly simple, in my opinion. to represent the vast majority of CEP classes of problems, perhaps explaining why Opher and I do not agree on the state-of-the-art of CEP.  Opher tends to view CEP as mostly an extension of active database technology where I see CEP as a technology that is much more closely aligned with the cognitive models represented in the art-and-science of multi-sensor data fusion (MSDF).  

Complex events represent situations, and situations must be accurately modelled if we are going to accurately detect them in real-time.  If your business cannot model a complex event (situation) then it does not matter what software you buy, how much money you spend, or what event processing and integration platform you use.   The models are hard.  The system integration is relatively easy.

The secret sauce is the situation and complex event models.

As mentioned here a few times, it does not matter how fast you process events in real-time, if your model is wrong, you just detect the wrong thing very fast.  This is very bad and quite dangerous.  You will make bad decisions fast.  You will waste time, money and resources.

This is why CEP benchmarks should be based on accuracy in situation detection, not in latency and other low-level performance metrics.   First, get the models right; then refine to detect faster, if speed is required.   What has happened in CEP to date, is that the models are so simple, they do not really detect complex events, they just process and act on simple events that are easy to model. 

The question whether biological viruses are forms of life has been debated, since they lack the respiratory and metabolic process of other accepted life forms. Naturally, different scientists have different reasons for opinions either way.

So how does the new virus-infecting-virus work?

With just 21 genes, Sputnik is tiny compared with its mama — but insidious. When the giant mamavirus infects an amoeba, it uses its large array of genes to build a ‘viral factory’, a hub where new viral particles are made. Sputnik infects this viral factory and seems to hijack its machinery in order to replicate. The team found that cells co-infected with Sputnik produce fewer and often deformed mamavirus particles, making the virus less infective. This suggests that Sputnik is effectively a viral parasite that sickens its host — seemingly the first such example.

…

“It was the cause of great excitement in virology,” says Eugene Koonin at the National Center for Biotechnology Information in Bethesda, Maryland. “It crossed the imaginary boundary between viruses and cellular organisms.”

Science fiction, fantasy and the popular imagination have been fueled in recent decades by the concept of the cyborg, that fusion of machine and creature — but under the scientists’ new definition, even your laptop might be evidence of life…provided it’s infected by a computer virus.

“Actually, it is fair to say that some of CEP can be found in other disciplines. Event processing has been going on in one form or another, for the past 50 years. Simulation, Networking, Active DBs, Middleware.

{ …. }

CEP has only just begun. The foundations are unexplored. Its an open field of research issues.”

Actually, on slide 12 of this presentation from 2006 Processing Patterns for PredictiveBusiness, we show that the foundations for complex event processing have been in place for many years and in many disciplines such as multisensor data fusion, control theory, sensor management, planning, correlation, estimation, tracking, information fusion, data fusion, data mining and more.

One obvious problem (or at least obvious to many of us) with the current group think marketing CEP is that many have ignored the established foundations for event processing and complex event processing that have been mature for many decades. It is not very efficient (nor good for customers) to pick a phrase, or concept, like “CEP” and ignore the relevant mulitiple disciplines that have been used to solve complex classes of distributed event processing problems for decades.

Therefore, “CEP has only begun” is only true for those who have ‘drank the CEP koolaid” and do not understand (yet) that they are “reinventing the event processing wheel” and ignoring (by accident or purposely, I have no idea of the motives) the prior-art and/or selectively picking the prior art or research associated with their company, byline, favorite researcher, CEO, etc. This is a fundamental issue (and constraint) with CEP, in my opinion. Complex event processing does not stand alone as an art or a science, nor should it, nor should it be based on single dimensional, or small groups of single dimensional, technologies.

If you want to see many of the foundations of CEP, you don’t need to go much further than slide 12 of this  presentation from 2006, Processing Patterns for PredictiveBusiness.

Based on my observation, it reminds me of a small group of folks on a discovery mission where their ship lands on the shore of a distant land and they call this “new land” — “CEP” because they feel they have discovered a new land.  Nevermind the big cities that already exist or the many people already “in the fields” of their new land.  These ”CEP explorers” are seemingly in some kind of modern day epic struggle to define themselves as “discoverers” or “founders” and they are coming up with new names of the lakes, rivers, streams and mountains that defined the landscape long before their ship arrived.

Note: It is encouraging to see folks slowly “catching up”…. maybe in a few years we will move CEP beyond the “not invented here” mind share that we see today.

Also note that, recently we saw a flurry of posts where many people rightly stated that “CEP was overhyped” - but then in rebuttal the EPTS community leaders came back with “Is CEP a mere hype?” or “Is CEP a hype?”. spinning the discussion to an extreme position that is wildly different than “CEP is Overhyped”.