First of all, let’s ground the discussion a bit by translating “smart order routing” to “rule-based message routing” since in this application “smart”  translates to “using rules” and “order” translates to “message”.    Basically, Mark (and other “new on the routing scene” stream processing players) argue that rule-based message routing is CEP.  I will argue that routing is not even close to CEP.  Here is why,

Let’s take a look at a router on the backbone of the global Internet.   A backbone router has very sophisticated software developed over many decades.   These routers run sophisticated, mature algorithms to determine how to route messages (packets) and use these algorithms to build complex routing tables. 

In addition, these routers process messages (packets) from countless sources and route messages (packets) to countless destinations.  Using some of the terms in early posts (above), there is a great “confluence of events” processed by routers.    Futhermore, there are normally quite complex authentication, authorization and other security parameters managed in a router, all in real time.   Routers do much more, but I don’t want to get too deep into routing in this post.

My point is that, without any doubt, global Internet routers process very “cloudy” “confluence of events” with much more sophistication than order routing applications.    However, we do not call Internet routing “CEP”, regardless of how many connections are processed or how much sophisticated processing occurs.  The reason is because the “C” in “CEP” defines a complexity that is at a higher abstraction than messaging and routing.

If you study the literature on CEP, some of which I posted recently, CEP was envisioned to solve complex event processing problems “on top of the routing layer” because the routing layer is a mature technology layer.  We can route, pure and simple.  Of course, we are always seeking faster, more scaleable and more secure routing. 

I admire some of the startups in the CEP/ESP/EP space for working hard to make money and for aggressively positioning their products and attempting to build market share.   However, issues surface when these same companies seem to believe they are the first companies to work in the event processing or message routing space and that they can define whatever they want as “complex event processing” as long as it benefits their sales targets.

There is no doubt that a router does much more sophisticated event processing than the new rule-based stream processing systems running continuous queries across streaming data.  There is no doubt that a router processes a complex “confluence of events”.   However, we don’t call routers “CEP”. 

We do not call routers “CEP” because CEP is about a higher level of knowledge processing.  CEP was created to detect the “complex events” that happen above the mediation and routing layer.     The literature and original examples on CEP are quite clear on this.

I agree with Mark that SOR is important and very interesting; but in his reply he seems to be confusing CEP with “complex EAI” or a “complex messaging” application.  For example, Mark says,

“It’s not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it’s a stunningly complicated environment in which to create a complex, real-time model in which to apply “simple” routing decisions. On this basis alone, SOR needs CEP.”

Connecting to many market feeds with multiple asset classes might be complicated, but “complicated connections” are an EAI  (adaptation layer) function, not a core CEP function.   In fact, TIBCO Software has been doing this type of low latency back-office order routing for many years, and TIBCO historically calls this “messaging.”  Adding some rules to high speed, low latency messaging does not make it a “CEP” application.

Mark goes on to set up a counter argument to ILOG’s Changhai Ke, comments with,

“SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.”

This is the well travelled argument the “new stream processing vendors in capital markets” have been saying, still unconvincingly, for the last few years.  Basically their perspective is that if you have a lot of ”feeds” and a core requirement for “speed” - “feeds and speed” - you are doing “complex event processing.” 

Mark Palmer forcefully stated his opinon that the folks who do not agree with him do not “understand” modern day SOR.    However,  a strong counter argument can be made that the “newcomers” to capital markets like StreamBase do not understand that “feeds and speeds” with order routing is little more than moderan day EAI.   This is a basic message routing capability and it has been around for a long time.  After all, Wall Street operated quite well before the term CEP was coined!  TIBCO technology was providing Wall Street back office, low latency, smart order routing a decade ago, and they called this technology “messaging”.  

So, I remain unconvinced, at least by Mark’s passionate counter post, that SOR is CEP.   SOR, as Mark and other have described it, is a low latency messaging technology.  Message routing rules have exisited in this technology space for decades.

I agree with Mark completely that low latency EAI (like SOR has been described) can be quite complex, from a “feeds and speeds” perspective.   However,  I remain skeptical that “feeds and speeds” is much more than  modern day messaging and message routing.

In closing, in the network and security management world we have been dealing with “myriad feeds and speeds” for as long as I can remember, but admitted not like capital markets.    Taking myriad feeds, running rules against the feeds and then routing the messages/events for further processing, regardless of the complexity of the feeds and the data, is actually more of a messaging/ESB technology than a CEP technology. 

I remain completely open minded to any convincing counter arguments.

My goal at that time was to show everyone that there was a very mature (functional) reference architecture with decades of maturity that applies to (complex) event processing, adapted from the JDL model for information fusion.  Kum Bai Ya.

There is plenty of room for everyone in this model.  Kum Bai Ya.

The model is inclusine not exclusive. Kum Bai Ya.

The JDL model is based on years of operational maturity.  Kum Bai Ya.

The model is functionally and technically accurate.  Kum Bai Ya.

Everyone at the first event processing symposium (March 2006) seemed to agree with this model, at least publicly, because there was no “push back” at the symposium.  Kum Bai Ya.

Professor David Luckham did not discuss architecture in his book, The Power of Events.  Kum Bai Ya.

David’s research at Stanford, some CEP related, was funded by DARPA, who also support the JDL information fusion model.  Kum Bai Ya.

TIBCO Software adopted the JDL model (Note: I worked for TIBCO the time.) Kum Bai Ya.

We built a functional reference architecture around this mature model.  Kum Bai Ya.

We did not claim we invented it.  Kum Bai Ya.

We did not patent the model, only shared it. Kum Bai Ya.

The model is free and open for everyone to use.  Kum Bai Ya.

The folks in the military and government totally agree with this model for CEP/EP.  Kum Bai Ya.

Complex operational problems are addressed every day with this model. Kum Bai Ya.

Air traffic control uses this model.  Kum Bai Ya.

Missile defense uses this model to protect us from harm everyday.  Kum Bai Ya.

Intrusion detection and network management now use this model (Note: I published an ACM paper on adapting this model for cybersecurity 10 years ago). Kum Bai Ya.

Oh, blog-o-sphere.  Kum Bai Ya.

As we get closer to the conference dates, I wish that I had made plans to fly back to the US to meet everyone.    However, I have been cutting back on public speaking, taking a break since May.  In addition, Gartner did not ask me to speak at their Event Processing Summit this year, I assume because they did not want to pay airfare for my flight from Thailand to the US.    Also, Gartner always likes to fill their conference speaking slots with as many Gartner speakers as they can, unless you are a paid sponsor; and I noticed a number of Gartner employees speaking in multiple slots.

(Editorial Note) Then again, maybe I complained to much about the lack of organization and conference problems when I was invited at be a Gartner keynote speaker last time - reservations not made propertly,  problems with the guest speaker registration list at sign-in, rooms shifted without notifying the speakers and panelists.   Admittedly, I was not happy with the conference organizers at the last get together.  This was my fault, as I am accustomed to better conference execution and am probally too “picky” about details these days - my bad.  Anyway, the Gartner organizers apologized numerous times, saying they had too many conferences going on at the same time and not enough people to cover them all.

One of the problems with spending so much time in Asia, especially in Thailand, is that guest speakers are really treated as VIPs.  There are usually special comfy couches set up for the speakers and the conference staff really treat you very nice, taking care of you every step of the way.   In fact, there is an entire very nice culture around how guest speakers are treated in Thailand.   Often, they pin flowers on the VIP speakers and take your photos like you are a star.    Very nice culture.

I absolutely look forward to speaking on event processing or CEP at a future venue and meeting everyone face-to-face instead of over the net.  My sincere and deepest apologies for not attending either the Gartner or the EPTS event this year.   

PS:  If you take up a collection and send me a RT business class air ticket, I might change my mind

“Event processing has been going on for more than fifty years.”

However, in On Event Processing as a Discipline and Some Subsets another colleague mistakenly blogs,

“… people who dealt in this area [network management and event correlation] have never investigated event processing in the larger sense (e.g. looking at additional patterns), and this area has also not spawned the event processing discipline.”

If you examine just one page from the CEP history at Stanford, researchers there outlined their view of the future applications for CEP, as follows:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer.
  • Analysing business processes
• Network Level Monitoring and Management
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring
• Analysis and Debugging of Distributed Systems

These applications areas mentioned by Stanford researchers, including Professor Luckham, support and validate our recent discussion Magic Quadrant for IT Event Correlation and Analysis, 2007 where we concluded that “event correlation and event analysis is Gartner’s closest magic quadrant (MQ)  [...] relates directly to complex event processing (and event processing in general).”  

If you take a detailed look at the 1999 CEP presentation, Defeating Large Scale Attacks: Technology and Strategies for Global Network Monitoring you will readily see that our colleagues are incorrect when they says that event correlational and network management folks have never investigated event processing in the “larger sense”.  For example, the 1999 slides above, Stanford, slide 6, is titled “Complex Event Processing,” defining CEP from the application perspective of event correlation;

Complex Event Processing

• Accept network ‘events’ from any source
  • CISCO NetFlow FlowCollector, tcpdump
• Correlates events based on content and temporal relationship between events
• Event Processing Agents (EPAs) connected in an Event Processing Network (EPNs)
• Both post-mortem and real-time processing

This single event correlational project example from David’s team at Stanford examined the challenging event correlation problems in the context of hierarchical events, maps, patterns, visualization tools, event processing models, patterns languages, network management abstraction layers, and more.  Those core event processing problems from this 1999 example, very large and complex then, still exist today and are much more large and complex - precisely why it is called “complex event processing.”

It is quite obvious, in just this one example, that many folks have been looking at event correlation as a motivating application for event processing, in a larger context, for a long time, contrary to what our colleagues write in their “history of event processing” posts.  

In a future post I will completely debuke these event processing “history revisionists.”   I will illustrate very clearly how the history of event processing goes back at least a decade, and perhaps two (twenty years) before the history outlined in posts like On Research and Practice in Event Processing and The History of Complex Event Processing. 

David Luckam stated that the art-and-science of event processing goes back around 50 years. 

I am not sure I will go all the way back to 1960 in my next post on the history of event processing.  However,  I will go back at least to the early days of Internet Protocol (IP) networking and illustrate why distributed IP networking, network management and network security, is one of the key  motivating factors for what we now call “event processing” and “complex event processing.”

Jeff uses the example, “Smart Order Routing” as an example of taking an event and routing the resulting market order match based on some simple rules.    Routing a order kicked off by a simple order match against a deep liquidity pool (or other market factor) does not define complex event processing nor detecting a complex event - the core idea behind CEP.   Order routing based on simple rules is BPM, plain and simple.

Let’s take another example, fraud.  In this example, there is some complex neural network monitoring for credit card fraud and a potential fraud is detected - this is CEP, detecting a complex event based on some sophisticated analytics.   

After a possible fraud has been detected, a process looks into a database and the routes the incident to someone in the company who is a (1) specialist in credit card fraud, (2) working at the same time of the discovered threat, and (3) immediately available to act on this type of task.   Routing the incident is not CEP, it is BPM.

Jeff makes the argument that it is OK to call an event-driven BPM task CEP because “it fits the EPTS definition” in the CEP glossary.   He also avoids the discussion of detection accuracy, and instead insists that latency is a ”very important” factor in a CEP application.

If you read the various post by vendors in the blog-o-sphere, it is obvious that they are continually defining CEP as BAM, BPM, BRE, BRMS, SOA and just about every other related processing activity that is complimentary to the event correlation and analysis required to detect an opportunity or threat to your business.

I’m not picking on Aleri.  TIBCO has been doing the same thing recently in their CEP blog, continually attempting to redefine CEP as BRMS.    Detecting business opportunities and threats with high confidence requires sophisticated analytics, and their tools have not yet evolved to “real CEP” capabilities.  Instead, vendors are attempting to redefine BPM, BRMS, BRE, and even SOA to some degree, as CEP. 

CEP is Not BPM, BAM, BRE, BRMS or SOA.

At the moment, event correlation and event analysis is Gartner’s closest magic quadrant (MQ)  that relates directly to complex event processing (and event processing in general).    

A number of our friends and colleagues would like to position CEP as BRE, BRMS, BPM, SOA, algo trading and just about every other technology under the sun, except event correlation!

In a nutshell, the state-of-the-state of CEP/EP is that a number of firms in the software industry have found some “uncharted magic quadrant waters” and are positioning themselves to be “chart worthy”. Instead of competing head on with the experienced players (event correlation and analysis) that have been in the event processing field for many years.   

As I have mentioned a few times here on The CEP Blog, if the current generation self-described CEP engines were leading the industry in event correlation and analysis (CEP’s core technology domain) they would be either be on Gartner’s Magic Quadrant for IT Event Correlation and Analysis, or possibly acquired by a one of these large giants in event processing to solve complex event processing and event correlation problems that remain, for the most part, still unsolved!

adaptive process management (n.) an element of resource and business process management, adaptive search and event processing. Sometimes referred to as “Level 4” event processing or process refinement.

application concept (n.) a definition of a set of properties that represent the data fields of an application entity. An application concept can describe relationships among themselves. For example, an order concept might have a parent/child relationship with an item concept. A department concept might be related to a purchase requisition concept based on the shared property, department_id. Application concepts can include an application state model.

application state modeler (n.) a UML-compliant application that allows you to model the life cycle of a concept instance — that is, for each instance of a given concept, you can define which states it will pass through and how it will transition from state to state. States have entry actions, exit actions, and conditions, providing precision control over the behavior of an event processing agent. Transitions between states also may have rules. Multiple types of states and transitions maximize the versatility and power of the application state modeler.

derived event (n.) an event that is created as a result of processing one or more other events.

complex event (n.) an event that is a situation-entity abstraction of two or more simple, derived or other complex events.

complex event processing (n.) CEP is a technology for extracting information from message-based systems. CEP is primarily an event processing concept that deals with the task of processing multiple events from an event cloud with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes.

event (n.) a instance of an event definition. It is an immutable object that represents a business activity that happened at a single point in time. Just as one cannot change the fact that a given activity occurred, one cannot change an event — events are immutable.

event aggregation (n.) the aggregation of simple, derived or complex events into higher levels of event abstractions.

event definition (n.) a set of properties related to a given activity that represents an important or interesting change of state in a human, system or computational activity. An event definition includes event properties such as event priority, event time to live (TTL), and a description of the payload, which is comprehensive information related to the activity that occurred. Events expire when the TTL has elapsed, unless the event processing agent has instructions to consume them prior to that time.

event channel (n.) a communications channel in which events are transmitted from event source to event receivers, typically received as electronic messages. Each channel can have multiple destination and. events can be configured to transmit to a default destination. JMS is an example of an event channel.

event cloud (n.) a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events. Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud.

event-driven (n.) the behavior of a human, system or computational entity whose execution or actuation is in response to events, typically received as electronic messages.

event-driven architecture (n.) an architectural style for distributed computing applications in which some of the components are event-driven and communicate by means of events.

event processing (n.) computing that performs operations on events, including modifying, creating and destroying events.

event-object (n.) an software object that represents an event, generally for the purpose of computer processing, that exhibits both encapsulation, inheritance and polymorphism.

event prediction (n.) computational activity where the impact of events, complex events, and situations caused by events identified, including both opportunity or threat. Sometimes referred to as “Level 2” event processing, impact assessment or predictive analytics.

event pre-processing (n.) computational activity where events are cleansed or normalized to produce semantically understandable data. Sometimes referred to as “Level 0” event processing.

event processing (n.) computational activities on events dealing with the association, correlation, and combination of event data and information from single and multiple event sources to achieve refined identity and situation estimates for observed event objects, and to achieve complete and timely assessments of opportunities, threats, and their significance. Event processing is characterized by continuous refinements of event estimates and assessments and by evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

event processing agent (n.) an EPA is a computational entity that performs event processing.

event processing network (n.) a set of event processing agents and a set of event channels connecting them.

event properties (n.) data representation of an event, typically by name-value pairs of type string, integer, real, boolean or a complex data type.

event refinement (n.) filter, identify and track events & make initial processing decisions based on association, correlation and state estimation. Sometimes referred to as “Level 1” event, or event-object, track and trace.

event stream (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

event stream processing (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

rule (n.) defines what triggers unusual, suspicious, problematic, or advantageous activity within an event processing agent and what the EPA does when it discovers these types of activities. Rules execute actions based on certain conditions on events, instances, or a combination of both. A rule includes a group of condition-rule statements and action-rule statements. The condition statements instruct the EPA what to look for in events, and action statements instruct the EPA how to respond when conditions are met. If all the conditions in a rule are satisfied by events or instances or both, the EPA fires the actions. The action might be to execute tasks, create an event instance, modify property values in an event instance, create and send an event, or something else.

rules engine (n.) a type of event processing agent that uses a declarative programming model to process events. Formally described as “an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet“. Informally, it can be any system that uses rules, in any form, that can be applied to data to produce outcomes.

rule language (n.) is an artificial language that is used to control the behavior of an event processing agent. Rules languages, like human languages, have syntactic and semantic rules to define meaning.

situation refinement (n.) identify situations, or complex events, based on event clustering, event-event relationships and relationship analysis and context. Sometimes referred to as “Level 2” event processing.

simple event (n.) an event that is not an abstraction or composition of other events.

virtual event (n.) an event that is imagined, modeled or simulated.

A good place to being to look for clues to an answer is Wikipedia, where the opinion of the author there is,

 ”A simple and practical definition, however, would be how an entity (i.e., business) arrives at an optimal or realistic decision based on existing data.”

Quoting the Wikipedia author(s) further,

“Common applications of Analytics include the study of business data using statistical analysis in order to discover and understand historical patterns with an eye to predicting and improving business performance in the future. Also, some people use the term to denote the use of mathematics in business. Others hold that field of analytics include the use of Operations Research, Statistics and Probability. However, it would be erroneous to limit the field of analytics to only statistics and mathematics.”

The Wikipedia author(s) continue their discussion of analytics, as follows;

“Analytics closely resembles statistical analysis and data mining, but tends to be based on modeling involving extensive computation. Some fields within the area of analytics are enterprise decision management, marketing analytics, predictive science, strategy science, credit risk analysis and fraud analytics.”

All of these topics above are CEP-related areas involving complex events and situations based on the need for optimal and reliable real-time capabilities to make meaningful (business) decisions. 

Simple pattern matching, event mediation and routing, and basic mathematical calculations do not really fall into the realm of complex event processing.  Instead, CEP is real-time decision support based on modeling and “extensive” computation.  In a nutshell, complex events and situations require analytical models that are non-trivial and that is why without analytics, there is no true “complex event processing.”

See also:

WIkipedia on Predictive Analytics

You can not process complex events in some very simple way and expect to get accurate results.  You need knowledge, represented by one or more situational models, to process complex events.

Some folks, like to say that a “complex event” is simply an event which is an aggregation of two more more event objects.    If you follow this (flawed) logic, then counting integers is complex event processing; because 1 plus 1 is 2, and 2 is an aggregation of 1 and 1, so 2 is a complex event (not!).  

Since we know that counting is not a complex processing operation, then some folks would say that you can process complex events with very simple operations because you are processing complex events , in the case adding 1 to the previous number (counting), enriching an event object.

This is simply nonsense.

The logic flaw is that the basic definition of a “complex event” (used by many people) is wrong.   A complex event is not simply an event object with two more more events as sub-components. 

A complex event is when two event objects are combined (processed) to form a complex object with a higher degree of inference, or situational knowledge.   One plus one equals more than two in complex event processing, because the combination of event objects requires knowledge (e.g. a situational model).

A Complex Event = Sum (EventsObjects) + Situational Knowledge

Let there be no mistake about it.    Complex event processing is the complex processing of complex events.   You cannot accurately process complex events with simple event processing models.

The simple processing of complex events is not CEP, it is simple event processing (event track-and-trace, simple event object enrichment, simple event object aggregation, and so forth).