First of all, let’s ground the discussion a bit by translating “smart order routing” to “rule-based message routing” since in this application “smart”  translates to “using rules” and “order” translates to “message”.    Basically, Mark (and other “new on the routing scene” stream processing players) argue that rule-based message routing is CEP.  I will argue that routing is not even close to CEP.  Here is why,

Let’s take a look at a router on the backbone of the global Internet.   A backbone router has very sophisticated software developed over many decades.   These routers run sophisticated, mature algorithms to determine how to route messages (packets) and use these algorithms to build complex routing tables. 

In addition, these routers process messages (packets) from countless sources and route messages (packets) to countless destinations.  Using some of the terms in early posts (above), there is a great “confluence of events” processed by routers.    Futhermore, there are normally quite complex authentication, authorization and other security parameters managed in a router, all in real time.   Routers do much more, but I don’t want to get too deep into routing in this post.

My point is that, without any doubt, global Internet routers process very “cloudy” “confluence of events” with much more sophistication than order routing applications.    However, we do not call Internet routing “CEP”, regardless of how many connections are processed or how much sophisticated processing occurs.  The reason is because the “C” in “CEP” defines a complexity that is at a higher abstraction than messaging and routing.

If you study the literature on CEP, some of which I posted recently, CEP was envisioned to solve complex event processing problems “on top of the routing layer” because the routing layer is a mature technology layer.  We can route, pure and simple.  Of course, we are always seeking faster, more scaleable and more secure routing. 

I admire some of the startups in the CEP/ESP/EP space for working hard to make money and for aggressively positioning their products and attempting to build market share.   However, issues surface when these same companies seem to believe they are the first companies to work in the event processing or message routing space and that they can define whatever they want as “complex event processing” as long as it benefits their sales targets.

There is no doubt that a router does much more sophisticated event processing than the new rule-based stream processing systems running continuous queries across streaming data.  There is no doubt that a router processes a complex “confluence of events”.   However, we don’t call routers “CEP”. 

We do not call routers “CEP” because CEP is about a higher level of knowledge processing.  CEP was created to detect the “complex events” that happen above the mediation and routing layer.     The literature and original examples on CEP are quite clear on this.

(Photo Credit: Gizmodo)

You lose some. You win some: Of course as NBC’s online partner, Microsoft gets a least a cut of the $100 million dollars in online advertising spent around the Olympics. And the millions of downloads of Silverlight aren’t too shabby either.

The Internet is Falling! Arbor Networks, a security and network management company, partnered with ninety network services and content providers from around the world to publish an extensive study of IPv6 traffic on the Internet. Craig Labovitiz, Arbor Networks chief scientist, stated that only 900 days were left until the end of the Internet, or at least the exhaustion of IPv4 registry allocations. For the past year, the study shows very little IPv6 traffic – something like 1/100^th of 1% of Internet traffic. Craig credits this to money issues. “The department of commerce estimates it will cost $25 billion for ISPs to upgrade to native IPv6.”

Blogger James Urquhart created a bill of rights for cloud computing. The purpose of the bill is to “help guide would-be cloud customers to those clouds best able to guarantee their freedom.” The blogosphere is a great place to get some open debate going, and I applaud James for trying to make something yet so “cloudy” a bit more clear and concrete. But what’s up with the creating a PAC for this?? (Check out the comments.)

Trying to get by on limited resources? Need more money, staff and the freedom to focus on long-term projects? Sound familiar? Then you just might be in IT at a midsize company. (or in marketing at a young but rapidly growing IT company ) Arrow Enterprise Computing Solutions conducted a survey of 200 tech leaders at midsize companies (500 to 3000 employees). The upside: 61% of those surveyed think they’ll be spending more on IT next year – is this bullish thinking about the economy or how much their own business (rev) will be growing?

Bill Snyder calls Dell “Bozo of the Month” for trying to trademark “cloud computing”. Yikes. Maybe not a “bozo” move but certainly inadvisable given how ubiquitous the term is. Here’s our take on it.

In order to help folks further understand the differences between CEP and SEP, prompted by Marc’s reply in the blogosphere, More Cloudy Thoughts, here is the scoop.

In the early days of spam filtering, let’s go back around 10 years, detecting spam was performed with rule-based systems.  In fact, here is a link to one of the first papers that documented rule-based approaches in spam filtering, E-Mail Bombs and Countermeasures: Cyber Attacks on Availability and Brand Integrity published in IEEE Network Magazine, Volume 12, Issue 2, p.10-17 (1998).   At the time, rule-based approaches were common (the state-of-the-art) in antispam filtering.

Over time, however, the spammers get more clever and they find many ways to poke holes in rule-based detection approaches.  They learn to write with spaces between the letters in the words, they change the subject and message text frequently, they randomize their originating IP addresses, they use IP addresses of your best friends, they changed the timing and frequency of the spam, etc. ad infinitium.

Not to sound like an elitist for speaking the truth,  but the more operational experience you have with detection-oriented solutions, the more you will understand that rule-based approaches (alone) are not scalable nor efficient.  If you followed a rules-based approach (only), against heavy, complex spam (the type of spam we see in cyberspace today), you would spend much of your time writing rules and still not stop very much of the spam!

The same is true for the security situation-detection example in Marc’s example.

Like Google’s Gmail spam filter, and Microsoft’s old Mr Clippy (the goofy help algorithm of the past), you need detection techiques that use advanced statistical methods to detect complex situations as they emerge.  With rules, you can only detect simple situations unless you have a tremendous amount of resources to build a maintain very complex rule bases (and even then rules have limitations for real-time analytics).

We did not make this up at Techrotech, BTW.   Neither did our favorite search engine and leading free email provider, Google!   

This is precisely why Gmail has a great spam filter.   Google detects spam with a Bayesian Classifer, not a rule-based system.    If they used (only) a rule-based approach, your Gmail inbox would be full of spam!!! 

The same is true for search and retrieval algorithms, but that is a topic for another day.  However, you can bet your annual paycheck that Google uses a Bayesian type of classifer in their highly confidential search and retreival (and - hint - classification) algorithms.

In closing, don’t let the folks selling software and analysts promoting three-letter-acronyms (TLAs) cloud your thinking. 

What we are seeing in the market place, the so-called CEP market place, are simple event processing engines.  CEP is already happening in the operations of Google, a company that needs real-time CEP for spam filtering and also for search-and-retrieval.  We also see real-time CEP in top quality security products that use advanced neural networks, and Bayesian networks, to detect problems (fraud, abuse, denial-of-service attacks, phishing, identity theft) in cyberspace.

In his post, Cloudy Thinking, Marc Adler asks how to implement the event cloud.

As a reminder, we process event clouds; we don’t implement them.   Event clouds simply exist, independent of our desire to process and extract meaningful information from the event cloud.

For example, there are many voices in a crowded stadium.  These voices make up the “sound cloud” (or maybe you prefer the term “voice cloud”), in a manner of speaking.   The “trick” is to have the processing capability to listen to the “sound cloud” and detect opportunities and threats in real-time.   So, in theory, we might call this “complex sound processing”.

Events exists.

The stated goal of CEP is to process event clouds in order to detect opportunities and threats in the business world, in real-time.

We don’t “implement” the event cloud because the events exist independent of our capability to process the cloud and extract meaningful and actionable situational knowledge from the cloud.

However, event clouds are represented as POSETS.  This is directly from the CEP literature.

 Note:  See also, these posts on POSETS.

Like the predictable ebb and flow of ocean tides, we see the rise, falling and passing away of lively debates about event procesing languages (EPLs).   For example, you might recall that Louis Lovas, Progress Apama,  did an excellent job in his post, Bending the Nail, where he described why SQL or Extended SQL is not the optimal EPL for event processing.  

A few of us in the “SQL is not necessarily the best EPL” choir started singing with Louis which motivated a counter voice the choir with the post, Fair and unfair criticism of an SQL EP approach only to have the same author counter that post with, One down side to an SQL EP approach.   

Many technologists, including some of my team members at Techrotech, enjoy focusing on linear event processing problems with strict determinism, for example, processing a stream of market data and looking for opportunties to enter or exit the market (algo trading).    These same technologists tend champion event processing problems that are basic transformations of simple streams of time-series data.  

Many of the so-called CEP cybertrading examples (I would argue that these are simple event processing, not complex event processing examples) are not rooted in event processing scenarios that require looking for causal linkages between seemingly unrelated events; for example, debugging complex distributed systems or detecting fraud over long periods of time where sliding time windows on continuous streaming data are only a part of the solution in the uncertain world of  cloudy event-causality relationships.

Time-series analysis with strict determinism are interesting, but I would not go so far at to call this processing ”complex event processing” relative to the myriad challenging complex problems in the real-world.