The “More” Part of CEP over ESP is Far from Mature

By Changhai Ke, ILOG

An EDA and CEP must be understood as 2 different areas. EDA is an architecture pattern for enterprise applications. The components are loosely coupled by the use of events. In its strict sense, this is more an architecture pattern than an algorithm.

CEP, on the other hand, targets at the event processing and pattern recognition level. For me, it’s the research for the right algorithm to use to recognize the situations. Pattern recognition, event correlation are all good characterizations for CEP. Back 15 years ago, the alarm correlation in the telecom area was done using production rules (it is still the case), and this perfectly falls into the CEP area.

In fact, EDA comes after CEP, but the CEP at that period was not explicitly called CEP. The nature of their respective study is not the same, one is at the architecture and middleware level, the other is at the algorithm side. As both are concerned by events, it seems that people more or less implicitly include CEP in EDA, mix the two and introduce confusion. Why not. But it’s important to understand that CEP (on its algorithm side) could mature on its way without being worried about the event transportation layer.

As a system, CEP needs input events for processing. If EDA is considered as the only way to bring and transport events to the CEP systems, then of course CEP won’t become successful without the prior success of EDA. But in my understanding, CEP targets some real-time or close to real-time applications, and the event transport layer in those applications are the most often ad-hoc and over-optimized. I fear that EDA has the same kind of performance goal.

Another distinction needs to be made. CEP is more general than ESP (event stream processing), characterized by an EPL for data aggregation with notifications. Even on the market most of the CEP vendors provide EPL languages, CEP has the vocation to cover more than that. The “more” part is not well defined, at least it should include the event correlation, and correlation is not just data aggregation.

The ESP part of CEP could be considered as quite mature. There are so many EPL languages, and tuning has been made on the runtime side. It seems also that some applications based on ESP have proved to work. But the “more” part of CEP over ESP is far from mature. It is often described that CEP could use several technologies, such as statistical models, Bayesian network, time series, rules, etc. I agree that there are a few systems using rules. But where are the others?

Sincerely,

Changhai Ke

I started blogging about 2 and half-years ago because I felt like it would be fun to add my two cents to the public debate.  When Brad Feld introduced me to the Feedburner guys I was given an insiders view into the quickly developing blogging world.  When Feedburner started networks, I thought it would be interesting to start a network of all the security blogs that I was reading.  I also inherently knew in my gut that eventually there would be some common good that would benefit all of the members of the network by aggregating our content and buying power for ads. I also believed and still do believe that there are other ways that a network such as the Security Bloggers Network can be a force for good.

However, reading the SBN feed tonight I was just blown away! From being on the road, I had not read the SBN feed in my Newsgator reader for almost 2 days.  I had over 160 articles cued up in the feed.  Forget for a moment that the Security Bloggers Network now has over 160 blogs and a combined feedburner subscriber base of almost 67,000 readers!  The content is king.  Going through the articles I could not believe the total coverage, the ongoing commentary and give and take, but most of all it was the quality.  There are so many great members of the network who are just so damn smart and are writing about such important stuff.

I am humbled and incredibly proud of the what the Security Bloggers Network has become. If you are interested in security, whether it be the technical aspects of security, the business of security or the security industry, you cannot afford to miss this SBN feed. 

We are kicking around a lot of new activities and ways to publicize the member blogs of the network over the coming months.  Stay tuned for details, but in the meantime keep reading, you won't be sorry!

http://www.facebook.com/group.php?gid=23977227924

Feel free to add any links or messages you like.  Vendors sites welcome too!   Just network, market, relax and have fun!

• Report itself [PDF] and brief on it from Verizon (and two fun follow-ups, this and this here)
• "90% of all statistics can be made to say anything… 50% of the time, aka my thoughts on the Verizon report"
• "Data Breach Post Mortem Offers Surprises" (well, to some people, they are surprises ...)
• "Insider Threat Exaggerated, Study Says" (not, it doesn't, BTW)
• "Verizon Business Report Speaks Volumes" (from Richard, thus a MUST read)

And of course, here is my favorite part: "In 82 percent of cases, our investigators noted that the victim possessed the ability to discover the breach had they had they been more diligent in monitoring and analyzing event-related information [AC - i.e. logs] available to them at the time of the incident." and this  "Furthermore, a crime scene devoid of any network and system logs, a key resource for computer forensics, is a disturbingly common occurrence."

What can I say? Back to battle stations for me - to fight the war of making logs more popular! :-)