Marc’s initial shoplifting model in his post is based on John Colapinto’s concepts of matching a pattern of customer movements in the store with their estimated patterns of shoplifting behavioral patterns.    Marc’s asks how Coral8 might address this.   We are not ready to seek a vendor solution.  We do not yet have a workable detection model.

As indicated above, I don’t think the example situation cited by John and Marc is a viable model for automated processing.    Tracking the behavior of customer’s movements, by machine, would require some very sophisticated image processing technology that would be too expensive compared to any possible loss at most retails stores.    This type of behavioral pattern recognition. in retail stores, is performed by people (security personnel), not machines, observing people.  

To develop a machine pattern recognition application to detect retail shoplifting we need to build detection models that are economically feasible.  If we are going to use a model of shoplifting pattern recognition versus anomaly detection, we need to define the objects we must track.  

In the most simple model, we have merchandise-objects.   Stores normally (physically) track merchandise-objects only at the exit/entry points of the store using some electromagnetic proximity detection technology.   In this model, the detection configuration is a combination of simple alerting with humans watching the store (”minding the store”).    This is not complex event processing.

However, if we added another object to our model, the customer-object, then we start to get more “complex,” but we have not defined “complexity” yet because we have not defined the object properties, the possible states of the objects, and the relationships between the objects that are the basis for estimated situations.

Hence, model building is constrained by available resources, simple economics and risk (cost-benefit).  If we are detecting shoplifting in Walmart the cost-benefit model for implementing an automated shoplifting detection system would be different than at a top diamond store on 5th Avenue in NYC.   Protecting loss at a weapons-grade uranium respository follows a different model than protecting loss at a handicraft shop, naturally.

Like Marc, I find models to automatically detect shoplifting interesting, so permit me to close with a general discussion of shoplifting in the context of our CEP/EP reference model.

One approach would be do determine what objects will be represented in our model.   For example, if we are going to track merchandise, we need to model the ”merchandise-object”.  If we are going to track people, we need to define the properties of this “person object.”  If we are going to represent the store layout, we need to define all these objects (store-object, table-object, shelf-object, entry-object and so forth).  The model can get “complex” quite quickly. 

Editorial Note:  An object-oriented approach greatly assists complex model building because we can benefit from OO properties such as encapsulation and polymorphism.  For example, we can define a basic “person object class” and then create superclasses of this object for “customer-object”, “manager-object”, “or criminal-object.”

Generally speaking, each object we define will require a state-model, for example, in Marc’s example of a customer moving around the store, we would need to model the possible states (customer at the entrance, at table 1, at table 2, at shelf 1, in the bathroom, at the cashier, etc.)  Indeed Marc, this is complex event processing if we have modelled multiple objects and defined object-object relationships that indicate situations of interest.   For example, customer-object at table2 where merchandise-object has the property of  ”very expensive, high risk” and then customer-object changes state to “in bathroom”.  Of course, we need more key indicators, but you get the idea.

Right now, I am typing from the Taste from Heaven Vegetarian Restaurant in Chiang Mai and my battery is running low.  The owner of this excellent restaurant also runs the Elephant Nature Park, a non-profit organization advocating and acting on behalf of the rights of the mighty elephants in Thailand.  Would be great if we could also automatically detect the situation of “elephant abuse” by poachers and other crimes against nature.   Time to get back to my delicious mushroom salad, Northeastern Thai style.

As always, thanks for reading, time for me to get back to eating!

1. Orbitz by Streambase (Real-time monitoring)
2. Netbank by Coral8  (Algo trading)
3. LiquidNet by Coral8 (Market monitoring and analysis)

Progress Apama had a few algo trading references, but they (wisely in my opinion) are not (seeming) calling algo trading platforms sales, CEP, in 2008.   This is good (and more accurate) and I applaud Apama for building a great event stream processing platform and not overhyping the phrase “complex event processing” every opportunity they get.    Maybe we should create another award category?    

You will not find any “secret council of elders” here, nor will you find any subjective opinions about the market from people we send out email asking for their opinions - only the facts in an open transparent way.  Here is the Google worksheet, if interested.

Please contact me or comment here if we missed anything and we will take a look and we will add your suggestion if it meets the criteria.

Revisions:

1. Added LiquidNet/Coral8 suggested by Marc Adler (see comments)

“I know for a fact that every major CEP vendor has several dozen paying customers.”

Somehow Mark, I don’t find a dozen paying customers by the top CEP vendors very impressive.

Then, as to somehow justify the lack of public reference clients, Mark takes the position of a Coral8 customer and says,

“We believe that the use of Coral8 gives us a strategic advantage over our competitors. Why would we want to clue them in?”

Naturally, the same thing could have been said about the first desktop computer, or the first back-office banking system, or the first calculator, or the first telephone, frankly speaking.

Of course, when the technology is mature, then it is “Hey we have lots of computers!” “Hey, look at my fully functional sexy iPhone!” “We have the best back office banking systems on the planet by <insert your favorite big vendor here>!”

Well, all this CEP Solution Secrecy (CEPSS) might just be similar to why the government keeps many IT projects a secret;  the main reason is so we don’t know how much taxpayer money they are spending!

So, folks, the debate counterpoint that there is some “Secret Life of CEP” and that the CEP solutions today are somehow changing the way C-Level executives, and corporate America, thinks is just wishful thinking.

Companies don’t need to keep their strong technical solutions a secret. Like, Wow! I am using Coral8 and it is so impressive that I have to keep it TOP SECRET.  (Sorry Mark, nothing personal, you simply gave me a big red target and painted “fire when ready” on it)

Note:  I happen to like Coral8, and Coral8 Studio, as an event stream processing platform.

Back on point, I consider my laptop and cellphone more indispensable than most of the first generation rule-based stream processing engines out there today, and I am sure most CEOs agree.

The Secret Life of CEP….   you just have to just love it

The two CEP/ESP related Water’s categories were, Best Streaming Data Management Solution and Best Complex Event Processing Solution.    Water’s awards Best Streaming Data Management Solution to data/event stream processing company StreamBase; and then awards Best Complex Event Processing Solution to Oracle’s BEA product, which is built on top of another data/event stream processing engine.  Confused?   

Alexander Alves,  currently employed by Oracle, previously having worked for BEA Systems, in Best Complex Event Processing Solution, observes:

“Regardless, I find it intriguing that Waters not only does not state the differences between the categories, but also uses the term CEP several times in the SDMS category.

I guess the verdict is that there is still confusion amongst the experts regarding event and stream processing… And that both products must be very good.”

Of course, Alex must be politically correct, and rightly so, since he works for Oracle/BEA and Water’s gave them an award.  But on what tangible, objective basis for Best Complex Event Processing Solution?

According to our 2007 survey, CEP/EP Reference Customers 2005-2007, BEA was in last place based on public CEP/EP reference clients.  

Waters’ award  for Best Streaming Data Management Solution to StreamBase is a much more credible.   Congratulations StreamBase.   Most would agree that StreamBase is a streaming data management solution (SDMS), but so are Apama and Coral8 (and BEA etc etc).   

Waters simply muddies the water, unfortunately.  

In On the Maturity of Complex Event Processing, the author concludes:

“I think [... the. ...] comment at the end of [... the. ...] post “we shouldn’t feel compelled to thwart that growth with a claim that the products are not ‘mature’ when they actually are in a lot of ways” is quite revealing. The fact that such a level of debate about CEP’s maturity is taking place, and the fact that [... someone ...] is concerned that the debate might stifle growth, is itself indicative of an immature market segment in my opinion.”

This quote is compelling.  When vendors disagree with the direction and tone a debate is going and they call to end the debate, labelling the discussion “a distraction” - it tends to prove the premise of the original post Deciphering the Myths Around Complex Event Processing  by Ivy Schmerken;  the CEP market, both exciting and promising, is today, mostly immature and brittle. 

For more conclusive evidence, I turn our readers attention to this post, An Overture to the 2007 CEP Blog Awards,  That analysis was based, in part, on CEP/EP Reference Customers 2005-2007 where we documented 18 public “CEP reference clients” in 2007 (27 for the entire period 2005 - 2007).

Twenty seven public reference clients over a three year period with 18 last year (2007) do not demonstrate a mature market or technology domain.

————————

Footnote:

Here were the results of the CEP/EP Reference Customers Survey” for 2005-2007:

Reducing complex problems sets to simple problem sets is an interesting, and sometimes valid, approach to complex event processing.    Transformations can be useful, especially when well defined.

For example, CEP was evisioned as a new technology to debug relatively large distributed systems, discover hidden causal relationships in seemingly disconnected event space.    This “discovery” requires backwards chaining with uncertainty, for example.  Most of the current so-called “CEP software” (on the market today (including Marc Adler’s SQL-based examples) do not perform backwards chaining (with uncertainty).   This is also true from other so-called CEP products, like most forward chaining RETE engines - for example, see this post.

Marc Adler says he is, “hunting for advice from people who might have implemented event clouds in Coral8, Streambase, and Aleri, all three which are based on SQL.”

Current steaming SQL engines cannot model true event clouds without reducing the cloud to causal-ordered sets of linear steaming data.   These software tools are stream processing engines that process events in a time window of continuous streaming data.  These products are not, in reality, CEP engines - calling them ”CEP engines” is marketing-speak, not technology-speak!

Reducing complex models to simple ones is a valid technique for problem solving.  Likewise, eliminating uncertainty and assuming causality is a way to reduce complexity. 

CEP was envisioned to discover causal relationships in complex, uncertain, ”cloudy data” and the current state-of-the-art of software from the streaming SQL vendors do not have this capability, unless you reduce all event models to ordered sets of streaming data (reduce POSETS to TOSETS).

Reductionism can be a valid technique, of course.  We can eliminate uncertainty, force strict determinism, demand apriori system rules and perform all sorts of tricks to permit us to reduce complex problems to simple ones.  

However this also resulting in reducing CEP  (complex event processing) to SEP (simple event processing).  

In Coral8 is Our Choice or “How the Hell Did We Get Here?”, Marc Adler does his normal (and now expected) fantastic job of cutting past the CEP marketing hype and getting to the meat of the issues, from an actual user’s perspective.  Marc is spot on in his evaluation of the various so-called CEP vendors.   I highly recommend you read Marc’s post above.

The bottom line, today, is that CEP software products have a long way to go to live up to the current CEP hype and none are really doing what we would call “CEP”.   So, in the current market, the intangibles, as Marc points out, are critically important.  

Coral8 has recently demonstrated to the event processing community that they are above-and-beyond the competition in that category. 

Coral8 has an open software evaluation and licensing model, one you would expect in the year 2003-3005 (this is 2008). 

Coral8 has significant white papers, thought leadership papers and documentation, all freely and readily available. 

Coral8 is standing by to support you in your event processing efforts, from Marc at the big and powerful Citigroup (be careful of your subprime portfolio) to consultants in Asia (be careful of mosquitos), you can count on Coral8’s leadership to support you.

As Marc keenly pointed out, it is not the final imaginary number in low latency that is important; nor is it important that you call yourself the “top leader” and the “creator of the standards” that makes you important; nor is it how innovative or smart you are (or think you are).  What is important is your customer service model.

Coral8 has demonstrated to many of us that they take the customer service model very seriously and this is the reason that Coral8 has caught our attention in the past 6 months.

Congrats to both Coral8 and Marc.   We look forward to hearing more about the results of your teamwork and event processing solutions at Citigroup.

Mark Tsimelzon, President & CTO, Coral8 replies to CEP Product Complexity at Coral8 with More on CEP and Complexity.

In Mark’s reply he gently reminds us that the Coral8 Engine is a developer’s tool, not a business users tool.  Mark also tacitly reminds us that his customers are a bit smarter than our development team at Techrotech.  Perhaps that is why we can never get our SOA project off the ground!  

Mark’s customers can learn a new concept in a single day; however, our developers need a full week to learn the same thing.   Making matters more difficult, our CIO at Techrotech, Jerry Fleck is clueless according to the marketing analysts.  Jerry has not yet figured out SOA; so concepts like windows, joins, design patterns, causal tracking, messaging layers, adapters, CCL, and persistance will cause Jerry to fall Off the Grid.

It took us years to get rid of most of our legacy C programmers, bring in a bunch of Java gurus, and at the same time, correct our stock option backdating “clerical errors”.    Maybe we should now replace our worthless legal and HR departments with a CEP Engine?

In What makes a Coral8 Expert?, Coral8 CTO Mark Tsimelzon outlines nearly 60 subject areas that a customer must master to become a Coral8 expert. 

While this complexity is impressive, it tends to demonstrate why CEP is, today, more hype than reality.

I can hear the team at Techrotech in my mind, “Yea! Greg purchased Coral8 for our CEP solutions yesterday!   Holy Cow!! Let’s go out and learn 60 topics in depth so we can become experts in using and deploying Coral8!”

So, let’s say you are intelligent and can master a subject in a single weeks time (if you have nothing else to do), so you can become a Coral8 expert in only one year if you don’t have a day job!!

I don’t know about you, but Coral8 sounds more like a lab tool for the engineeing department of Caltech or Stanford than a tool for everyday business users, based on Mark’s post, where he concludes:

Not too scary, is it?  - Mark Tsimelzon, President & CTO, Coral8

Hmmmm.   I think I’ll ask the software team at Techrotech to write some event processing applications in C since we have a strong team of C programmers coming off another project next week….