Marc’s initial shoplifting model in his post is based on John Colapinto’s concepts of matching a pattern of customer movements in the store with their estimated patterns of shoplifting behavioral patterns.    Marc’s asks how Coral8 might address this.   We are not ready to seek a vendor solution.  We do not yet have a workable detection model.

As indicated above, I don’t think the example situation cited by John and Marc is a viable model for automated processing.    Tracking the behavior of customer’s movements, by machine, would require some very sophisticated image processing technology that would be too expensive compared to any possible loss at most retails stores.    This type of behavioral pattern recognition. in retail stores, is performed by people (security personnel), not machines, observing people.  

To develop a machine pattern recognition application to detect retail shoplifting we need to build detection models that are economically feasible.  If we are going to use a model of shoplifting pattern recognition versus anomaly detection, we need to define the objects we must track.  

In the most simple model, we have merchandise-objects.   Stores normally (physically) track merchandise-objects only at the exit/entry points of the store using some electromagnetic proximity detection technology.   In this model, the detection configuration is a combination of simple alerting with humans watching the store (”minding the store”).    This is not complex event processing.

However, if we added another object to our model, the customer-object, then we start to get more “complex,” but we have not defined “complexity” yet because we have not defined the object properties, the possible states of the objects, and the relationships between the objects that are the basis for estimated situations.

Hence, model building is constrained by available resources, simple economics and risk (cost-benefit).  If we are detecting shoplifting in Walmart the cost-benefit model for implementing an automated shoplifting detection system would be different than at a top diamond store on 5th Avenue in NYC.   Protecting loss at a weapons-grade uranium respository follows a different model than protecting loss at a handicraft shop, naturally.

Like Marc, I find models to automatically detect shoplifting interesting, so permit me to close with a general discussion of shoplifting in the context of our CEP/EP reference model.

One approach would be do determine what objects will be represented in our model.   For example, if we are going to track merchandise, we need to model the ”merchandise-object”.  If we are going to track people, we need to define the properties of this “person object.”  If we are going to represent the store layout, we need to define all these objects (store-object, table-object, shelf-object, entry-object and so forth).  The model can get “complex” quite quickly. 

Editorial Note:  An object-oriented approach greatly assists complex model building because we can benefit from OO properties such as encapsulation and polymorphism.  For example, we can define a basic “person object class” and then create superclasses of this object for “customer-object”, “manager-object”, “or criminal-object.”

Generally speaking, each object we define will require a state-model, for example, in Marc’s example of a customer moving around the store, we would need to model the possible states (customer at the entrance, at table 1, at table 2, at shelf 1, in the bathroom, at the cashier, etc.)  Indeed Marc, this is complex event processing if we have modelled multiple objects and defined object-object relationships that indicate situations of interest.   For example, customer-object at table2 where merchandise-object has the property of  ”very expensive, high risk” and then customer-object changes state to “in bathroom”.  Of course, we need more key indicators, but you get the idea.

Right now, I am typing from the Taste from Heaven Vegetarian Restaurant in Chiang Mai and my battery is running low.  The owner of this excellent restaurant also runs the Elephant Nature Park, a non-profit organization advocating and acting on behalf of the rights of the mighty elephants in Thailand.  Would be great if we could also automatically detect the situation of “elephant abuse” by poachers and other crimes against nature.   Time to get back to my delicious mushroom salad, Northeastern Thai style.

As always, thanks for reading, time for me to get back to eating!

Solace provides sophisticated middleware functionality in hardware to monitor, filter, route, transform and secure very large volumes of events in real time and with minimal processing overhead.  Solace uses leading-edge FPGA, ASIC and network processor technology to increase throughput and lower latency of event processing. Applications such as fraud detection, algorithmic trading, compliance, insider trade monitoring, risk management and more can be tackled more effectively by separating the simple monitoring, filtering and normalization of raw events from the complex processing of select events. This event pre-processing takes the burden off CEP engines allowing individual engines to be much more effective. 

I agree that these gentlemen are top notch researchers, witnessed by the fact that the authors do not mention nor claim to be “complex event processing” anywhere in their paper!  This paper is not about CEP, nor does it claim to be about CEP, it is about stream processing and unifying SQL standards.

ABSTRACT: This paper describes a unification of two different SQL extensions for streams and its associated semantics. We use the data models from Oracle and StreamBase as our examples. Oracle uses a time-based execution model while StreamBase uses a tuple-based execution model. Time-based execution provides a way to model simultaneity while tuple-based execution provides a way to react to primitive events as soon as they are seen by the system.

Asmentioned on numerous occasions, stream processing is a very important area in CEP/EP.   It is important not to confuse the higher situational knowledge from object-object correlation and state management with the single-object event refinement that occurs in stream processsing.    Event stream processing is fundamentally different than complex event processing. 

Event stream processing performs operations on streaming event objects.   In almost all advanced CEP/EP applications is is necessary to perform robust track and trace operations on streaming event objects, like tracking the position of an airplane.    Tracking the position of an aircraft can be modelled very nicely with event stream processing.  Tracking individual event objects is a precuror to multiclass object situation refinement.

When we manage the state of all the aircraft in the skies over New York, you need more than a stream processing construct.  You need to manage the state of all the aircraft.  Paul Vincent of TIBCO Software being to address this important point in The Value of State…  

Again, we will be better equiped to solve complex distributed event processing problems if we do not confuse the notion of event stream processing and complex event processing.   These technologies are indeed complimentary, both very important, but they are not the same.

I applaud Oracle and StreamBase’s work toward a unified standard for SQL extensions for streams.

I arrived in Austin Sunday night to get ready for a factory tour on Monday, a kickoff dinner and then two days of briefings from Dell executives, including Michael Dell himself! Dell’s ISG business is growing at a very fast pace and continues to build momentum and focus within the broader organization.

We had a nice overview of the product roadmap, including some of the exciting enhancements Dell is making to their storage products such as the MD3000 and the new EqualLogic PS5000 series iSCSI solutions.

I really enjoyed the Council meeting and it reminds me all over again; what I admire about Dell is the way they and Michael Dell himself stay close to the customer. The entire purpose of this event is to “get it right” and determine meaningful ways to embrace change (including change in the manufacturing process) in order to make their customers more successful. Ah shucks, you may say that all companies behave this way… well I must tell you that is not true and at times, I find it difficult as we continue to grow to stay as close as I would like to all of our customers varying needs and directions.

This concept of gathering, internalizing and embracing customer feedback is a simple principle of Business Success stories. Always trying to improve the pace of change and build meaningful sticky relationships with customers. Dell’s very successful Ideastorm site where customers post product feedback and are active participants in the Dell community is a great example of how to do this right. No other hardware vendor that we have worked with or attempted to work with has ever gone to the extent of embracing change that Dell has during our 5-year relationship.

From the custom factory integration services to the attention to detail in the order and manufacturing, and logistics processes, Dell helps us execute for our customers and I must admit that we could not have built the business as quickly or efficiently without Dell!

So thank you Michael Dell for building a business that embraces change and is focused on helping your ISG customers succeed.

The following excellent discussion is reposted from Streaming SQL approaches insist in ignoring causality by PatternStorm.

The recent paper “Towards a Streaming SQL Standard” by Oracle and Streambase unifies and generalizes two different execution models of Streaming SQL: Oracle’s and StreamBase’s.

While it’s true that the generalization succeeds in overcoming the unability of both execution models of producing correct results for astonishing simple queries (showing evidence of the actual limitations of these two Streaming SQL languages) it is also true that the generalization is closer to being overly complex than natural and intuitive.

The root cause behind the actual limitations of these two Streaming SQL languages is that their execution models “hardcode” the way events can be related to each other: in the Oracle case events are partially ordered by timestamp, in the StreamBase case events are totally ordered by time of arrival. These design decisions (natural in a stream oriented lamguage) have strong implications on what queries can be answered correctly, particularly when these queries involve joins of derived streams.

The generalization, of course, mainly consists in providing a new operator that allows the user to establish custom ordering relationships among the events (the SPREAD operator), which is good news but takes us to the fundamental issue: event processing cannot be reduced to stream processing, that is, to the processing of events that are totally or partially ordered by a pre-defined relationship (as Oracle and StreamBase actual implementations do), on the contrary, no particular ordering can be assumed because the user needs to be able to order the events in different ways in order to solve different problems. This is what event processing is about and the paper provides evidence that Streaming SQL approaches have found the need to move towards that direction and are having trouble in their way.

For instance, one of the queries used in the paper as an example of a query that StreamBase cannot solve (but Oracle can) is the following: correlate the stream that contains the total number of cars on the road for each time interval with the stream that contains the total average speed of the cars on the road for each time interval in order to detect the situation where the avergae speed is below 45 and the total number of cars is two or more. This query can be very easily and more robustly solved if you order the events by causality rather than by time, that is, if you have each position report update the average speed stream and the total number of cars stream and then you causally relate each position report to the new average speed event and the new total number of cars event that it generates; then the query is just a matter of detecting all report speeds that are causally related both to an average speed event below 45 and a total number of cars event of two or more (notice that this approach is more robust than Oracle’s time-based one because it works without requiring derived streams to be synchronized with the report speed stream)

Conclusions:

• Event Processing is a generalization of Stream Processing (as the paper shows)
• Event Processing requires providing the ability to the user of creating custom relationships among events and then define patterns/queries using those custom relationships.
• Causality is more often than not a more robust and easier criteria to order events than time or order of arrival.
• Event Processing Languages should support causality.

Regards,
PatternStorm

First of all, let’s ground the discussion a bit by translating “smart order routing” to “rule-based message routing” since in this application “smart”  translates to “using rules” and “order” translates to “message”.    Basically, Mark (and other “new on the routing scene” stream processing players) argue that rule-based message routing is CEP.  I will argue that routing is not even close to CEP.  Here is why,

Let’s take a look at a router on the backbone of the global Internet.   A backbone router has very sophisticated software developed over many decades.   These routers run sophisticated, mature algorithms to determine how to route messages (packets) and use these algorithms to build complex routing tables. 

In addition, these routers process messages (packets) from countless sources and route messages (packets) to countless destinations.  Using some of the terms in early posts (above), there is a great “confluence of events” processed by routers.    Futhermore, there are normally quite complex authentication, authorization and other security parameters managed in a router, all in real time.   Routers do much more, but I don’t want to get too deep into routing in this post.

My point is that, without any doubt, global Internet routers process very “cloudy” “confluence of events” with much more sophistication than order routing applications.    However, we do not call Internet routing “CEP”, regardless of how many connections are processed or how much sophisticated processing occurs.  The reason is because the “C” in “CEP” defines a complexity that is at a higher abstraction than messaging and routing.

If you study the literature on CEP, some of which I posted recently, CEP was envisioned to solve complex event processing problems “on top of the routing layer” because the routing layer is a mature technology layer.  We can route, pure and simple.  Of course, we are always seeking faster, more scaleable and more secure routing. 

I admire some of the startups in the CEP/ESP/EP space for working hard to make money and for aggressively positioning their products and attempting to build market share.   However, issues surface when these same companies seem to believe they are the first companies to work in the event processing or message routing space and that they can define whatever they want as “complex event processing” as long as it benefits their sales targets.

There is no doubt that a router does much more sophisticated event processing than the new rule-based stream processing systems running continuous queries across streaming data.  There is no doubt that a router processes a complex “confluence of events”.   However, we don’t call routers “CEP”. 

We do not call routers “CEP” because CEP is about a higher level of knowledge processing.  CEP was created to detect the “complex events” that happen above the mediation and routing layer.     The literature and original examples on CEP are quite clear on this.

I agree with Mark that SOR is important and very interesting; but in his reply he seems to be confusing CEP with “complex EAI” or a “complex messaging” application.  For example, Mark says,

“It’s not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it’s a stunningly complicated environment in which to create a complex, real-time model in which to apply “simple” routing decisions. On this basis alone, SOR needs CEP.”

Connecting to many market feeds with multiple asset classes might be complicated, but “complicated connections” are an EAI  (adaptation layer) function, not a core CEP function.   In fact, TIBCO Software has been doing this type of low latency back-office order routing for many years, and TIBCO historically calls this “messaging.”  Adding some rules to high speed, low latency messaging does not make it a “CEP” application.

Mark goes on to set up a counter argument to ILOG’s Changhai Ke, comments with,

“SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.”

This is the well travelled argument the “new stream processing vendors in capital markets” have been saying, still unconvincingly, for the last few years.  Basically their perspective is that if you have a lot of ”feeds” and a core requirement for “speed” - “feeds and speed” - you are doing “complex event processing.” 

Mark Palmer forcefully stated his opinon that the folks who do not agree with him do not “understand” modern day SOR.    However,  a strong counter argument can be made that the “newcomers” to capital markets like StreamBase do not understand that “feeds and speeds” with order routing is little more than moderan day EAI.   This is a basic message routing capability and it has been around for a long time.  After all, Wall Street operated quite well before the term CEP was coined!  TIBCO technology was providing Wall Street back office, low latency, smart order routing a decade ago, and they called this technology “messaging”.  

So, I remain unconvinced, at least by Mark’s passionate counter post, that SOR is CEP.   SOR, as Mark and other have described it, is a low latency messaging technology.  Message routing rules have exisited in this technology space for decades.

I agree with Mark completely that low latency EAI (like SOR has been described) can be quite complex, from a “feeds and speeds” perspective.   However,  I remain skeptical that “feeds and speeds” is much more than  modern day messaging and message routing.

In closing, in the network and security management world we have been dealing with “myriad feeds and speeds” for as long as I can remember, but admitted not like capital markets.    Taking myriad feeds, running rules against the feeds and then routing the messages/events for further processing, regardless of the complexity of the feeds and the data, is actually more of a messaging/ESB technology than a CEP technology. 

I remain completely open minded to any convincing counter arguments.

It's become a big deal in IT security, too. Many corporate customers are demanding ROI models to demonstrate that a particular security investment pays off. And in response, vendors are providing ROI models that demonstrate how their particular security solution provides the best return on investment.

It's a good idea in theory, but it's mostly bunk in practice.

Before I get into the details, there's one point I have to make. "ROI" as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It's an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn't make sense in this context.

But as anyone who has lived through a company's vicious end-of-year budget-slashing exercises knows, when you're trying to make your numbers, cutting costs is the same as increasing revenues. So while security can't produce ROI, loss prevention most certainly affects a company's bottom line.

And a company should implement only security countermeasures that affect its bottom line positively. It shouldn't spend more on a security problem than the problem is worth. Conversely, it shouldn't ignore problems that are costing it money when there are cheaper mitigation alternatives. A smart company needs to approach security as it would any other business decision: costs versus benefits.

The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money.

Of course, that $1,000 has to reduce the chance of being robbed to zero in order to be cost-effective. If a security measure cuts the chance of robbery by 40 percent -- to 6 percent a year -- then you should spend no more than $400 on it. If another security measure reduces it by 80 percent, it's worth $800. And if two security measures both reduce the chance of being robbed by 50 percent and one costs $300 and the other $700, the first one is worth it and the second isn't.

The Data Imperative

The key to making this work is good data; the term of art is "actuarial tail." If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one.

Cybersecurity is considerably harder, because there just isn't enough good data. There aren't good crime rates for cyberspace, and we have a lot less data about how individual security countermeasures -- or specific configurations of countermeasures -- mitigate those risks. We don't even have data on incident costs.

One problem is that the threat moves too quickly. The characteristics of the things we're trying to prevent change so quickly that we can't accumulate data fast enough. By the time we get some data, there's a new threat model for which we don't have enough data. So we can't create ALE models.

But there's another problem, and it's that the math quickly falls apart when it comes to rare and expensive events. Imagine you calculate the cost -- reputational costs, loss of customers, etc. -- of having your company's name in the newspaper after an embarrassing cybersecurity event to be $20 million. Also assume that the odds are 1 in 10,000 of that happening in any one year. ALE says you should spend no more than $2,000 mitigating that risk.

So far, so good. But maybe your CFO thinks an incident would cost only $10 million. You can't argue, since we're just estimating. But he just cut your security budget in half. A vendor trying to sell you a product finds a Web analysis claiming that the odds of this happening are actually 1 in 1,000. Accept this new number, and suddenly a product costing 10 times as much is still a good investment.

It gets worse when you deal with even more rare and expensive events. Imagine you're in charge of terrorism mitigation at a chlorine plant. What's the cost to your company, in money and reputation, of a large and very deadly explosion? $100 million? $1 billion? $10 billion? And the odds: 1 in a hundred thousand, 1 in a million, 1 in 10 million? Depending on how you answer those two questions -- and any answer is really just a guess -- you can justify spending anywhere from $10 to $100,000 annually to mitigate that risk.

Or take another example: airport security. Assume that all the new airport security measures increase the waiting time at airports by -- and I'm making this up -- 30 minutes per passenger. There were 760 million passenger boardings in the United States in 2007. This means that the extra waiting time at airports has cost us a collective 43,000 years of extra waiting time. Assume a 70-year life expectancy, and the increased waiting time has "killed" 620 people per year -- 930 if you calculate the numbers based on 16 hours of awake time per day. So the question is: If we did away with increased airport security, would the result be more people dead from terrorism or fewer?

Caveat Emptor

This kind of thing is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They've jiggered the numbers so that they do.

This doesn't mean that ALE is useless, but it does mean you should 1) mistrust any analyses that come from people with an agenda and 2) use any results as a general guideline only. So when you get an ROI model from your vendor, take its framework and plug in your own numbers. Don't even show the vendor your improvements; it won't consider any changes that make its product or service less cost-effective to be an "improvement." And use those results as a general guide, along with risk management and compliance analyses, when you're deciding what security products and services to buy.

This essay previously appeared in CSO Magazine.