A similar arrogance is happening in CEP-land, where, it seems, each and every financial services event processing application is now a “CEP application” just because someone in capital markets puts “CEP” in the same paragraph. I find it ridiculous that the same market of folks who have helped destroy the global economy are now the world’s self-proclaimed authorities on complex event processing. Amazing, if you really think about it, isn’t it?

I read many posts these days by folks in the capital markets trading world, claiming their message routing application is “CEP,” or their algo trading application is “CEP,” - feeds and speed, typical of what “turns on” the financial services folks. As an editorial note: I recall when I worked for a software company, folks on the same team who worked on Wall Street would look down on folks with many years of IT experience outside of financial services. Some would say “he is only a security guy” in their attempt to put down anyone who does not have trading floor IT experience on their resume. I found it all quite ridiculous and foolish.

My resume, for what it is worth, has a number of financial services companies, including either assessing, architecting or building large scale security systems for S.W.I.F.T, Chase or SBC. This experience does not seem to “count” with the trading floor folks, since security is more about getting things right, not just supporting a form of gaming or gambling with other peoples money, with more feeds and speeds the better.

Of late, as I have watched the CEP/EP space evolve, and unfortunately, I see a similar type of “capital markets virus” spreading into CEP-land. Folks on the trading side of financial services seem to think that whatever they say or do is right, and whatever others outside of the trading side do is wrong. These folks are quick to ridicule others who have far more experience than they do, outside of the trading floor of capital markets.

After all, mostly what they do on the trading side is route orders - and if a little old lady in a small town in Iowa loses her life savings because of a bad investment decision, it means little to the folks on the trading floor, the market folks are into feeds and speed - just keep the beast alive. Place your bet on this market or that one! Away we go, faster and faster!!!!

I am sometimes a little sad to observe the same audacity in the CEP world. Instead of focusing on the hard complex problems that require accuracy, the original set of problems defined when the phrase “complex event processing” was minted, the capital market folks have hijacked the term for their marketing purposes in algo trading and order managment systems. These same people ridicule others who are working to solve the (originally stated) complex event processing problems, problems the capital market traders seemingly cannot understand, since they have never worked on complex network or security management problems.

Nevermind, that these “ultra low latency” systems cannot accurately detect a complex money laundering scheme or an elaborate fraud. Nevermind that these “CEP engines” cannot accuracy insure that Average Joe does not lose his hard earned money in a fraud scheme.

I have no problem with folks in capital markets using the term CEP, but they should not ridicule those in technical areas that are not focused on keeping the “trading beast” alive so people can lose their life savings in a blink of an eye; but instead focused on solving complex problems such as the class of problems called out when the three letter acronym “CEP” was created.

ATC is an excellent working example of complex event processing.   Radar and GPS provide the basic sensory information to accurately track and trace the position of each aircraft in the area of responsibility (AOR) of a particular control tower/zone.     Naturally,  sensory information is preprocessed and formatted in such a way that the data can be processed upstream by multiple real-time applications.

Before we look at complex ATC scenarios, such as “potential collision” or “aircraft off approach vector” we must trace and trace individual objects, aircraft-objects, accurately with very high confidence.    In addition to tracking aircraft-objects, there is a database of information about the aircraft (ideally), such as make, model, age, range, passengers and other properties about the aircraft-object.      In addition, there is a state-model for each aircraft, for example the aircraft might be “on the ground”, “approaching the runway”, “cleared for takeoff”, “cruising altitude”, “approaching runway”, “final decent” etc.  

Tracking and tracing individual aircraft is what is generally referred to as “object refinement” in our CEP/EP reference architecture.   The reason we call this function “object refinement” is that system engineers are focused on optimizing the situational knowledge about individual objects.     Sometimes we refer to this function as “track and trace” because that is what we are doing to  each object in the model.  In Marc Adler’s recent shoplifting scenario, Marc was interested in tracking and tracing people in a store using imaging processing techniques to estimate their behavioral patterns.  In the same way, before we can process for scenarios such as “potential shoplifter” or “suspicious criminal gang activity” we must be able to accurately process (track and trace) individual object, such as people or merchandise.

Back to aircraft and ATC, the “complex event processing” begins when we are looking about object-object relationships, in this model, aircraft-to-aircraft, but this is an overly simplistic model, as we have not yet added (to our model) ground features (towers, buildings, power lines), weather (storm cells, wind) and other flying objects (known migratory bird paths, swarms of insects) to our simple model.  

Complex event processing occurs when we are processing multiple objects in our model looking for threats in real-time.     Practically speaking, all ATC applications are CEP applications.  This means that vendors and integrators who build ATC applications are also CEP vendors.   

Editorial Note: CEP/EP has been around for a long time and was not recently invented in the past decade as some “inventors” would like for us to believe. 

As you can imagine, there is considerable “complex event processing” that goes on “behind the scenes” to provide air traffic controllers and pilots situational knowledge into the “friendly skies”.   As you might further imagine, the situation is more complex when the skies are “not so friendly”, for example, in air combat situations.   

Processing myriad objects is not the end of the processing “chain”.  For example, decisions are being made constantly about potential damage, alternative airports, and more.    In our reference model, we refer to this, generally speaking, as “impact assessment” because we must take an estimated detected complex event, for example “aircraft collision,” and estimate potential damage based on numerous factors such as, the amount of jet fuel in the aircrafts and the location of the aircrafts (over a large city or rural area, near a hospital and emergency services).   Regardless of the scenario, an impact assessment is normally required before optimal decisions can be made.

This is true, by the way, for our shoplifting example (the impact is different if a piece of gum is stolen versus a $1,000,000 diamond necklace or weapons-grade nuclear material) and other scenarios and models.  Static data (information about objects) is required for accurate decision processing.  

Impact assessment is not the end of the “knowledge chain”.    Decisions are constantly being made that effect resources.  For example, suggestion an alternative route for an aircraft is a resource management decision.    Turning on and off radar or switching to alternative tracking devices is a resource management function.  In our CEP/EP reference model (based on the JDL data fusion model), we call this “resource management”.   This function includes contacting emergency services and directing them to a potential crash location or sending out a message to instruct all aircraft to stay off a certain radio frequency.  Resource management is critical.

Our simple ATC model today is by no means complete, it just scratches the surface.  In fact, I have a very close friend, Mark Secrist, who is a former Marine fighter pilot and currently a senior captain for American Airlines.   I have asked Mark to read this post and help me further refine this crude “laymans” ATC model (Thanks Mark!).

Marc’s initial shoplifting model in his post is based on John Colapinto’s concepts of matching a pattern of customer movements in the store with their estimated patterns of shoplifting behavioral patterns.    Marc’s asks how Coral8 might address this.   We are not ready to seek a vendor solution.  We do not yet have a workable detection model.

As indicated above, I don’t think the example situation cited by John and Marc is a viable model for automated processing.    Tracking the behavior of customer’s movements, by machine, would require some very sophisticated image processing technology that would be too expensive compared to any possible loss at most retails stores.    This type of behavioral pattern recognition. in retail stores, is performed by people (security personnel), not machines, observing people.  

To develop a machine pattern recognition application to detect retail shoplifting we need to build detection models that are economically feasible.  If we are going to use a model of shoplifting pattern recognition versus anomaly detection, we need to define the objects we must track.  

In the most simple model, we have merchandise-objects.   Stores normally (physically) track merchandise-objects only at the exit/entry points of the store using some electromagnetic proximity detection technology.   In this model, the detection configuration is a combination of simple alerting with humans watching the store (”minding the store”).    This is not complex event processing.

However, if we added another object to our model, the customer-object, then we start to get more “complex,” but we have not defined “complexity” yet because we have not defined the object properties, the possible states of the objects, and the relationships between the objects that are the basis for estimated situations.

Hence, model building is constrained by available resources, simple economics and risk (cost-benefit).  If we are detecting shoplifting in Walmart the cost-benefit model for implementing an automated shoplifting detection system would be different than at a top diamond store on 5th Avenue in NYC.   Protecting loss at a weapons-grade uranium respository follows a different model than protecting loss at a handicraft shop, naturally.

Like Marc, I find models to automatically detect shoplifting interesting, so permit me to close with a general discussion of shoplifting in the context of our CEP/EP reference model.

One approach would be do determine what objects will be represented in our model.   For example, if we are going to track merchandise, we need to model the ”merchandise-object”.  If we are going to track people, we need to define the properties of this “person object.”  If we are going to represent the store layout, we need to define all these objects (store-object, table-object, shelf-object, entry-object and so forth).  The model can get “complex” quite quickly. 

Editorial Note:  An object-oriented approach greatly assists complex model building because we can benefit from OO properties such as encapsulation and polymorphism.  For example, we can define a basic “person object class” and then create superclasses of this object for “customer-object”, “manager-object”, “or criminal-object.”

Generally speaking, each object we define will require a state-model, for example, in Marc’s example of a customer moving around the store, we would need to model the possible states (customer at the entrance, at table 1, at table 2, at shelf 1, in the bathroom, at the cashier, etc.)  Indeed Marc, this is complex event processing if we have modelled multiple objects and defined object-object relationships that indicate situations of interest.   For example, customer-object at table2 where merchandise-object has the property of  ”very expensive, high risk” and then customer-object changes state to “in bathroom”.  Of course, we need more key indicators, but you get the idea.

Right now, I am typing from the Taste from Heaven Vegetarian Restaurant in Chiang Mai and my battery is running low.  The owner of this excellent restaurant also runs the Elephant Nature Park, a non-profit organization advocating and acting on behalf of the rights of the mighty elephants in Thailand.  Would be great if we could also automatically detect the situation of “elephant abuse” by poachers and other crimes against nature.   Time to get back to my delicious mushroom salad, Northeastern Thai style.

As always, thanks for reading, time for me to get back to eating!

One source at the Knight Center for Digital Excellence notes, "The new network owners are supposed to have a much more sustainable business model." Supposed to. Later, "Network Acquisition Company, which acquired the network, hasn't talked publicly about the details of its new plan, but it has hinted that its strategy will differ from EarthLink's." Hasn't talked publicly. Then, "[NAC and Tropos] spokespeople said the companies would talk more about the network later this month when details of the new business plan are ready." Huh.

Reardon explains digital divide issues and looks into what Wireless Philadelphia has been up to, although doesn't note that delays in EarthLink's deployment and other factors have led to just a few hundred individuals that have been assisted by the non-profit; numbers may have changed, but that was as of a few months ago. Still, Wireless Philadelphia has apparently diversified its funding sources--Reardon cites 30 now.

I think we're still coming off the doldrums of August.

As we get closer to the conference dates, I wish that I had made plans to fly back to the US to meet everyone.    However, I have been cutting back on public speaking, taking a break since May.  In addition, Gartner did not ask me to speak at their Event Processing Summit this year, I assume because they did not want to pay airfare for my flight from Thailand to the US.    Also, Gartner always likes to fill their conference speaking slots with as many Gartner speakers as they can, unless you are a paid sponsor; and I noticed a number of Gartner employees speaking in multiple slots.

(Editorial Note) Then again, maybe I complained to much about the lack of organization and conference problems when I was invited at be a Gartner keynote speaker last time - reservations not made propertly,  problems with the guest speaker registration list at sign-in, rooms shifted without notifying the speakers and panelists.   Admittedly, I was not happy with the conference organizers at the last get together.  This was my fault, as I am accustomed to better conference execution and am probally too “picky” about details these days - my bad.  Anyway, the Gartner organizers apologized numerous times, saying they had too many conferences going on at the same time and not enough people to cover them all.

One of the problems with spending so much time in Asia, especially in Thailand, is that guest speakers are really treated as VIPs.  There are usually special comfy couches set up for the speakers and the conference staff really treat you very nice, taking care of you every step of the way.   In fact, there is an entire very nice culture around how guest speakers are treated in Thailand.   Often, they pin flowers on the VIP speakers and take your photos like you are a star.    Very nice culture.

I absolutely look forward to speaking on event processing or CEP at a future venue and meeting everyone face-to-face instead of over the net.  My sincere and deepest apologies for not attending either the Gartner or the EPTS event this year.   

PS:  If you take up a collection and send me a RT business class air ticket, I might change my mind

I would guess that every one of you reading this blog have a stock portfolio with 5 to 10 individual stocks or mutual funds. There are more than 5,000 publicly listed companies to choose from, and another 5,000 mutual funds. But, out of 10,000 possible companies you chose 10 to invest in. Why? Why did you reject the other 9,990 companies? Obviously there are more than 10 good companies to invest in. Other investors chose to invest their money in the other 9,990 companies…why not you?

I suppose the difference must be that many investors aren’t actively involved in their investments (maybe entrepreneurs are more so, since they have to know a certain investment space quite well)…

It sounds to me a lot like the editorial selection process for book manuscripts, articles, and so forth — editors receive a ton of submissions and they have to be choosy. Sometimes they don’t pick winners; sometimes they pick losers. More importantly, each has a personal style, opinions, preferences, and they are trying to appeal to a certain audience. It’s interesting to think that VCs are similar but makes sense–the end question of “What will be successful” really depends on the consumer base and industry, and VCs are just people who probably know and prefer to interact with a certain type of consumer base or audience.