A good place to being to look for clues to an answer is Wikipedia, where the opinion of the author there is,

 ”A simple and practical definition, however, would be how an entity (i.e., business) arrives at an optimal or realistic decision based on existing data.”

Quoting the Wikipedia author(s) further,

“Common applications of Analytics include the study of business data using statistical analysis in order to discover and understand historical patterns with an eye to predicting and improving business performance in the future. Also, some people use the term to denote the use of mathematics in business. Others hold that field of analytics include the use of Operations Research, Statistics and Probability. However, it would be erroneous to limit the field of analytics to only statistics and mathematics.”

The Wikipedia author(s) continue their discussion of analytics, as follows;

“Analytics closely resembles statistical analysis and data mining, but tends to be based on modeling involving extensive computation. Some fields within the area of analytics are enterprise decision management, marketing analytics, predictive science, strategy science, credit risk analysis and fraud analytics.”

All of these topics above are CEP-related areas involving complex events and situations based on the need for optimal and reliable real-time capabilities to make meaningful (business) decisions. 

Simple pattern matching, event mediation and routing, and basic mathematical calculations do not really fall into the realm of complex event processing.  Instead, CEP is real-time decision support based on modeling and “extensive” computation.  In a nutshell, complex events and situations require analytical models that are non-trivial and that is why without analytics, there is no true “complex event processing.”

See also:

WIkipedia on Predictive Analytics

He’s a bit indignant that so much energy goes to sporting events like the Olympics rather than more important news that isn’t getting reported around the world.

This is in a year when tons of journalists are getting laid off.

This is in a year when there are tons of stories around the world that aren’t getting reported on.

Could we take half of those photographers and send them to Russia, for instance

Reminds me of a feeling I had back in college as an undergrad student studying social sciences and humanities, about the way my friends who were physicists interacted with the world. They were so awed by the stars, Mars, astrophysics, and it seemed to me interesting but altogether unimportant. They argued they may find something outside our planet that could help solve Earth-bound problems like disease, or find the origins of earth and humanity — but really they were doing it because they loved it. One of my friends had a good argument, though — there are enough people right now that we can specialize in what we care about, and there will still be others covering other topics. He could be a physicist and look into the universe’s origin, while I studied social interaction and writing, and our other friends looked into solving cancer or eradicating invasive plants in the native wetlands. We have to specialize, and there are enough of us to do it too.

I think it’s the same way in journalism — whether it’s sports, celebrity journalism, or coverage of politics and war, there are a lot of opportunities right now for journalists. Of course the business model is changing, and some old-schoolers won’t know how to roll with that, but generations change slowly; we’re learning.

Also, the Olympics is seen as more than a sporting event, it’s also a symbol of world competition and cooperation too — a way for countries to come together and share entertainment globally. I think that’s worth covering.

In the second post, Robert Scoble says there are plenty of great journalists but the public doesn’t care. In some ways I have to agree with that, but I don’t think it’s negative, necessarily. I had a conversation with someone the other day about world news reportage. He says, “I was just reading this story, but what does it matter to me if there’s a flood in some city in another country I’ll never visit and some farmer lost his sheep?” World news is only important when it’s relevant, so it’s no wonder that many people don’t care — if they don’t know much about the area, and it doesn’t affect them, they have no incentive to give it full attention. You can call that apathy, but I think it’s an important selectivity skill that humans have. We have to choose what to give priority to, so if nothing stands out as being particularly important, we just ignore it or gloss over it. Human nature…

Also I think the common person today just gets desensitized and doesn’t know where to turn their energy, when surrounded by so many crises. Either you focus on one specialty and do your best to work toward one cause in your life — and maybe that’s just in the course of your daily work — or you become a complete Attention-Deficit-Disorder case and bounce from one problem to the next, without knowing how to solve anything. That just causes a sense of bewilderment, despair, and either that bogs you down or eventually you get desensitized.

There’s a commenter on Scoble’s blog, Spencer, who talks about this generation’s apathy. There are so many people who want to blame today’s generation or the young generation for this “apathy” that they sense. But I see it as a survival mechanism that arises from the way information flows these days. We’re surrounded by crises, everyone wants us to know about them — the water shortage, global warming, death in Iraq, the national deficit. Okay, crisis, I get it. But no one gives a real clear idea on what any individual is really supposed to do to solve the problem. You can’t get involved with one global cause, without ignoring all the others, and if you do get involved it’s likely to become your life’s purpose. Most people are concerned with other things — their families, their work, personal development, their homes and futures, and really that’s enough to take up all their time.

I’m always amazed when I read about the early unionists. Emma Goldman for example, the activist who pushed for the 8-hr workday, and campaigned for free love in the early 1900s when women were still wearing corsets, used to work 16 hour factory days as a seamstress, then lead meetings late into the night. Today we lead cushy lives comparatively–8 hour days, plus commute and lunch, family time, dinner time, gym maybe, sleep… but it still doesn’t seem like we ever have enough energy and time.

What Emma had that most people today don’t, is a community living in the same conditions as herself, with clear goals about what they were campaigning for, and a cause that affected their own daily lives. Today, unionism and local activism is in much shorter supply, in part due to the many people who work fairly comfy desk jobs, and the problem that everyone has his own specialization, works in a cubicle, does his or her own thing. The problems we’re facing today in terms of global warming, global water shortage, aren’t the same kinds of problems that activists have fought for in the past, and there’s no clear road map for how to solve them. Our leaders sure aren’t leading the way.

What we do have, at least, is the Olympics, which is an age old symbol of international cooperation, play and competition…so, uh, go sports! As for full disclosure, I don’t actually have a TV and haven’t watched the Olympics in many years, but I do try taking short showers–does that help?

You can not process complex events in some very simple way and expect to get accurate results.  You need knowledge, represented by one or more situational models, to process complex events.

Some folks, like to say that a “complex event” is simply an event which is an aggregation of two more more event objects.    If you follow this (flawed) logic, then counting integers is complex event processing; because 1 plus 1 is 2, and 2 is an aggregation of 1 and 1, so 2 is a complex event (not!).  

Since we know that counting is not a complex processing operation, then some folks would say that you can process complex events with very simple operations because you are processing complex events , in the case adding 1 to the previous number (counting), enriching an event object.

This is simply nonsense.

The logic flaw is that the basic definition of a “complex event” (used by many people) is wrong.   A complex event is not simply an event object with two more more events as sub-components. 

A complex event is when two event objects are combined (processed) to form a complex object with a higher degree of inference, or situational knowledge.   One plus one equals more than two in complex event processing, because the combination of event objects requires knowledge (e.g. a situational model).

A Complex Event = Sum (EventsObjects) + Situational Knowledge

Let there be no mistake about it.    Complex event processing is the complex processing of complex events.   You cannot accurately process complex events with simple event processing models.

The simple processing of complex events is not CEP, it is simple event processing (event track-and-trace, simple event object enrichment, simple event object aggregation, and so forth).
 

My favorite talk, as expected, was the Sotirov/Dowd talk on How To Impress Girls With Browser Memory Protection Bypasses. The attack is a conceptually simple, yet completely reliable technique for exploiting vulnerabilities in web browsers. Of course, the media has sensationalized the impact of their findings, but ultimately, this is still significant as far as browser-based exploits are concerned. It’s worth mentioning that part of the technique allowing them to load a .NET DLL at an arbitrary location under Vista was reliant on an implementation bug wherein the OS disables ASLR if the version in the .NET COR header was below a certain value. However, the address space spraying and stack spraying techniques are likely to be extended to other platforms utilizing similar memory protection mechanisms.

As for the girls? I can report first-hand that the ladies at TAO on Wednesday night were hanging on Alex’s every word. They were particularly impressed when he whipped out the laptop for a live demo. Unfortunately, none of the dozen iPhone owners in the immediate vicinity thought to snap a picture (too busy Twittering). Oh well.

I also enjoyed Hovav Shacham’s talk on return-oriented programming. Simply put, he described a generalization of the return-to-libc shellcode approach with the intent to demonstrate that one could achieve Turing-complete computation using “found code” in process images. By chaining together series of mini-computations ending in return (RET) instructions, it was possible to build higher-level programming constructs such as branches and loops. The nature of the x86 instruction set provides some flexibility because instructions are interpreted differently depending on how you align the instruction pointer (i.e. the old shellcode trick of searching the process image for any JMP EBX instruction and using that as your EIP). In RISC architectures such as SPARC, however, you don’t have that luxury; if your %pc isn’t aligned properly you get a bus error. So it was quite interesting to see that they were able to extend the concept to RISC. The practicality of the attack technique is limited by the fact that the shellcode is tuned to a particular binary image — if the shellcode was built using instructions extrapolated from glibc 2.3.5, it won’t work for a system running glibc 2.4.

I thought Scott Stender’s talk on Concurrency Attacks in Web Applications was interesting as well. In a nutshell, spewing thousands of simultaneous requests at web application transactions that are not thread-safe can create interesting problems. In the presentation, Scott ran his demo against a VM running on the attack machine. I found myself wondering how effective the same attack would be over the Internet — would it be significantly less reliable (or not at all)? Race conditions are generally easier to exploit locally than remotely due to more predictable execution conditions. Certainly this is an under-tested vulnerability class though.

One presentation I wasn’t able to attend but want to follow up on is Nate McFeters, John Heasman, and Rob Carter’s talk which discussed the GIFAR attack I’ve been hearing so much about lately. The gist is that you can create a file that is both a valid GIF and a valid JAR, then use some Java applet tricks to initiate HTTP requests on behalf of the victim.

Finally, the Pwnie Awards didn’t fail to disappoint. Drama ensued over the Most Overhyped award, but at least this year some of the winners showed up to claim their awards! Halvar rapping Symantec lyrics was also quite memorable.

All in all, a fun and informative week, but as usual, I was relieved to get the hell out of Vegas and head home on Friday morning.

P.S. For a much more entertaining BlackHat/Defcon Recap, read Jennifer Jabbusch’s account of the week’s events. It’s my favorite one so far!

My favorite talk, as expected, was the Sotirov/Dowd talk on How To Impress Girls With Browser Memory Protection Bypasses. The attack is a conceptually simple, yet completely reliable technique for exploiting vulnerabilities in web browsers. Of course, the media has sensationalized the impact of their findings, but ultimately, this is still significant as far as browser-based exploits are concerned (here is a more accurate report). It’s worth mentioning that part of the technique allowing them to load a .NET DLL at an arbitrary location under Vista was reliant on an implementation bug wherein the OS disables ASLR if the version in the .NET COR header was below a certain value. However, the address space spraying and stack spraying techniques are likely to be extended to other platforms utilizing similar memory protection mechanisms.

As for the girls? I can report first-hand that the ladies at TAO on Wednesday night were hanging on Alex’s every word. They were particularly impressed when he whipped out the laptop for a live demo. Unfortunately, none of the dozen iPhone owners in the immediate vicinity thought to snap a picture (too busy Twittering). Oh well.

I also enjoyed Hovav Shacham’s talk on return-oriented programming. Simply put, he described a generalization of the return-to-libc shellcode approach with the intent to demonstrate that one could achieve Turing-complete computation using “found code” in process images. By chaining together series of mini-computations ending in return (RET) instructions, it was possible to build higher-level programming constructs such as branches and loops. The nature of the x86 instruction set provides some flexibility because instructions are interpreted differently depending on how you align the instruction pointer (i.e. the old shellcode trick of searching the process image for any JMP EBX instruction and using that as your EIP). In RISC architectures such as SPARC, however, you don’t have that luxury; if your %pc isn’t aligned properly you get a bus error. So it was quite interesting to see that they were able to extend the concept to RISC. The practicality of the attack technique is limited by the fact that the shellcode is tuned to a particular binary image — if the shellcode was built using instructions extrapolated from glibc 2.3.5, it won’t work for a system running glibc 2.4.

I thought Scott Stender’s talk on Concurrency Attacks in Web Applications was interesting as well. In a nutshell, spewing thousands of simultaneous requests at web application transactions that are not thread-safe can create interesting problems. In the presentation, Scott ran his demo against a VM running on the attack machine. I found myself wondering how effective the same attack would be over the Internet — would it be significantly less reliable (or not at all)? Race conditions are generally easier to exploit locally than remotely due to more predictable execution conditions. Certainly this is an under-tested vulnerability class though.

One presentation I wasn’t able to attend but want to follow up on is Nate McFeters, John Heasman, and Rob Carter’s talk which discussed the GIFAR attack I’ve been hearing so much about lately. The gist is that you can create a file that is both a valid GIF and a valid JAR, then use some Java applet tricks to initiate HTTP requests on behalf of the victim.

Finally, the Pwnie Awards didn’t fail to disappoint. Drama ensued over the Most Overhyped award, but at least this year some of the winners showed up to claim their awards! Halvar rapping Symantec lyrics was also quite memorable.

All in all, a fun and informative week, but as usual, I was relieved to get the hell out of Vegas and head home on Friday morning.

P.S. For a much more entertaining BlackHat/Defcon Recap, read Jennifer Jabbusch’s account of the week’s events. It’s my favorite one so far!

LAS VEGAS -- Last weekend, more than 9,000 hackers, freaks, feds and geeks gathered for the 16th annual DefCon, the world's largest computer security convention.

Wired.com brought you live coverage of the most newsworthy events at DefCon 16. Here are some photos from the lighter side of the conference.

Left: South Korean hackers compete in the Capture the Flag competition. The goal is to hack into and keep control of targeted servers.

Mr. Sinister and Dragon Cracker battle it out in a round of Guitar Hero -- one of DefCon's newest competitions.

Bringing-your-own-booze supply ensures optimal buzz at DefCon. Shortly after this picture was taken, hotel security escorted this backpack-hacker to his room.

Computer geeks from the National Institute of Standards and Technology set up a network secured with quantum encryption in a conference room at DefCon. The quantum-entangled photons are being used to encrypt a video stream across a line-of-site network.

A compact optical bench and an atomic clock (left) are used to secure a network with quantum encryption.

In the Lock Pick Pavilion, DefCon attendees Dustin, Jennalynn and Kunfoozball practice their lock-picking skills.

DefCon founder and organizer Jeff Moss, aka Dark Tangent, at the conference's closing ceremony Sunday.

A collection of black badges awaits the winners of the various competitions. These badges give their holders lifetime entry to DefCon.

One of DefCon's logos, the smiley-faced skull and crossbones, is welded inside a yellow sphere. The sphere is the primary stage of one of the most difficult competitions at DefCon: The Mystery Challenge.

Unbeknownst to attendees, this laptop is sniffing RFID tags and taking photos of their owners when they pass in front of the detectors. RFID tags are used in everything from building access to some credit cards.

At the closing ceremony, DefCon organizers turn off the lights while the attendees wave their high-tech badges back and forth.

As a reference, you may have seen this briefing, one of many where I show these functional relationships, Mythbusters: Event Stream Processing Versus Complex Event Processing, from DEBS2007.  For example slide 23 shows the functional relationship between events, pre-processing, event tracking, situational detection, historical patterns (the output of BI tools, for example), visualization and business process management.

In Faithful Representation, Richard Veryard reminds his readers that the most challenging part is in the situation models (not the system integration).  Unfortunately, by accident, Richard incorrectly attributes Opher Etzion’s “first order situation model approximation” to both Opher and I in this quote from Richard’s post, “a simple situation model of complex events, in which events (including derived, composite and complex events) represent the “situation”.   

Actually, that simple situation model above is Opher’s, not mine.  I have offered a more general and comprehensive (first draft) situation model, in A Simple Situation Model for Complex Events based on a cognitive situation model used by researchers at the University of Notre Dame.  I do not believe that complex events and situations can be modelled accurately using Opher’s simple model of derived, composite and complex events.   This model is overly simple, in my opinion. to represent the vast majority of CEP classes of problems, perhaps explaining why Opher and I do not agree on the state-of-the-art of CEP.  Opher tends to view CEP as mostly an extension of active database technology where I see CEP as a technology that is much more closely aligned with the cognitive models represented in the art-and-science of multi-sensor data fusion (MSDF).  

Complex events represent situations, and situations must be accurately modelled if we are going to accurately detect them in real-time.  If your business cannot model a complex event (situation) then it does not matter what software you buy, how much money you spend, or what event processing and integration platform you use.   The models are hard.  The system integration is relatively easy.

The secret sauce is the situation and complex event models.

As mentioned here a few times, it does not matter how fast you process events in real-time, if your model is wrong, you just detect the wrong thing very fast.  This is very bad and quite dangerous.  You will make bad decisions fast.  You will waste time, money and resources.

This is why CEP benchmarks should be based on accuracy in situation detection, not in latency and other low-level performance metrics.   First, get the models right; then refine to detect faster, if speed is required.   What has happened in CEP to date, is that the models are so simple, they do not really detect complex events, they just process and act on simple events that are easy to model. 

Buying a lemon is always a bad thing – but when you pay $1 billion for it?! Back in 2005, Google bought a 5% stake in AOL for $1 billion and now is calling that investment “impaired”. That’s one way of putting it, so it’s a good thing Google has money to burn.

At LinuxWorld this week, Bob Sutor, VP of open source and standards at IBM, said that the next 10 years is “do or die” for open source software designed for specific industries. 10 years? That’s like 70 years in open source development time.

And finally…8/8/08…the Olympics are here! Network administrators around the world, except for Terry Childs, will be eyeing office network bandwidth closely as people go online to watch streaming video of the games. NBC and Microsoft will offer 2,200 hours of live video coverage with up to 20 simultaneous live streams of different events. Plus NBCOlympics.com will offer 3,000 hours of on-demand video content. The time difference means that much of the primetime events will be broadcast while the Western hemisphere is supposed to be hard at work. Me – I’m just glad it’s the weekend, and I can get the Olympics fix I’ve been waiting years for.

This use case is a clear example of a subfunction of complex event processing, folks in the mult-sensor data fusion field (and here at The CEP Blog) refer to as event (object) refinement, sometimes called “track and trace.”

The reason we call this processing function “event (object) refinement” is that, in the way the use case was described in the press release, the medical staff are basically tracking body temperature and comparing it to a key indicator to generate an alarm, in this case “body temperature too high.”   This is a simple event, not complex, because the level of inference is quite very low in an overall knowledge hierarchy.

For example, we cannot infer from the alarm that “body temperature too high” is caused by a previous medical condition.  There is no causality at this stage of the game.   We cannot infer from the alarm that the walker has embarked up a steep hill, and the body temperature is expected to exceed a key indicator for a period of time.

Looking at another complex event model,  the system does not (yet) combine all of the body temperatures of the entire group of walkers, correlated by the situation of an approaching thunderstorm, and infer that the walkers have increased their pace because they don’t want to be caught in a driving rainstorm with high winds.

In other words, tracking a single object like “body temperature” is a basic-step in a CEP application, but not really a CEP application yet, because to really be a complex event, there should be some inference of higher knowledge, or estimated situation.    For example, tracking and tracing the position of an aircraft is good data, but being able to infer the complex situation “potential mid-air crash” between two airplanes is better (defining a complex event vs simply tracking state changes).

Steam processing engines are well suited for track and trace processing of individual event objects, like a walker’s body temperature, or a similar temperature monitoring application from a network device, as demonstrated by the Apama use case.  Tracking events such as “temperature in an object reaches critical threshold” have been going on for decades, in your network, in your car,  in your washing machine, in as spacecraft, just about everywhere we sense-and-respond to temperature changes.

The real marvel of this application was not the event processing on the back end, but in the sensor network, comprised of the human body, an RFID sensor, and a transmission network to a centeralized data collection facility.

Back to CEP and EPTS, there are folks who appear to believe they may define “CEP” by the current use cases from self-described CEP vendors. Frankly speaking, I am puzzled by the bottom-up approach.

The bottom-up approach is a bit like saying “We have a lot of prototype rockets being built, so let’s define the future of space travel based on the prototypes!”

It really makes little sense, at least to me, to attempt to define CEP based on what the current generation products (self-described CEP products) are capable of doing.   

From my persective, it would be more beneficial to customers to define the types of complex events (and situations) businesses need to detect in real-time and match the technologies and solution architectures to detect those events, in real-time, with high confidence.

A lot of this “top down thinking” has been already done.

IT businesses need to detect operational threats and problems, and be able to pinpoint, with very high accuracy, where the problem is in a complex network, for example.  This problem remains mostly unsolved with a very low signal-to-noise ratio.

Also, most businesses would like to detect fraud and other criminal activity on their network before the activities adversely impacts their business.   This problem remains unsolved for most companies.

Scientific researchers seek models of weather, epidemiology, and so much more; and they need event processing solutions to obtain situational knowledge into current events and predict future ones.  We know how difficult predicting the weather can be!

Folks on the ground need to model urban traffic as events and design better event-driven traffic models and solutions.

The list of important event processing challenges we face go on and on.  

While I see some merit in the bottom-up approach, it is better for users to define what are practical “complex event” related problems and then look for the solutions, vs. define the solution and then look for the problem.

From a strategic perspective,  self-fulfilling CEP use case studies are interesting, but they hould not limit the vision, definition, and future of processing complex events; and be careful of use case fallacies.