First of all, let’s ground the discussion a bit by translating “smart order routing” to “rule-based message routing” since in this application “smart”  translates to “using rules” and “order” translates to “message”.    Basically, Mark (and other “new on the routing scene” stream processing players) argue that rule-based message routing is CEP.  I will argue that routing is not even close to CEP.  Here is why,

Let’s take a look at a router on the backbone of the global Internet.   A backbone router has very sophisticated software developed over many decades.   These routers run sophisticated, mature algorithms to determine how to route messages (packets) and use these algorithms to build complex routing tables. 

In addition, these routers process messages (packets) from countless sources and route messages (packets) to countless destinations.  Using some of the terms in early posts (above), there is a great “confluence of events” processed by routers.    Futhermore, there are normally quite complex authentication, authorization and other security parameters managed in a router, all in real time.   Routers do much more, but I don’t want to get too deep into routing in this post.

My point is that, without any doubt, global Internet routers process very “cloudy” “confluence of events” with much more sophistication than order routing applications.    However, we do not call Internet routing “CEP”, regardless of how many connections are processed or how much sophisticated processing occurs.  The reason is because the “C” in “CEP” defines a complexity that is at a higher abstraction than messaging and routing.

If you study the literature on CEP, some of which I posted recently, CEP was envisioned to solve complex event processing problems “on top of the routing layer” because the routing layer is a mature technology layer.  We can route, pure and simple.  Of course, we are always seeking faster, more scaleable and more secure routing. 

I admire some of the startups in the CEP/ESP/EP space for working hard to make money and for aggressively positioning their products and attempting to build market share.   However, issues surface when these same companies seem to believe they are the first companies to work in the event processing or message routing space and that they can define whatever they want as “complex event processing” as long as it benefits their sales targets.

There is no doubt that a router does much more sophisticated event processing than the new rule-based stream processing systems running continuous queries across streaming data.  There is no doubt that a router processes a complex “confluence of events”.   However, we don’t call routers “CEP”. 

We do not call routers “CEP” because CEP is about a higher level of knowledge processing.  CEP was created to detect the “complex events” that happen above the mediation and routing layer.     The literature and original examples on CEP are quite clear on this.

(Photo Credit: Gizmodo)

You lose some. You win some: Of course as NBC’s online partner, Microsoft gets a least a cut of the $100 million dollars in online advertising spent around the Olympics. And the millions of downloads of Silverlight aren’t too shabby either.

The Internet is Falling! Arbor Networks, a security and network management company, partnered with ninety network services and content providers from around the world to publish an extensive study of IPv6 traffic on the Internet. Craig Labovitiz, Arbor Networks chief scientist, stated that only 900 days were left until the end of the Internet, or at least the exhaustion of IPv4 registry allocations. For the past year, the study shows very little IPv6 traffic – something like 1/100^th of 1% of Internet traffic. Craig credits this to money issues. “The department of commerce estimates it will cost $25 billion for ISPs to upgrade to native IPv6.”

Blogger James Urquhart created a bill of rights for cloud computing. The purpose of the bill is to “help guide would-be cloud customers to those clouds best able to guarantee their freedom.” The blogosphere is a great place to get some open debate going, and I applaud James for trying to make something yet so “cloudy” a bit more clear and concrete. But what’s up with the creating a PAC for this?? (Check out the comments.)

Trying to get by on limited resources? Need more money, staff and the freedom to focus on long-term projects? Sound familiar? Then you just might be in IT at a midsize company. (or in marketing at a young but rapidly growing IT company ) Arrow Enterprise Computing Solutions conducted a survey of 200 tech leaders at midsize companies (500 to 3000 employees). The upside: 61% of those surveyed think they’ll be spending more on IT next year – is this bullish thinking about the economy or how much their own business (rev) will be growing?

Bill Snyder calls Dell “Bozo of the Month” for trying to trademark “cloud computing”. Yikes. Maybe not a “bozo” move but certainly inadvisable given how ubiquitous the term is. Here’s our take on it.

Citing sources within the aviation industry, ABC News reports an overzealous TSA employee attempted to gain access to the parked aircraft by climbing up the fuselage... reportedly using the Total Air Temperature (TAT) probes mounted to the planes' noses as handholds.

"The brilliant employees used an instrument located just below the cockpit window that is critical to the operation of the onboard computers," one pilot wrote on an American Eagle internet forum. "They decided this instrument, the TAT probe, would be adequate to use as a ladder."

They harass innocents:

James Robinson is a retired Air National Guard brigadier general and a commercial pilot for a major airline who flies passenger planes around the country.

He has even been certified by the Transportation Security Administration to carry a weapon into the cockpit as part of the government's defense program should a terrorist try to commandeer a plane.

But there's one problem: James Robinson, the pilot, has difficulty even getting to his plane because his name is on the government's terrorist "watch list."

It's easy to sneak by them:

The third-grader has been on the watch list since he was 5 years old. Asked whether he is a terrorist, he said, "I don't know."

Though he doesn't even know what a terrorist is, he is embarrassed that trips to the airport cause a ruckus, said his mother, Denise Robinson.

[...]

Denise Robinson says she tells the skycaps her son is on the list, tips heavily and is given boarding passes. And booking her son as "J. Pierce Robinson" also has let the family bypass the watch list hassle.

And here's how to sneak lockpicks past them.

As a reference, you may have seen this briefing, one of many where I show these functional relationships, Mythbusters: Event Stream Processing Versus Complex Event Processing, from DEBS2007.  For example slide 23 shows the functional relationship between events, pre-processing, event tracking, situational detection, historical patterns (the output of BI tools, for example), visualization and business process management.

In Faithful Representation, Richard Veryard reminds his readers that the most challenging part is in the situation models (not the system integration).  Unfortunately, by accident, Richard incorrectly attributes Opher Etzion’s “first order situation model approximation” to both Opher and I in this quote from Richard’s post, “a simple situation model of complex events, in which events (including derived, composite and complex events) represent the “situation”.   

Actually, that simple situation model above is Opher’s, not mine.  I have offered a more general and comprehensive (first draft) situation model, in A Simple Situation Model for Complex Events based on a cognitive situation model used by researchers at the University of Notre Dame.  I do not believe that complex events and situations can be modelled accurately using Opher’s simple model of derived, composite and complex events.   This model is overly simple, in my opinion. to represent the vast majority of CEP classes of problems, perhaps explaining why Opher and I do not agree on the state-of-the-art of CEP.  Opher tends to view CEP as mostly an extension of active database technology where I see CEP as a technology that is much more closely aligned with the cognitive models represented in the art-and-science of multi-sensor data fusion (MSDF).  

Complex events represent situations, and situations must be accurately modelled if we are going to accurately detect them in real-time.  If your business cannot model a complex event (situation) then it does not matter what software you buy, how much money you spend, or what event processing and integration platform you use.   The models are hard.  The system integration is relatively easy.

The secret sauce is the situation and complex event models.

As mentioned here a few times, it does not matter how fast you process events in real-time, if your model is wrong, you just detect the wrong thing very fast.  This is very bad and quite dangerous.  You will make bad decisions fast.  You will waste time, money and resources.

This is why CEP benchmarks should be based on accuracy in situation detection, not in latency and other low-level performance metrics.   First, get the models right; then refine to detect faster, if speed is required.   What has happened in CEP to date, is that the models are so simple, they do not really detect complex events, they just process and act on simple events that are easy to model. 

Definitely worth half an hour of your time.

And this is a video of Virginia Beach Police Department Officer George Bruch, who basically says that Duane is right.

Dan Munz, project manager of the Collaboration Project commented on the unique work that the National Academy of Public Administration (NAPA) is doing to bring together government leaders. The Collaboration Project seeks to innovate across government not just down the silos and create a safe place for leaders to have discussions around innovation.

ScienceLogic: What is the National Academy of Public Administration?

Dan Munz: The Academy is an independent, non-partisan, non-profit organization dedicated to tackling government’s most complex challenges. We were founded in 1967 by James Webb, the NASA administrator who took us to the moon – he saw that he could consult the National Academy of Sciences for expert technical advice, but had no counterpart in government for expert management advice. That’s been our mission ever since.

ScienceLogic: What is the Collaboration Project? How long has it been around?

Dan Munz: The Collaboration Project is the Academy’s response to two parallel trends we see in government. The first is the government’s need to transform the way it does business. There is a strong demand for change out there driven by a number of challenges that are forcing the government to rethink its mission and structure. Challenges include a public disconnected from government; a multi-sector workforce and increasing reliance on contractors; financial instability; and new types of security threats, just to name a few. More and more, the challenges facing government reach across the traditional boundaries of agency and mission. But government isn’t configured to work that way.

The second trend is the unprecedented opportunity collaborative technology offers to drive transformational change in government. Tools like blogs, wikis, and mashups are changing the way leaders think about problems. They’re focusing not on what they can do just within their offices or agencies, but what voices they need to pull together across government, non-profits, the general citizenry, and other stakeholders to solve these problems. The Collaboration Project’s goal is to encourage this type of thinking and empower leaders committed to use collaborative technology to:

• strengthen citizen civic engagement;
• enhance government transparency;
• improve service delivery and operational efficiency; and
• facilitate coordination and innovation within and between agencies.

ScienceLogic: Why focus on Web 2.0 in the government?

Dan Munz: The question of how web 2.0 will impact federal IT departments is a critical one. Our view is that “the era of big systems” is basically over. Things like disk space, bandwidth, and computing power are basically shifting from being assets to being commodities.

There’s also a shift in expectations. People both inside and outside government – especially Gen-X and Gen-Y – are incredibly frustrated by being able to use lightning-fast apps like Flickr, YouTube, and Facebook that don’t even live on their hard drives while the government and other large organizations still operate clunky PCs, space-limited e-mail accounts, and sluggish e-mail servers.

So aside from the opportunity for transformative leadership, the idea of web 2.0 at a government level is very appealing in terms of getting the most out of the IT infrastructure we already have, rather than embarking on costly, large-scale projects in an era of diminishing budgets.

ScienceLogic: How do you build a sense of community at the Collaboration Project?

Dan Munz: Some community feel emerges naturally, from a sense that mass collaboration really is a tool for “doing government” in a whole new way.

The more formal community building mechanisms we have include our web page, where we share insights, news, case studies, and other content – The virtual space serves as an anchor for people, whether they’re experts or beginners, to learn about what we do.

Finally, we are conducting an ongoing series of in-person meetings, usually featuring a leader who has harnessed collaborative technology in what we think is a truly revolutionary new way.

ScienceLogic: How do you hear about cool new government Web 2.0 projects?

Dan Munz: That’s a key question, because part of our mission is to inspire action by finding leaders who have succeeded and highlight their accomplishments. We’ve done that with folks like Kip Hawley, TSA, Molly O’Neill, EPA, and Jim Walker, Alabama DHS.

We also feel that the Academy’s position as a “safe space” for leaders means that we’re a place people can turn to when they hear about an emerging trend or project and want some help making sense of it.

ScienceLogic: What are the most innovative uses of Web 2.0 technology you’ve seen in the government?

Dan Munz: It’s important to distinguish between agencies that are simply adjusting to the reality of web 2.0, and those that are “using” it. Getting a YouTube account for your agency, or putting some photos on Flickr, is a great first step, but we want to inspire leaders to really transform their normal ways of doing business. At the moment a few that come to mind are the EPA Puget Sound Mashup, ODNI’s Intellipedia, TSA IdeaFactory, the PTO Peer-to-Patent Project, and Virtual Alabama, to name a few.

The TSA launched the IdeaFactory in February 2008. TSA set up a collaboration platform with commenting, voting, etc. to form communities in a way to bring people to consensus and offer ways to improve the agency’s performance.

ScienceLogic: Do you see a difference between state and local versus federal adoption of Web 2.0?

Dan Munz: That’s a hard generalization to make – at all levels you see leaders who recognize the potential in this technology to bring new voices into the governance process.

ScienceLogic: What are the obstacles to Web 2.0 adoption by government agencies?

Dan Munz: The three main challenges that we see are in the areas of technology, culture, and policy/governance.

The technology issue is probably the simplest to solve – it’s important to choose a technology that fits the problem you’re trying to solve, but these technologies are usually inexpensive and almost never very complex.

The question of culture is harder, particularly given the way that baby boomers, gen-xers, and millenials are beginning to interact in the workforce. How do you gain acceptance and buy-in among groups that have very different comfort levels with collaborative tools and environments?

Finally, the most daunting challenge might be the questions of policy and governance, if only because those are the things that most commonly prevent leaders from even dipping a toe in the waters of collaboration. Most of the policies, regulations, and statutes governing the way government does business don’t anticipate things like wikis, blogs, or instant messaging. One of our most important missions is helping leaders who just want to get to action navigate these obstacles.

ScienceLogic: Is there any advice you can give to government employees getting started with Web 2.0? Or any places you would point them to for more info?

Dan Munz: It’s shameless plug time! I’d of course point them to our web page, collaborationproject.org, where, among other things, we’ve collected a case library of over 40 instances of collaborative technology being used in the government and non-profit sectors. The library is growing every day and is a sort of “database of record” for what is and isn’t working in terms of collaborative government. I think that would be a great place to start for anyone looking to get started but not really knowing the way.

In terms of advice, the best thing to say is that, once you’ve settled on a problem you want to solve and an audience you want to reach out to, just do it! We believe strongly that there are a lot of organizational and leadership issues that still need to be addressed regarding collaboration in government, but our biggest mantra is about getting leaders to action. The most successful projects we’ve seen are ones that try something daring and new, and discover the true power of what they’ve done as it catches on more and more widely.

ShareThis

Complex event processing (CEP) is a new technology. It can be applied to extracting and analyzing information from any kind of distributed message-based system. It is developed from the Rapide concepts of (1) causal event modeling, (2) event patterns and pattern matching, and (3) event pattern maps and constraints. Complex event processing can be applied to a wide variety of Enterprise monitoring and management problems, from low level network management to high level enterprise intelligence gathering.

Applications of Complex Event Processing:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer. (coming soon)
  • Analysing business processes (paper in pdf format)
• Network Level Monitoring and Management (Powerpoint presentation)
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management (coming soon)
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring. (coming soon)
• Analysis and Debugging of Distributed Systems (coming soon)

Presentations:

• “Complex Event Processing: An Essential Technology for Instant Insight into the Operation of Enterprise Information Systems,” lecture at the Stanford University Computer Systems Laborary EE380 Colloquium series. Video of the lecture (duration: 60 minutes).

Publications:

• Complex Event Processing in Distributed Systems. David C. Luckham and Brian Frasca, Stanford University Technical Report CSL-TR-98-754, March 1998, 28 pages.Abstract: Complex event processing is a new technology for extracting information from distributed message-based systems. This technology allows users of a system to specify the information that is of interest to them. It can be low level network processing data or high level enterprise management intelligence, depending upon the role and viewpoint of individual users. And it can be changed from moment to moment while the target system is in operation. This paper presents an overview of Complex Event Processing applied to a particular example of a distributed message-based system, a fabrication process management system. The concepts of causal event histories, event patterns, event filtering, and event aggregation are introduced and their application to the process management system is illustrated by simple examples. This paper gives the reader an overview of Complex Event Processing concepts and illustrates how they can be applied using the Rapide toolset to one specific kind of system.
   
• Event Mining with Event Processing Networks. Louis Perrochon and Walter Mann and Stephane Kasriel and David C. Luckham, The Third Pacific-Asia Conference on Knowledge Discovery and Data Mining. April 26-28, 1999. Beijing, China, 5 pages.Abstract: Event Mining discovers and delivers information and knowledge in a real-time stream of data, or events. We show that the process of delivering knowledge by searching patterns in data and subsequent abstraction of found patterns can be applied in real-time to a complex, asynchronous system. Our event processing engine consists of a network of event processing agents (EPAs) running in parallel that interact using a dedicated event processing infrastructure. The agents can be configured at run-time using a formal pattern language. The underlying infrastructure (1) provides an abstract communication mechanism and thus allows dynamic reconfiguration of the communication topology between agents at run-time and (2) provides transparent, location-independent access to all data. These features allow dynamic allocation of EPAs to different threads and processes on different machines at run time.
   
• eJava - Extending Java with Causality. Alexandre Santoro and Walter Mann and Neel Madhav and David Luckham, Proceedings of the 10th International Conference on Software Engineering and Knowledge Engineering, June 1998, 10 pages.Abstract: Programming languages like Java provide designers with a variety of classes that simplify the process of program development. Some of these classes allow one to easily build multithreaded programs. Though useful, especially in the creation of reactive systems, multithreaded programs present challenging problems such as race conditions and synchronization issues. Validating these programs against a specification is not trivial since Java does not clearly indicate thread interaction. These problems can be solved by modifying Java so that it produces computations, collections of events with both causal and temporal ordering relations defined for them. Specifically, the causal ordering is ideal for identifying thread interaction. This paper presents eJava, an extension to Java that is both event based and causally aware, and shows how it simplifies the process of understanding and debugging multithreaded programs.
   
• Event-Based Execution Architectures for Dynamic Software Systems. James Vera, Louis Perrochon, David C. Luckham.
  Proceedings of the First Working IFIP Conf. on Software Architecture. 1999. San Antonio, Texas.Abstract: Distributed systems’ runtime behavior can be difficult to understand. Concurrent, distributed activity make notions of global state difficult to grasp. We focus on the runtime structure of a system, its execution architecture, and propose representing its evolution as a partially ordered set of predefined architectural event types. This representation allows a system’s topology to be visualized, analyzed and con-strained. The use of a predefined event types allows the execution architectures of different systems to be readily compared.
   
• Using Context-Based Correlation in Network Operations and Management. Louis Perrochon (work in progress, mail author for newest version)Abstract: Network operation consists to a large degree of reaction to activities happening in the network. Better knowledge of the network at any time allows more appropriate reactions. On the example of intrusion detection, we show how context-based correlation of such activities can provide a more detailed view of the network in shorter time. We first present how we model context and then describe the architecture of the Stanford University CEP context-based correlator. Correlation is specified as event patterns in a declarative language that allows us to specify what needs to be detected, instead of specifying how it should be detected. CEP introduces the concept of causal context to intrusion detection. The correlator is able to process events on-line, as they are generated and it can be reconfigured at dynamically. We then show how it increases detection rate, reduce false alarms, and detect large-scale attack patterns at an early stage.