“Experienced end users are very intelligent.

These end users know the complex event processing problems they need to solve; and they know the limitations of the current COTS approaches marketed by the CEP community.  Even in Thailand, a country many of you might mistakenly think is not very advanced technologically, there are experts in telecommunications (who run large networks) who are working on very difficult fraud detection applications, and they use neural networks and say the results are very good.   However, there is not one CEP vendor, that I know of, who offers true CEP capability in the form of neural nets.

Almost every major bank, telco, etc. has the same opinion, and the same problem. They need much more capability than streaming joins, selects and rules to solve their complex event processing problems that Dr. Luckham outlined in his book.   The software vendors are attempting to define the CEP market to match their capability; unfortunately, their capabilities do not meet the requirements of the vast majority of end users who have CEP problems to solve.

If the current CEP platforms were truely solving complex event processing problems, annual sales would be orders of magnitudes higher.  Hence, the users have already voted.   The problem is that the CEP community is not listening.”

Not to be overly repetitive,  but the last part of this quote from a year ago is worth highlighting:

“If the current CEP platforms were truely solving complex event processing problems, annual sales would be orders of magnitudes higher.  Hence, the users have already voted.   The problem is that the CEP community is not listening.”

Frankly speaking, nothing in the “CEP world” has changed, technologically speaking, since this September 2007 post was written.  From a sales perspective, we have seen less CEP-related sales in 2008 than in prior years.   If these so called CEP products were actually capability of detecting “real” complex network-centric situations (threats) in real-time, they would be selling faster than a cup of ice water in the blazing hot Sahara desert.

Don’t shoot the messenger.  Build better detection engines!

On the other hand, maybe complex detection is too hard for most of these companies and that is why they focus on routing, mediation and relatively simple rule-based scenarios, versus complex event processing?

OnAir has so far tested their calling/texting-only service on two aircraft--one operated by Air France, one by TAP Portugal--even though RyanAir announced plans that its planes would started being unwired with the service by late 2007. Still no word on that fleet progress.

Qantas will apparently launch cached Web browsing and limited Web email (probably through a proxy) along with instant messaging, with full Internet service coming "later in 2009." This is clearly due to a lack of satellite coverage that was just remediated a few weeks ago (see below). The first plane with limited service, a new A380, should be in flight 20-October-2008.

The Morning Herald seems to overstate the importance and scope of a complaint filed by the union representing American Airlines' flight attendants. The detailed coverage in the U.S. had more to do with the potential for issues, and likely attendants lack of interest in policing yet another media on the plane. Filtering doesn't work, the attendants probably already know, and this may just be a negotiating point with the airline.

On why Qantas is waiting until late 2009? This requires unwinding how OnAir gets its signal.

Aeromobile and OnAir both rely on Inmarsat satellites for their service. Both companies had several years ago staked their futures on the fourth-generation network Inmarsat was to inaugurate with three satellites that would use beamforming to allow precise delivery of nearly 500 Kbps per receiver, with hundreds or thousands of regions being able to be targeted from a single satellite. Inmarsat's third-gen network--don't confuse this with 3G cellular ground-based networks--can deliver about 64 Kbps per channel.

Now, unfortunately, Inmarsat was three years late on launching its trans-Pacific bird. While the company claims 85 percent coverage of the earth and 98 percent coverage of population, there's a big gap over the Pacific that also prevents them from having good overlap between the U.S. and Japan/China/Korea, as well as the southern Pacific, covering Australia. Since the biggest market for long-haul flights would likely be Australia, Japan, and China, traveling trans-Pacific or trans-hemispheric routes, that gap is rather large.

Aeromobile opted to build out a service, deployed only by Emirates airline as far as I can tell, that uses the 3G service since it was available, and most necessary equipment is already installed on most over-water planes. OnAir was waiting for 4G, which has necessitated a long wait, but allowed them to launch in Europe with a seemingly next-generation service. Given that OnAir is controlled by an airline-owned integration firm, SITA, and by Airbus, they're not going anywhere.

Inmarsat finally lofted its third satellite on Baikonur Cosmodrome in Kazakhstan on 19-August-2008, and the launch and separation was reported as successful. Previously, the company has needed up to a year to verify and deploy its 4G satellites. (You can read extremely close coverage of the launch at a Web site devoted to space enthusiasm.)

However, the dirty little secret about Inmarsat's BGAN is that it costs a fortune to heft bandwidth across it. Thus, in-flight broadband over BGAN, if it's ever available, is going to be changed on an extremely high per-MB rate. None of the providers want to say this. This is in contrast to Row 44 (and, once, Connexion by Boeing), which relies on leased Ku-band transponders where they can fix costs and they require high volumes to keep per-bit costs efffectively low.

OnAir's launch of calling on Air France's service involves paying a few euros per minute for calls, which might help you understand what data costs could ultimately run.

“Event processing has been going on for more than fifty years.”

However, in On Event Processing as a Discipline and Some Subsets another colleague mistakenly blogs,

“… people who dealt in this area [network management and event correlation] have never investigated event processing in the larger sense (e.g. looking at additional patterns), and this area has also not spawned the event processing discipline.”

If you examine just one page from the CEP history at Stanford, researchers there outlined their view of the future applications for CEP, as follows:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer.
  • Analysing business processes
• Network Level Monitoring and Management
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring
• Analysis and Debugging of Distributed Systems

These applications areas mentioned by Stanford researchers, including Professor Luckham, support and validate our recent discussion Magic Quadrant for IT Event Correlation and Analysis, 2007 where we concluded that “event correlation and event analysis is Gartner’s closest magic quadrant (MQ)  [...] relates directly to complex event processing (and event processing in general).”  

If you take a detailed look at the 1999 CEP presentation, Defeating Large Scale Attacks: Technology and Strategies for Global Network Monitoring you will readily see that our colleagues are incorrect when they says that event correlational and network management folks have never investigated event processing in the “larger sense”.  For example, the 1999 slides above, Stanford, slide 6, is titled “Complex Event Processing,” defining CEP from the application perspective of event correlation;

Complex Event Processing

• Accept network ‘events’ from any source
  • CISCO NetFlow FlowCollector, tcpdump
• Correlates events based on content and temporal relationship between events
• Event Processing Agents (EPAs) connected in an Event Processing Network (EPNs)
• Both post-mortem and real-time processing

This single event correlational project example from David’s team at Stanford examined the challenging event correlation problems in the context of hierarchical events, maps, patterns, visualization tools, event processing models, patterns languages, network management abstraction layers, and more.  Those core event processing problems from this 1999 example, very large and complex then, still exist today and are much more large and complex - precisely why it is called “complex event processing.”

It is quite obvious, in just this one example, that many folks have been looking at event correlation as a motivating application for event processing, in a larger context, for a long time, contrary to what our colleagues write in their “history of event processing” posts.  

In a future post I will completely debuke these event processing “history revisionists.”   I will illustrate very clearly how the history of event processing goes back at least a decade, and perhaps two (twenty years) before the history outlined in posts like On Research and Practice in Event Processing and The History of Complex Event Processing. 

David Luckam stated that the art-and-science of event processing goes back around 50 years. 

I am not sure I will go all the way back to 1960 in my next post on the history of event processing.  However,  I will go back at least to the early days of Internet Protocol (IP) networking and illustrate why distributed IP networking, network management and network security, is one of the key  motivating factors for what we now call “event processing” and “complex event processing.”

Pouring over endless details of risks, regulations, taxonomies, and technologies can sometimes give us a narrow view of the world, so it seems worthwhile to take a minute to mark the 125th anniversary of the cataclysmic eruption of Krakatoa this week. For those of us that want to think big but can’t remember that far back, this week is also the 3rd anniversary of Hurricane Katrina’s devastating sweep across a wide stretch of the US Gulf Coast.

By now, I expect that most of you have read or are familiar with the 2007 book, The Black Swan, by Nassim Nicholas Taleb, which argues that these kinds of unpredictable, outlying occurrences are the ones that really shape businesses, countries, economies, and people. Taleb argues that although these “Black Swan” events are almost completely unforeseeable, we mistakenly try to explain the circumstances at the time and make predictions about similar events in the future.

In my ERM work with clients, and especially in the context of research I’ve been doing with my colleague Stephanie Balaouras on business continuity and resiliency, questions come up about how to plan for catastrophes... and they’re good questions. Were the CardSystems or TJX data breaches foreseeable? What about the Societe General debacle or the 2004 Indian Ocean tsunami? What’s next? Should these types of events be included in our risk assessments?

We’d like to get your opinion on these and other risks that may be on the very edge of the statistical tail. At what point do they belong in your risk register?

Of course, it’s possible to define mitigating controls for crises, disasters, or incidents without knowing for sure what they’re going to look like. That’s one of the hallmarks of a good crisis management plan. And that’s an important point, because trying to predict the next unforeseeable event can be a real challenge sometimes.

Upping The IPS Ante

Motorola announced this week its intentions to acquires Wireless IDS/IPS vendor AirDefense. The acquisition may provide a bit of deja vu to readers who recall the acquisition of Network Chemistry's wireless IDS/IPS assets by Aruba Networks in 2007.

Meru Networks, eschewing acquisition for product introduction made its own announcement on Monday, announcing the company's RF Barrier, an active RF management solution that aims to solve the problem of what the vendor is calling "leaky RF." The Meru solution actively blocks 802.11 RF from escaping the physical confines of a WLAN deployment to thwart external "parking lot" attacks by closing Wi-Fi based attack avenues.

In fact, 2007 - 2008 has been a time focused on shoring up the security of the WLAN as the networks become more critical to over 50% of enterprises Forrester sees investing in the networks today. As the networks are more pervasive, moving toward covering the entire physical environment, and more employees are relying on Wi-Fi to access corporate data and applications, it's high-time to secure the WLAN.

In the case of Motorola, the Wi-Fi network is especially critical. As the vendor embarks on selling its message of the all-wireless enterprise, where WLANs will interconnect not only users to the network, but networke edge devices -- such as WLAN access points -- to the network along with storage, printers and other peripheral devices, the WLAN is citical and, therefore, a major focus for security.

In markets such as retail, standards like the Payment Card Industry's Data Security Standard dictate wireless security, but compliance and regulation aside, it is becoming easier to secure the WLAN, regardless of the industry you are in. Vendors are rapily working to close security gaps with product enhancements and new product introductions. Look for a broader suite of solutions to address security coming from your primary network vendor; while this won't negate the need to  integrate these add-on network elements, the single source should ease integration to some degree.

How secure do you feel your organization's WLAN is today? What are your concerns either about securing the network or its current lack of security?

One of his favorite OSS stories involved a colleague sent to occupied France to destroy a seemingly impenetrable German tank at a key crossroads. The French resistance found that grenades were no use.

The OSS man, fluent in German and dressed like a French peasant, walked up to the tank and yelled, "Mail!"

The lid opened, and in went two grenades.

Hall's book about his OSS days, You're Stepping on My Cloak and Dagger, is a must read.

However, I should briefly clarify Paul’s note that “blackboard systems historically used a single memory model (i.e. multiple threads or processes using a single machine’s memory model)“.

In fact, there were many blackboard systems, some more than a decade old, that used a distributed memory data-model. What I think Paul meant to say, and my apologies to Paul for being so literal, is that “blackboard systems originally used a single memory model (i.e. multiple threads or processes using a single machine’s memory model)”

John McManus, former CTO of NASA, wrote an excellent PhD dissertation in 1992, Design and Analysis Techniques for Concurrent Blackboard Systems. John’s thesis, now more than 16 years old, examined many details of concurrent blackboards where memory is distributed. For example, refer to Figure 2.3. Distributed Blackboard System with Distributed Blackboard Data Structure, page 36 of John’s dissertation.

Quoting directly from page 37 of John’s disseration;

Rice, Aiello and Nii [20] present several options for gaining speedups in a distributed blackboard system.

• 1) Eliminate the centralized scheduling mechanism
• 2) Optimize system design for a distributed memory, message-passing hardware
• 3) Distribute the data across the blackboard to reduce hotspots

Quoting further from the same page;

Poligon [21] is based on a distributed memory hardware model when each processor is viewed as a blackboard node. They define a blackboard node as follows: “a blackboard node is a process on a processor, surrounded by a collection of processors able to service its requests to execute rules.” [22] The implicit assumption in this definition is that all knowledge sources are rule–based systems. This assumption may severely limit the performance of systems implemented using Poligon, and limits the types of problems it is suited to address.

In Blackboards for Complex Event Processing, Paul concludes,

“One suspects the blackboard systems domain and terminology is overdue some updates thanks to developments in the Complex Event Processing space.”

If you look at the historical literature, I would say that the following restatement is more accurate:

“The CEP domain and terminology is overdue some updates because folks working in CEP did not reference or incorporate the advanced event processing prior art in a number of very important areas, blackboard systems being only one.”

On the other hand, commercial off-the-shelf rule-processing technology such as TIBCO’s BusinessEvents (BE), advances the ability to economically implement myriad complex problems that blackboard systems are designed to address.