I agree that these gentlemen are top notch researchers, witnessed by the fact that the authors do not mention nor claim to be “complex event processing” anywhere in their paper!  This paper is not about CEP, nor does it claim to be about CEP, it is about stream processing and unifying SQL standards.

ABSTRACT: This paper describes a unification of two different SQL extensions for streams and its associated semantics. We use the data models from Oracle and StreamBase as our examples. Oracle uses a time-based execution model while StreamBase uses a tuple-based execution model. Time-based execution provides a way to model simultaneity while tuple-based execution provides a way to react to primitive events as soon as they are seen by the system.

Asmentioned on numerous occasions, stream processing is a very important area in CEP/EP.   It is important not to confuse the higher situational knowledge from object-object correlation and state management with the single-object event refinement that occurs in stream processsing.    Event stream processing is fundamentally different than complex event processing. 

Event stream processing performs operations on streaming event objects.   In almost all advanced CEP/EP applications is is necessary to perform robust track and trace operations on streaming event objects, like tracking the position of an airplane.    Tracking the position of an aircraft can be modelled very nicely with event stream processing.  Tracking individual event objects is a precuror to multiclass object situation refinement.

When we manage the state of all the aircraft in the skies over New York, you need more than a stream processing construct.  You need to manage the state of all the aircraft.  Paul Vincent of TIBCO Software being to address this important point in The Value of State…  

Again, we will be better equiped to solve complex distributed event processing problems if we do not confuse the notion of event stream processing and complex event processing.   These technologies are indeed complimentary, both very important, but they are not the same.

I applaud Oracle and StreamBase’s work toward a unified standard for SQL extensions for streams.

adaptive process management (n.) an element of resource and business process management, adaptive search and event processing. Sometimes referred to as “Level 4” event processing or process refinement.

application concept (n.) a definition of a set of properties that represent the data fields of an application entity. An application concept can describe relationships among themselves. For example, an order concept might have a parent/child relationship with an item concept. A department concept might be related to a purchase requisition concept based on the shared property, department_id. Application concepts can include an application state model.

application state modeler (n.) a UML-compliant application that allows you to model the life cycle of a concept instance — that is, for each instance of a given concept, you can define which states it will pass through and how it will transition from state to state. States have entry actions, exit actions, and conditions, providing precision control over the behavior of an event processing agent. Transitions between states also may have rules. Multiple types of states and transitions maximize the versatility and power of the application state modeler.

derived event (n.) an event that is created as a result of processing one or more other events.

complex event (n.) an event that is a situation-entity abstraction of two or more simple, derived or other complex events.

complex event processing (n.) CEP is a technology for extracting information from message-based systems. CEP is primarily an event processing concept that deals with the task of processing multiple events from an event cloud with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes.

event (n.) a instance of an event definition. It is an immutable object that represents a business activity that happened at a single point in time. Just as one cannot change the fact that a given activity occurred, one cannot change an event — events are immutable.

event aggregation (n.) the aggregation of simple, derived or complex events into higher levels of event abstractions.

event definition (n.) a set of properties related to a given activity that represents an important or interesting change of state in a human, system or computational activity. An event definition includes event properties such as event priority, event time to live (TTL), and a description of the payload, which is comprehensive information related to the activity that occurred. Events expire when the TTL has elapsed, unless the event processing agent has instructions to consume them prior to that time.

event channel (n.) a communications channel in which events are transmitted from event source to event receivers, typically received as electronic messages. Each channel can have multiple destination and. events can be configured to transmit to a default destination. JMS is an example of an event channel.

event cloud (n.) a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events. Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud.

event-driven (n.) the behavior of a human, system or computational entity whose execution or actuation is in response to events, typically received as electronic messages.

event-driven architecture (n.) an architectural style for distributed computing applications in which some of the components are event-driven and communicate by means of events.

event processing (n.) computing that performs operations on events, including modifying, creating and destroying events.

event-object (n.) an software object that represents an event, generally for the purpose of computer processing, that exhibits both encapsulation, inheritance and polymorphism.

event prediction (n.) computational activity where the impact of events, complex events, and situations caused by events identified, including both opportunity or threat. Sometimes referred to as “Level 2” event processing, impact assessment or predictive analytics.

event pre-processing (n.) computational activity where events are cleansed or normalized to produce semantically understandable data. Sometimes referred to as “Level 0” event processing.

event processing (n.) computational activities on events dealing with the association, correlation, and combination of event data and information from single and multiple event sources to achieve refined identity and situation estimates for observed event objects, and to achieve complete and timely assessments of opportunities, threats, and their significance. Event processing is characterized by continuous refinements of event estimates and assessments and by evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

event processing agent (n.) an EPA is a computational entity that performs event processing.

event processing network (n.) a set of event processing agents and a set of event channels connecting them.

event properties (n.) data representation of an event, typically by name-value pairs of type string, integer, real, boolean or a complex data type.

event refinement (n.) filter, identify and track events & make initial processing decisions based on association, correlation and state estimation. Sometimes referred to as “Level 1” event, or event-object, track and trace.

event stream (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

event stream processing (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

rule (n.) defines what triggers unusual, suspicious, problematic, or advantageous activity within an event processing agent and what the EPA does when it discovers these types of activities. Rules execute actions based on certain conditions on events, instances, or a combination of both. A rule includes a group of condition-rule statements and action-rule statements. The condition statements instruct the EPA what to look for in events, and action statements instruct the EPA how to respond when conditions are met. If all the conditions in a rule are satisfied by events or instances or both, the EPA fires the actions. The action might be to execute tasks, create an event instance, modify property values in an event instance, create and send an event, or something else.

rules engine (n.) a type of event processing agent that uses a declarative programming model to process events. Formally described as “an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet“. Informally, it can be any system that uses rules, in any form, that can be applied to data to produce outcomes.

rule language (n.) is an artificial language that is used to control the behavior of an event processing agent. Rules languages, like human languages, have syntactic and semantic rules to define meaning.

situation refinement (n.) identify situations, or complex events, based on event clustering, event-event relationships and relationship analysis and context. Sometimes referred to as “Level 2” event processing.

simple event (n.) an event that is not an abstraction or composition of other events.

virtual event (n.) an event that is imagined, modeled or simulated.

You can not process complex events in some very simple way and expect to get accurate results.  You need knowledge, represented by one or more situational models, to process complex events.

Some folks, like to say that a “complex event” is simply an event which is an aggregation of two more more event objects.    If you follow this (flawed) logic, then counting integers is complex event processing; because 1 plus 1 is 2, and 2 is an aggregation of 1 and 1, so 2 is a complex event (not!).  

Since we know that counting is not a complex processing operation, then some folks would say that you can process complex events with very simple operations because you are processing complex events , in the case adding 1 to the previous number (counting), enriching an event object.

This is simply nonsense.

The logic flaw is that the basic definition of a “complex event” (used by many people) is wrong.   A complex event is not simply an event object with two more more events as sub-components. 

A complex event is when two event objects are combined (processed) to form a complex object with a higher degree of inference, or situational knowledge.   One plus one equals more than two in complex event processing, because the combination of event objects requires knowledge (e.g. a situational model).

A Complex Event = Sum (EventsObjects) + Situational Knowledge

Let there be no mistake about it.    Complex event processing is the complex processing of complex events.   You cannot accurately process complex events with simple event processing models.

The simple processing of complex events is not CEP, it is simple event processing (event track-and-trace, simple event object enrichment, simple event object aggregation, and so forth).
 

This use case is a clear example of a subfunction of complex event processing, folks in the mult-sensor data fusion field (and here at The CEP Blog) refer to as event (object) refinement, sometimes called “track and trace.”

The reason we call this processing function “event (object) refinement” is that, in the way the use case was described in the press release, the medical staff are basically tracking body temperature and comparing it to a key indicator to generate an alarm, in this case “body temperature too high.”   This is a simple event, not complex, because the level of inference is quite very low in an overall knowledge hierarchy.

For example, we cannot infer from the alarm that “body temperature too high” is caused by a previous medical condition.  There is no causality at this stage of the game.   We cannot infer from the alarm that the walker has embarked up a steep hill, and the body temperature is expected to exceed a key indicator for a period of time.

Looking at another complex event model,  the system does not (yet) combine all of the body temperatures of the entire group of walkers, correlated by the situation of an approaching thunderstorm, and infer that the walkers have increased their pace because they don’t want to be caught in a driving rainstorm with high winds.

In other words, tracking a single object like “body temperature” is a basic-step in a CEP application, but not really a CEP application yet, because to really be a complex event, there should be some inference of higher knowledge, or estimated situation.    For example, tracking and tracing the position of an aircraft is good data, but being able to infer the complex situation “potential mid-air crash” between two airplanes is better (defining a complex event vs simply tracking state changes).

Steam processing engines are well suited for track and trace processing of individual event objects, like a walker’s body temperature, or a similar temperature monitoring application from a network device, as demonstrated by the Apama use case.  Tracking events such as “temperature in an object reaches critical threshold” have been going on for decades, in your network, in your car,  in your washing machine, in as spacecraft, just about everywhere we sense-and-respond to temperature changes.

The real marvel of this application was not the event processing on the back end, but in the sensor network, comprised of the human body, an RFID sensor, and a transmission network to a centeralized data collection facility.

The first post described the problem: ASP.NET wasn't reporting inner exception stack traces.

The second post described my solution.

This post shows the code I used to solve the problem: a custom email provider for the Health Monitoring system in ASP.NET. Enjoy!

Here's the provider. Note that I opted *not* to build a buffering provider to keep things simple:

public class MyMailWebEventProvider : WebEventProvider
{
    string to;
    string from;
    string subjectPrefix;

    public override void Initialize(string name,
        NameValueCollection config)
    {
        base.Initialize(name, config);

        to = GetAndRemoveStringAttribute(config, "to", true);
        from = GetAndRemoveStringAttribute(config, "from", true);
        subjectPrefix = GetAndRemoveStringAttribute(config,
            "subjectPrefix", false);
    }
    public override void ProcessEvent(WebBaseEvent raisedEvent)
    {
        SendMail(raisedEvent);
    }

    private void SendMail(WebBaseEvent raisedEvent)
    {
        string subject = ComputeEmailSubject(raisedEvent);
        string body = ComputeEmailBody(raisedEvent);

        MailMessage msg = new MailMessage(from, to, subject, body);
        new SmtpClient().Send(msg);
    }

    private string ComputeEmailBody(WebBaseEvent raisedEvent)
    {
        WebRequestErrorEvent errorEvent =
            raisedEvent as WebRequestErrorEvent;
        if (null != errorEvent)
            return ErrorEventFormattingHelper.FormatRequestErrorEvent(errorEvent);
        else return raisedEvent.ToString();
    }

    private string ComputeEmailSubject(WebBaseEvent raisedEvent)
    {
        StringBuilder subjectBuilder = new StringBuilder();

        // surface some details in subject about error events
        WebBaseErrorEvent errorEvent = raisedEvent as WebBaseErrorEvent;
        if (null != errorEvent)
        {
            Exception unhandledException = errorEvent.ErrorException;

            // drill through reflection exceptions to show the root cause
            TargetInvocationException invocationException =
                unhandledException as TargetInvocationException;
            if (null != invocationException)
            {
                Exception innerException =
                    DrillIntoTargetInvocationException(invocationException);
                subjectBuilder.AppendFormat("{0}",
                    (innerException ?? invocationException).GetType().Name);
                if (null != innerException)
                    subjectBuilder.Append(" (via reflection)");
            }
            else subjectBuilder.Append(unhandledException.GetType().Name);
        }

        // if we've not got anything better
        // just show the event type in the subject
        if (0 == subjectBuilder.Length)
            subjectBuilder.AppendFormat("Event type: {0}",
                raisedEvent.GetType().Name);

        if (!string.IsNullOrEmpty(subjectPrefix)) {
            subjectBuilder.Insert(0, ' ');
            subjectBuilder.Insert(0, subjectPrefix);
        }
        return subjectBuilder.ToString();
    }

    /// <summary>
    /// Reflection often hides exception details, so we try to drill down
    /// through the plumbing exceptions to find a likely cause
    /// </summary>
    private Exception DrillIntoTargetInvocationException(
        TargetInvocationException outerException)
    {
        Exception innerException = outerException.InnerException;
        TargetInvocationException innerInvocationException =
            innerException as TargetInvocationException;
        if (null != innerInvocationException)
            return DrillIntoTargetInvocationException(innerInvocationException);
        else if (null != innerException)
            return innerException;
        else return null;
    }

    private static string GetAndRemoveStringAttribute(NameValueCollection config,
        string attributeName, bool required)
    {
        string value = config.Get(attributeName);
        if (required && string.IsNullOrEmpty(value))
            throw new ConfigurationErrorsException(string.Format(
                "Expected attribute {0}, which is missing or empty.",
                attributeName));
        config.Remove(attributeName);
        return value;
    }

    public override void Flush()
    {
        // nothing to do - this is not a buffering provider
    }

    public override void Shutdown()
    {
        // nothing to do here either
    }
}

Here's a helper class that formats the error messages the way I want to see them. Note that I've omitted some fields that I personally didn't care about, and I've reordered things a bit, so you might want to tweak this if you're going to use it in your own system.

internal static class ErrorEventFormattingHelper
{
    internal static string FormatRequestErrorEvent(
        WebRequestErrorEvent errorEvent)
    {
        CustomEventFormatter formatter = 
            new CustomEventFormatter();

        formatter.AppendLine(string.Format(
            "Unhandled Exception in {0}:",
            WebBaseEvent.ApplicationInformation
            .ApplicationVirtualPath));
        formatter.Indent();
        EmitExceptionAtAGlance(formatter, 
            errorEvent.ErrorException);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Exception stack trace(s):");
        EmitExceptionStackTrace(formatter, 
            errorEvent.ErrorException);

        formatter.AppendLine();
        formatter.AppendLine("Event information:");
        formatter.Indent();
        EmitEventInfo(formatter, errorEvent);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Application information:");
        formatter.Indent();
        EmitApplicationInfo(formatter, 
            WebBaseEvent.ApplicationInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Process/thread information:");
        formatter.Indent();
        EmitProcessInfo(formatter, 
            errorEvent.ProcessInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Request information:");
        formatter.Indent();
        EmitRequestInfo(formatter, 
            errorEvent.RequestInformation);
        formatter.RevertIndent();

        return formatter.ToString();
    }

    private static void EmitEventInfo(
        CustomEventFormatter formatter,
        WebBaseEvent theEvent)
    {
        formatter.AppendLine(string.Format(
            "Event code: {0}",
            theEvent.EventCode.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event message: {0}", 
            theEvent.Message));
        formatter.AppendLine(string.Format(
            "Event time: {0}", 
            theEvent.EventTime.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event ID: {0}", 
            theEvent.EventID.ToString("N", 
            CultureInfo.InvariantCulture)));
    }

    private static void EmitApplicationInfo(
        CustomEventFormatter formatter, 
        WebApplicationInformation appInfo)
    {
        formatter.AppendLine(string.Format(
            "Application domain: {0}", 
            appInfo.ApplicationDomain));
        formatter.AppendLine(string.Format(
            "Application Virtual Path: {0}", 
            appInfo.ApplicationVirtualPath));
        formatter.AppendLine(string.Format(
            "Application Physical Path: {0}", 
            appInfo.ApplicationPath));
    }

    private static void EmitProcessInfo(
        CustomEventFormatter formatter, 
        WebProcessInformation webProcessInfo)
    {
        formatter.AppendLine(string.Format(
            "Process ID: {0}", 
            webProcessInfo.ProcessID.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Process name: {0}", 
            webProcessInfo.ProcessName));
        formatter.AppendLine(string.Format(
            "Account name: {0}", 
            webProcessInfo.AccountName));
    }

    private static void EmitRequestInfo(
        CustomEventFormatter formatter, 
        WebRequestInformation webRequestInfo)
    {
        string name = null;
        if (webRequestInfo.Principal != null)
            name = webRequestInfo.Principal.Identity.Name;

        formatter.AppendLine(string.Format(
            "Request URL: {0}", 
            webRequestInfo.RequestUrl));
        formatter.AppendLine(string.Format(
            "Request path: {0}", 
            webRequestInfo.RequestPath));
        formatter.AppendLine(string.Format(
            "User name: {0}", 
            name ?? "[ANONYMOUS]"));
        formatter.AppendLine(string.Format(
            "User host address: {0}", 
            webRequestInfo.UserHostAddress));
    }

    private static void EmitExceptionAtAGlance(
        CustomEventFormatter formatter, 
        Exception exception)
    {
        formatter.AppendLine(string.Format(
            "Type: {0}", 
            exception.GetType().Name));
        formatter.AppendLine(string.Format(
            "Message: {0}", 
            exception.Message));
        if (null != exception.InnerException)
        {
            formatter.Indent();
            formatter.AppendLine("-->Inner Exception");
            EmitExceptionAtAGlance(formatter, 
                exception.InnerException);
            formatter.RevertIndent();
        }
    }

    private static void EmitExceptionStackTrace(
        CustomEventFormatter formatter, Exception exception)
    {
        formatter.AppendLine(exception.StackTrace);

        if (null != exception.InnerException)
        {
            // no point indenting
            // since stack traces typically wrap like crazy
            formatter.AppendLine();
            formatter.AppendLine("-->Inner exception stack trace:");
            EmitExceptionStackTrace(formatter, exception.InnerException);
        }
    }
}

And finally, here's a helper class that manages indentation levels for the output email message:

public class CustomEventFormatter
{
    const int TabSpaces = 4;

    StringBuilder sb = new StringBuilder();
    private int indentLevel;
    private bool startingNewLine = true;

    public void Indent()
    {
        ++indentLevel;
    }

    public void RevertIndent()
    {
        if (indentLevel > 0)
            --indentLevel;
    }

    public void Append(string text)
    {
        if (startingNewLine)
            EmitIndent();
        sb.Append(text);
        startingNewLine = false;
    }

    public void AppendLine(string lineOfText)
    {
        if (startingNewLine)
            EmitIndent();
        EmitIndent();
        sb.AppendLine(lineOfText);
        startingNewLine = true;
    }

    private void EmitIndent()
    {
        sb.Append(' ', TabSpaces * indentLevel);
    }

    public void AppendLine()
    {
        AppendLine(string.Empty);
    }

    public override string ToString()
    {
        return sb.ToString();
    }
}

Build this into a library application and reference it in your config file. Here's an example:

<healthMonitoring>
  <providers>
    <add name="mailWebEventProvider"
         type="MyMailWebEventProvider"
         to="web-fault@fabrikam.com"
         from="website@fabrikam.com"
         buffer="false"
         subjectPrefix="[WEB-ERROR]"
       />
  </providers>
  <rules>
    <add name="All Errors Email"
         eventName="All Errors"
         provider="mailWebEventProvider"
         profile="Default"
         minInstances="1"
         maxLimit="Infinite"
         minInterval="00:01:00"
         custom=""/>
  </rules>
</healthMonitoring>

ASP.NET's health monitoring system does list the inner exceptions (apparently up to a maximum depth of two, from spelunking the code with reflector), but it does not emit the stack traces for these exceptions, which would be really helpful. I've spent some time this morning trying to figure out how I'd customize things to emit this, and it looks like what I'd have to do is catch the exception and generate a custom WebEvent that overrides ToString(bool, bool) and does everything that WebRequestErrorEvent does, but also generate the inner stack trace.

That seems a bit ugly. A search for "ASP.NET web event inner exception stack trace" yielded no interesting results, so if you've dealt with this and have a cleaner solution, let me know. I'll post my solution once I get it worked out.

The designer, Carter Bryant, has been accused by Mattel of using Evidence Eliminator on his laptop computer just two days before investigators were due to copy its hard drive. Carter hasn't denied that the program was run on his computer, but he said it wasn't to destroy evidence. He said he had legitimate reasons to use the software. [...] But the wiper programs don't ensure a clean getaway. They leave behind a kind of digital calling card. "Not only do these programs leave a trace that they were used, they each have a distinctive fingerprint," Kessler said. "Evidence Eliminator leaves one that's different from Window Washer, and so on." It's the kind of information that can be brought up in court. And if the digital calling card was left by Evidence Eliminator, it could raise some eyebrows, even if the wiper was used for the most innocent of reasons.