...doctored photographs are the least of our worries. If you want to trick someone with a photograph, there are lots of easy ways to do it. You don't need Photoshop. You don't need sophisticated digital photo-manipulation. You don't need a computer. All you need to do is change the caption.

The photographs presented by Colin Powell at the United Nations in 2003 provide several examples. Photographs that were used to justify a war. And yet, the actual photographs are low-res, muddy aerial surveillance photographs of buildings and vehicles on the ground in Iraq. I'm not an aerial intelligence expert. I could be looking at anything. It is the labels, the captions, and the surrounding text that turn the images from one thing into another. Photographs presented by Colin Powell at the United Nations in 2003.

Powell was arguing that the Iraqis were doing something wrong, knew they were doing something wrong, and were trying to cover their tracks. Later, it was revealed that the captions were wrong. There was no evidence of chemical weapons and no evidence of concealment. Morris's mockery of the sweeping interpretations made in Powell's photographs.

There is a larger point. I don't know what these buildings were really used for. I don't know whether they were used for chemical weapons at one time, and then transformed into something relatively innocuous, in order to hide the reality of what was going on from weapons inspectors. But I do know that the yellow captions influence how we see the pictures. "Chemical Munitions Bunker" is different from "Empty Warehouse" which is different from "International House of Pancakes." The image remains the same but we see it differently.

Change the yellow labels, change the caption and you change the meaning of the photographs. You don't need Photoshop. That's the disturbing part. Captions do the heavy lifting as far as deception is concerned. The pictures merely provide the window-dressing. The unending series of errors engendered by falsely captioned photographs are rarely remarked on.

The first post described the problem: ASP.NET wasn't reporting inner exception stack traces.

The second post described my solution.

This post shows the code I used to solve the problem: a custom email provider for the Health Monitoring system in ASP.NET. Enjoy!

Here's the provider. Note that I opted *not* to build a buffering provider to keep things simple:

public class MyMailWebEventProvider : WebEventProvider
{
    string to;
    string from;
    string subjectPrefix;

    public override void Initialize(string name,
        NameValueCollection config)
    {
        base.Initialize(name, config);

        to = GetAndRemoveStringAttribute(config, "to", true);
        from = GetAndRemoveStringAttribute(config, "from", true);
        subjectPrefix = GetAndRemoveStringAttribute(config,
            "subjectPrefix", false);
    }
    public override void ProcessEvent(WebBaseEvent raisedEvent)
    {
        SendMail(raisedEvent);
    }

    private void SendMail(WebBaseEvent raisedEvent)
    {
        string subject = ComputeEmailSubject(raisedEvent);
        string body = ComputeEmailBody(raisedEvent);

        MailMessage msg = new MailMessage(from, to, subject, body);
        new SmtpClient().Send(msg);
    }

    private string ComputeEmailBody(WebBaseEvent raisedEvent)
    {
        WebRequestErrorEvent errorEvent =
            raisedEvent as WebRequestErrorEvent;
        if (null != errorEvent)
            return ErrorEventFormattingHelper.FormatRequestErrorEvent(errorEvent);
        else return raisedEvent.ToString();
    }

    private string ComputeEmailSubject(WebBaseEvent raisedEvent)
    {
        StringBuilder subjectBuilder = new StringBuilder();

        // surface some details in subject about error events
        WebBaseErrorEvent errorEvent = raisedEvent as WebBaseErrorEvent;
        if (null != errorEvent)
        {
            Exception unhandledException = errorEvent.ErrorException;

            // drill through reflection exceptions to show the root cause
            TargetInvocationException invocationException =
                unhandledException as TargetInvocationException;
            if (null != invocationException)
            {
                Exception innerException =
                    DrillIntoTargetInvocationException(invocationException);
                subjectBuilder.AppendFormat("{0}",
                    (innerException ?? invocationException).GetType().Name);
                if (null != innerException)
                    subjectBuilder.Append(" (via reflection)");
            }
            else subjectBuilder.Append(unhandledException.GetType().Name);
        }

        // if we've not got anything better
        // just show the event type in the subject
        if (0 == subjectBuilder.Length)
            subjectBuilder.AppendFormat("Event type: {0}",
                raisedEvent.GetType().Name);

        if (!string.IsNullOrEmpty(subjectPrefix)) {
            subjectBuilder.Insert(0, ' ');
            subjectBuilder.Insert(0, subjectPrefix);
        }
        return subjectBuilder.ToString();
    }

    /// <summary>
    /// Reflection often hides exception details, so we try to drill down
    /// through the plumbing exceptions to find a likely cause
    /// </summary>
    private Exception DrillIntoTargetInvocationException(
        TargetInvocationException outerException)
    {
        Exception innerException = outerException.InnerException;
        TargetInvocationException innerInvocationException =
            innerException as TargetInvocationException;
        if (null != innerInvocationException)
            return DrillIntoTargetInvocationException(innerInvocationException);
        else if (null != innerException)
            return innerException;
        else return null;
    }

    private static string GetAndRemoveStringAttribute(NameValueCollection config,
        string attributeName, bool required)
    {
        string value = config.Get(attributeName);
        if (required && string.IsNullOrEmpty(value))
            throw new ConfigurationErrorsException(string.Format(
                "Expected attribute {0}, which is missing or empty.",
                attributeName));
        config.Remove(attributeName);
        return value;
    }

    public override void Flush()
    {
        // nothing to do - this is not a buffering provider
    }

    public override void Shutdown()
    {
        // nothing to do here either
    }
}

Here's a helper class that formats the error messages the way I want to see them. Note that I've omitted some fields that I personally didn't care about, and I've reordered things a bit, so you might want to tweak this if you're going to use it in your own system.

internal static class ErrorEventFormattingHelper
{
    internal static string FormatRequestErrorEvent(
        WebRequestErrorEvent errorEvent)
    {
        CustomEventFormatter formatter = 
            new CustomEventFormatter();

        formatter.AppendLine(string.Format(
            "Unhandled Exception in {0}:",
            WebBaseEvent.ApplicationInformation
            .ApplicationVirtualPath));
        formatter.Indent();
        EmitExceptionAtAGlance(formatter, 
            errorEvent.ErrorException);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Exception stack trace(s):");
        EmitExceptionStackTrace(formatter, 
            errorEvent.ErrorException);

        formatter.AppendLine();
        formatter.AppendLine("Event information:");
        formatter.Indent();
        EmitEventInfo(formatter, errorEvent);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Application information:");
        formatter.Indent();
        EmitApplicationInfo(formatter, 
            WebBaseEvent.ApplicationInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Process/thread information:");
        formatter.Indent();
        EmitProcessInfo(formatter, 
            errorEvent.ProcessInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Request information:");
        formatter.Indent();
        EmitRequestInfo(formatter, 
            errorEvent.RequestInformation);
        formatter.RevertIndent();

        return formatter.ToString();
    }

    private static void EmitEventInfo(
        CustomEventFormatter formatter,
        WebBaseEvent theEvent)
    {
        formatter.AppendLine(string.Format(
            "Event code: {0}",
            theEvent.EventCode.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event message: {0}", 
            theEvent.Message));
        formatter.AppendLine(string.Format(
            "Event time: {0}", 
            theEvent.EventTime.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event ID: {0}", 
            theEvent.EventID.ToString("N", 
            CultureInfo.InvariantCulture)));
    }

    private static void EmitApplicationInfo(
        CustomEventFormatter formatter, 
        WebApplicationInformation appInfo)
    {
        formatter.AppendLine(string.Format(
            "Application domain: {0}", 
            appInfo.ApplicationDomain));
        formatter.AppendLine(string.Format(
            "Application Virtual Path: {0}", 
            appInfo.ApplicationVirtualPath));
        formatter.AppendLine(string.Format(
            "Application Physical Path: {0}", 
            appInfo.ApplicationPath));
    }

    private static void EmitProcessInfo(
        CustomEventFormatter formatter, 
        WebProcessInformation webProcessInfo)
    {
        formatter.AppendLine(string.Format(
            "Process ID: {0}", 
            webProcessInfo.ProcessID.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Process name: {0}", 
            webProcessInfo.ProcessName));
        formatter.AppendLine(string.Format(
            "Account name: {0}", 
            webProcessInfo.AccountName));
    }

    private static void EmitRequestInfo(
        CustomEventFormatter formatter, 
        WebRequestInformation webRequestInfo)
    {
        string name = null;
        if (webRequestInfo.Principal != null)
            name = webRequestInfo.Principal.Identity.Name;

        formatter.AppendLine(string.Format(
            "Request URL: {0}", 
            webRequestInfo.RequestUrl));
        formatter.AppendLine(string.Format(
            "Request path: {0}", 
            webRequestInfo.RequestPath));
        formatter.AppendLine(string.Format(
            "User name: {0}", 
            name ?? "[ANONYMOUS]"));
        formatter.AppendLine(string.Format(
            "User host address: {0}", 
            webRequestInfo.UserHostAddress));
    }

    private static void EmitExceptionAtAGlance(
        CustomEventFormatter formatter, 
        Exception exception)
    {
        formatter.AppendLine(string.Format(
            "Type: {0}", 
            exception.GetType().Name));
        formatter.AppendLine(string.Format(
            "Message: {0}", 
            exception.Message));
        if (null != exception.InnerException)
        {
            formatter.Indent();
            formatter.AppendLine("-->Inner Exception");
            EmitExceptionAtAGlance(formatter, 
                exception.InnerException);
            formatter.RevertIndent();
        }
    }

    private static void EmitExceptionStackTrace(
        CustomEventFormatter formatter, Exception exception)
    {
        formatter.AppendLine(exception.StackTrace);

        if (null != exception.InnerException)
        {
            // no point indenting
            // since stack traces typically wrap like crazy
            formatter.AppendLine();
            formatter.AppendLine("-->Inner exception stack trace:");
            EmitExceptionStackTrace(formatter, exception.InnerException);
        }
    }
}

And finally, here's a helper class that manages indentation levels for the output email message:

public class CustomEventFormatter
{
    const int TabSpaces = 4;

    StringBuilder sb = new StringBuilder();
    private int indentLevel;
    private bool startingNewLine = true;

    public void Indent()
    {
        ++indentLevel;
    }

    public void RevertIndent()
    {
        if (indentLevel > 0)
            --indentLevel;
    }

    public void Append(string text)
    {
        if (startingNewLine)
            EmitIndent();
        sb.Append(text);
        startingNewLine = false;
    }

    public void AppendLine(string lineOfText)
    {
        if (startingNewLine)
            EmitIndent();
        EmitIndent();
        sb.AppendLine(lineOfText);
        startingNewLine = true;
    }

    private void EmitIndent()
    {
        sb.Append(' ', TabSpaces * indentLevel);
    }

    public void AppendLine()
    {
        AppendLine(string.Empty);
    }

    public override string ToString()
    {
        return sb.ToString();
    }
}

Build this into a library application and reference it in your config file. Here's an example:

<healthMonitoring>
  <providers>
    <add name="mailWebEventProvider"
         type="MyMailWebEventProvider"
         to="web-fault@fabrikam.com"
         from="website@fabrikam.com"
         buffer="false"
         subjectPrefix="[WEB-ERROR]"
       />
  </providers>
  <rules>
    <add name="All Errors Email"
         eventName="All Errors"
         provider="mailWebEventProvider"
         profile="Default"
         minInstances="1"
         maxLimit="Infinite"
         minInterval="00:01:00"
         custom=""/>
  </rules>
</healthMonitoring>

I realize that security is a system level issue and it takes a long time to change things at that level, but what's more concerning to me is that the typical infosec mindset remains the same. Should we be surprised by rampant phishing and fraud? I am frankly surprised the numbers are so low given the opportunities that the attackers have via the glacial pace of security improvements. Its been three years since that list and I could write the same exact one today for SOAP, REST, SOA, Web 2.0 whatever. Maybe the main reason, beyond failure of imagination, why infosec is so far behind developers is that infosec lacks tools. Developers automate everything possible. Security doesn't. The most promising thing about static analysis is not the ability to find everything, its the ability to find many important things in an automated way. Infosec needs to stop giving people fish and teaching people to fish. Look at Fortify's vulncat site which has a Taxonomy of Coding Errors. Fortify's Seven (plus one) pernicious kingdoms are:

What I found is that the two providers were using entirely different means to format the events! The email provider calls ToString(bool, bool) on the event to ask it to format itself. But the EventLogWebEventProvider does its own formatting of individual fields of the event. Indeed, its ProcessEvent method has a big list of checks:

if (eventRaised is WebBaseErrorEvent)
    AddErrorStuff();
if (eventRaised is WebAuthenticationSuccessAuditEvent)
    AddLogonStuff();

So it seemed like a better approach would be to write my own provider. I left the event log provider alone, and I wrote a custom email provider to display errors in a more useful way. This also allowed me to drop some fields from the event report that aren't useful for us. And I was able to construct a much more concise and useful subject line (the subject line that SimpleMailWebEventProvider uses is rather clunky since it assumes it might be spitting out a whole bunch of buffered events in one go).

Not only does my provider include the stack traces for all of the exceptions in the chain, but in the subject line, I display the type of error that is at the root of the problem. So if I am formatting a TargetInvocationException, I drill into its InnerException chain until I find a different exception type, and display that exception type instead.

Oh, one other benefit of building the custom provider instead of using a custom WebEvent was that I was then able to remove the Error handler from global.asax. All I had to do was replace the SimpleMailWebEventProvider with my own provider, and I got the behavior I wanted. Now my email notifications include detailed stack traces.

I'll post the code for this provider once it's run for a little while in production and I'm satisfied that it works reasonably well.

• Open Scripting Architecture—Privilege elevation bug when loading plugins.
• CarbonCore—A stack overflow in handling long file names. Potential code execution.
• CoreGraphics—2 bugs, both code execution, one for malicious graphics the other for malicious PDFs.
• Data Detectors Engine—Engine may crash when parsing maliciously crafted content.
• Disk Utility—A local user may obtain System privileges.
• OpenLDAP—An ASN parsing bug can lead to a crash.
• OpenSSL—A range checking error from last September (Red Hat patched it in 2 weeks) can lead to remote code execution.
• PHP—5 different bugs, the worst of which can lead to remote code execution.
• QuickLook—A maliciously-crafted Microsoft Office file can cause QuickLooks to crash or allow remote code execution.
• rsync—Path validation errors, which were also reported in 2007, are resolved.

• Open Scripting Architecture—Privilege elevation bug when loading plug-ins.
• CarbonCore—A stack overflow in handling long file names. Potential code execution.
• CoreGraphics—Two bugs, both code execution, one for malicious graphics, the other for malicious PDFs.
• Data Detectors Engine—Engine may crash when parsing maliciously crafted content.
• Disk Utility—A local user may obtain System privileges.
• OpenLDAP—An ASN parsing bug can lead to a crash.
• OpenSSL—A range checking error from last September (Red Hat patched it in two weeks) can lead to remote code execution.
• PHP—Five different bugs, the worst of which can lead to remote code execution.
• QuickLook—A maliciously crafted Microsoft Office file can cause QuickLooks to crash or allow remote code execution.
• rsync—Path validation errors, which were also reported in 2007, are resolved.

LogLogic Lunch and Learn Presentation
- "Worst Practices" of Log Management
- Speaker: Dr. Anton Chuvakin, GCIA, GCIH, GCFA
- Wednesday, July 23rd, 2008 * 12:30pm - 1:15 pm

Want to learn all the embarrassing mistakes and pitfalls that await you on the path to log management nirvana? Attend "'Worst Practices' of Log Management" presentation by LogLogic's Logging Evangelist Dr Anton Chuvakin that covers all the things that can go wrong while planning, evaluating, deploying and running a log management solution. Insufficient planning, unrealistic expectations, choosing tools on price alone, lack of logging configuration guidance are among such "worst practices." Each common "worst practice" will be accompanied by suggestions to avoid the errors and do things correctly! Everybody touts "best practices", but this is the place to learn how to avoid the opposite - and have fun in the process.

if you want to meet, drop me an email/call or just show up for "lunch and learn." Unfortunately, I am going back right after my presentation tomorrow...

My reply is that I don’t care must about reported dollar sales because these numbers are, for the most part, meaningless and mythical at this point in time. Large companies sell enterprise licenses and make up allocated numbers for the CEP/EP share of the pie based on a subjective formulation. They can sell an enterprise site license for $2,000,000 USD that includes CEP/EP software and claim 20% is CEP revenue, regardless of if the software is used or not.

Small companies nearly give software away with the hope of developing a strong public reference client, which are few and far between in 2008. Soon, I will start a Google spreadsheet, similar to what we did last year on this topic. Some folks don’t seem to like this initiative because, unfortunately, we will see that for this half of 2008, this year has been very lean for CEP/EP. Some would prefer I blog as a cheerleading evangelist versus an objective analyst. Go Fight Win! Rah Rah Rah!

Much of the current gloomy situation, of course, is because the entire market has fallen and IT spending is down. Financial companies announce record losses. Bankruptcies and restructuring are in the daily news.

In this depressed market, some companies have tried to tie the subprime crash to CEP, somehow implying that CEP would have helped, but that positioning is mostly fantasy. I work in the field of risk management at the corporate level and the current problems are not caused by a lack of technology, it is simply corporate greed - corporations taking high risks to stay competitive in a bull market and then they experience a frighteningly negative reversal during a market free fall.

Of course, the US Federal Reserve did not help matters when they decided to poke a gaping hole in the real estate bubble by dramatically raising interest rates without thinking about how they would manage the consequences, but that is another story! After all, the current top government executives in Washington DC are so politically, scientifically and economically incompetent that all we can do is hold our breath and count the days.

One risk management colleague often says,

“When then tide is high, you can’t see that the swimmers are naked.”

….and so it is in business. The current problems in the global market are based on human, social, and political errors and incompetence; nothing that technology can cure at this point in the game. So, the entire market is in decline, and folks are overhyping all software to keep the buzz going, as if CEP or SOA or BPM would have helped stopped the current global meltdown. Yes, CEP can stop global warming! Buy one today, save a cute polar bear!

Then again, maybe we only need a CEP engine in Washington; even a simple rules-based one would be good. Naturally, some would suggest that we need Neural Nets and Bayesian analytics; but I think just a simple rules-engine looking out the window that can process if-then-else conditions would be a great improvement over the mind-numbing leadership in Washington today.