HANSEI - WHAT IS “RELENTLESS REFLECTION?” - And why we’re talking about it in the context of Risk Analysis.

Recall from yesterday’s post about how I got to thinking about the concept of Hansei-Kaizen, “relentless reflection” and “continuous improvement” and how we might apply that to risk management.  It’s a concept born of Toyota and is, in some way, the foundation for “Lean” production.

Call me biased, but I think that Hansei - the act of ‘relentless reflection’ made structured is the analytical function.  And I hate to debate (post-mortem) the father of Toyota quality success when he says that Hansei is the “check” in Plan/Do/Check/Act, but I think that Hansei also applies to the “Plan” of the P/D/C/A or Deming cycle.

You’ll recall the P/D/C/A cycle can be thought of even as an implementation of Scientific Method, in that it is Observation & Hypothesis Creation (P), Experiment (D), Analysis (Check), and Act (Revise/New Hypothesis, etc…).  Well then as such, the Hypothesis creation involves creating a model or creating an expected outcome for data using the currently accepted model.

So in our industry there is an opportunity for Relentless Reflection in both the Observation and Hypothesis (Plan) creation steps, and the Check step.  We create an estimate for control strength, or probable losses in the context of risk- then we go to Experiment step.  That hypothesis can be put it into production, have an audit, have a penetration test, whatever, in the context of the Do step.  BTW - using Hansei/Analytics in Plan is one way that strong analytical functions can really make penetration testing more useful - as a means to test the estimates and inputs into a model.  It’s Penetration Testing 2.0!  (<- tongue fully in cheek, yes)

Those who are versed in the reasons to merge Six Sigma and Lean together are probably already seeing where I’m going with this today.  But before you think that a simple DMAIC function is all that is needed to create proper “Hansei”, let me encourage you to keep reading.

Now if the analytical function can said to be “reflection”, why must it be relentless?

One word.  Change. There are essentially four separate “landscapes” or sources of change that we face (more on those tomorrow).  But anyone who has tried to manage system compliance, log management or policy exceptions knows that change is possibly the most difficult thing we security professionals must manage.  And when you think about it, there aren’t too many other business functions like information security where significant visibility and insight about the environment is needed for “complete” information (get bullish on Log Management is my recommendation).

HANSEI STEPS ADAPTED TO INFORMATION SECURITY

This is one of those quality control concepts that we can mangle adopt.  At Toyota, Hansei-Kaizen includes the following basic steps:

1. Initial problem perception
2. Clarify the problem
3. Locate area/point of cause
4. Investigate root cause (using an ask why 5 times approach)
5. Countermeasure
6. Evaluate
7. Standardize

Now it’s important to note that part of this includes the concept of Go See For Yourself, called “Gemba“.  Gemba can be translated as “the actual place” or “the place where virtue or truth is found.” At Toyota this might mean going to the shop floor to see the issue at hand in the production line.  But for us, that’s a problem because we live in the virtual world.  There’s usually not much use in hanging out in the wiring closets to try to see the problems.

But if you combine the concept of Gemba with the concept of “Nemawashi” –the process of discussing problems and potential solutions with all those affected- we can forge a similar concept using risk analysis.  That is discussing the issue and the risk associated with an issue (what some people would call “risk management”) with the business/LOB/data owner and let them accept authority and the risk decision.  We, the risk analyst, our goal is simply to perform items 1-5 (presenting countermeasure options that include transferring or accepting risk).  By going to the line of business and involving them, responsibility is shared.  Also, if you structure organizational behavior right, personal risk is transferred!

This sort of approach is also in harmony with concepts like “mutual ownership of problems,” or “genchi genbutsu,” (solving problems at the source instead of behind desks), and the “kaizen mind,” (an unending sense of crisis behind the company’s constant drive to improve).

One of the criticisms I have with the way most people try to implement DMAIC into “Lean”

REQUIREMENTS

Now to get this done, I really see three significant requirements.

1.)  A change in political structure.

2.)  Models that provide consistent, defensible analysis.

3.)  A Quantitative approach.  This means using actual units of measurement (not just amorphous percents, ordinal scales, etc.)  for risk and it’s subsequent factors.  Sure there are times when Q&D qualitative approaches are acceptable, but policy should be to have quantitative analysis whenever and wherever possible.

That last item - the quantitative approach - is really quite important.  And the reasons why will be discussed further in tomorrow’s post:

“What should we be reflecting about? & What is needed for reflection?”

P.S.  Your comments and suggestions, as always, are welcome.

P.P.S  Those who may be familiar with Lean/SixSigma/Kaizen sorts of mashups may be thinking - “hey, an Analytical step is built into SixSigma”.  Well, yes there is some prevision for analytical functions based on statistics, but I find SixSigma geared towards creating a State of Knowledge about operational processes, not towards creating a State of Wisdom for CISO’s around security & risks “big questions”.  In otherwords, the analytical function in DMAIC is in the context of Kaizen, and a different step than “reflective” analytics.

MI5 has concluded that there is no easy way to identify those who become involved in terrorism in Britain, according to a classified internal research document on radicalisation seen by the Guardian.

[...]

The main findings include:

• The majority are British nationals and the remainder, with a few exceptions, are here legally. Around half were born in the UK, with others migrating here later in life. Some of these fled traumatic experiences and oppressive regimes and claimed UK asylum, but more came to Britain to study or for family or economic reasons and became radicalised many years after arriving.

• Far from being religious zealots, a large number of those involved in terrorism do not practise their faith regularly. Many lack religious literacy and could actually be regarded as religious novices. Very few have been brought up in strongly religious households, and there is a higher than average proportion of converts. Some are involved in drug-taking, drinking alcohol and visiting prostitutes. MI5 says there is evidence that a well-established religious identity actually protects against violent radicalisation.

• The "mad and bad" theory to explain why people turn to terrorism does not stand up, with no more evidence of mental illness or pathological personality traits found among British terrorists than is found in the general population.

• British-based terrorists are as ethnically diverse as the UK Muslim population, with individuals from Pakistani, Middle Eastern and Caucasian backgrounds. MI5 says assumptions cannot be made about suspects based on skin colour, ethnic heritage or nationality.

• Most UK terrorists are male, but women also play an important role. Sometimes they are aware of their husbands', brothers' or sons' activities, but do not object or try to stop them.

• While the majority are in their early to mid-20s when they become radicalised, a small but not insignificant minority first become involved in violent extremism at over the age of 30.

• Far from being lone individuals with no ties, the majority of those over 30 have steady relationships, and most have children. MI5 says this challenges the idea that terrorists are young men driven by sexual frustration and lured to "martyrdom" by the promise of beautiful virgins waiting for them in paradise. It is wrong to assume that someone with a wife and children is less likely to commit acts of terrorism.

• Those involved in British terrorism are not unintelligent or gullible, and nor are they more likely to be well-educated; their educational achievement ranges from total lack of qualifications to degree-level education. However, they are almost all employed in low-grade jobs.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.   I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

1. Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.  There will be no physical access to the server. No exceptions.
2. Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
3. Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
4. Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
5. Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,  10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
6. Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
7. If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
8. It should send an email to one or more email addresses with a message, for example:  “New Bot Detected - Confidence 8″ with IP address, etc. in the message.  Another example would be an email, “Bot Blocked” - with details, etc.
9. You cannot automatically block any traffic that is not a bot.  Blocking one “non-bot” results in failure, no exceptions.
10. The Prize:  The winner will get their logo (w/link) on this site in a block called “Bot Hunter Winner” (or something like that.)

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!  Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.   So, you will need to work with both long and short time windows.  Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

he Court of Federal Claims that first heard the case threw it out, and the new Appellate ruling upholds that decision. The reasoning behind the decisions focuses on the US government's sovereign immunity, which the court describes thusly: "The United States, as [a] sovereign, 'is immune from suit save as it consents to be sued . . . and the terms of its consent to be sued in any court define that court's jurisdiction to entertain the suit.'"

In the case of copyright law, the US has given up much of its immunity, but the government retains a few noteworthy exceptions. The one most relevant to this case says that when a government employee is in a position to induce the use of the copyrighted material, "[the provision] does not provide a Government employee a right of action 'where he was in a position to order, influence, or induce use of the copyrighted work by the Government.'" Given that Davenport used his position as part of the relevant Air Force office to get his peers to use his software, the case fails this test.

But the court also addressed the DMCA claims made by Blueport, and its decision here is quite striking. "The DMCA itself contains no express waiver of sovereign immunity," the judge wrote, "Indeed, the substantive prohibitions of the DMCA refer to individual persons, not the Government." Thus, because sovereign immunity is not explicitly eliminated, and the phrasing of the statute does not mention organizations, the DMCA cannot be applied to the US government, even in cases where the more general immunity to copyright claims does not apply.

It appears that Congress took a "do as we say, not as we need to do" approach to strengthening digital copyrights.

Q1: How do you handle variety of log sources? There are so many, almost beyond my capability.

A1: Sorry to ponder the meaning of "is" here, but what is meant by "handle"? It is really not that hard to collect logs from a large number of diverse sources (as long as the logs can be delivered via syslog or exist as files and can be collected). Now, there will certainly be challenges  when the volume of logs gets large, but if by "handle" you mean "collect + store", it is really not that hard, given the right tools. Now, if "handle" means "make sense of what all those logs are trying to tell you," it is a different story altogether.

Q2: You talked about the importance of logging; however for an intermediate or novice admin what are the starting steps .. what are the minimal logs they should start at once?

A2: Answered in "Log Management - Day 1" If you want a simple list of things to "enable today,"  I cannot really answer it since I know neither your needs, nor your environment. In other words, this is the "what is the meaning of life question?" :-)

Q3: What regulations, rules or guidance exist regarding sharing or visibility of logs to users?

A3: PCI DSS says in Requirement 10.5:  "Secure audit trails so they cannot be altered.
10.5.1 Limit viewing of audit trails to those with a job-related need
10.5.2 Protect audit trail files from unauthorized modifications
10.5.3 Promptly back-up audit trail files to a centralized log server or media that is difficult to
alter"

NIST guidance for FISMA also says something similar (for example, look in NIST 800-92 doc). Overall, log protection and security are mentioned in many other regulations as well.

Q4: Privileged groups membership monitoring in AD one of the most important from my point of view. However I did not find effective way to monitor/report on changes in those groups. Any recommendations?

A4: This is indeed a tricky one which might take more space to answer than I have here; it might also take you 'beyond logs.' One good source of information is Randy Smith's site and, specifically, his webinar on 'Active Directory "Logging Gap"' (here somewhere) - which covers how to audit things of that sort when then native logging is not sufficient.

Q5: How I can learn what exactly I need to log?

A5: OMG, this is a $1,000,000 question :-) Let me answer "how can I learn" part and not the "what exactly I need to log part,"  (also see discussion on "MUST-DO Logging for PCI?") as it is actually answerable. To learn what you need to log, first ask "Why?" (and then see this) - basically establish what you want to accomplish with logs, catalogue your systems, figure how to tweak the logging knobs - and then do it!

Q6: How granular should logging be? What is your recommendation for enterprise servers like domain servers and Windows servers?

A6: Again, too long to answer here in details (it will become a subject of a longer blog post later), but some pointers follow: here for Windows (MS site also have a few recommendations on audit policies)

Q7: What is "more control" and what is "less control" that you mention in the webcast? Can you give an example?

A7: OK, I did say that "sometimes when you implement more controls, you actually have less control." What do I mean? If you buy a firewall (a network security control) and then - over time, of course - configure it with 7800 rules (!) that are supposed to give you control over who can and cannot access your network, you will not gain control over your environment. You will actually be less in control of who is touching your network, compared to, say, having only 20 rules.

Q8: What about mandated NIST controls for government systems? Auditing is a specific control for Moderate and High risk systems. What list of events do you recommend for auditing?

A8: This is too long to answer here, but NIST 800-92 Guide is a really good source of such info ("Guide to Computer Security Log Management [PDF]") Also, see my presentation on NIST 800-92 Guide in the Real World.

Q9: The issue that many organizations get stuck on, is the monitoring process, and defining what exceptions to monitor for? Is there guidance / framework for this? How much of it is system specific and how much is applicable generally to all systems?

A9: I outlined some general ideas back in 2004 via this presentation (note to self - update that to be more 2008-relevant); it is mostly general, but also has pointers to specific system. Keep in mind that it is focused on security, not operational monitoring (which is often no less important - in fact, often MORE important)

Enjoy! Sorry for being brief with some of the answers - I am woefully late with this even as they are...

Other questions that I answered in the past:

• More Log Management Questions - Answered!
• Some Burning Logging Questions - Answered!

The first post described the problem: ASP.NET wasn't reporting inner exception stack traces.

The second post described my solution.

This post shows the code I used to solve the problem: a custom email provider for the Health Monitoring system in ASP.NET. Enjoy!

Here's the provider. Note that I opted *not* to build a buffering provider to keep things simple:

public class MyMailWebEventProvider : WebEventProvider
{
    string to;
    string from;
    string subjectPrefix;

    public override void Initialize(string name,
        NameValueCollection config)
    {
        base.Initialize(name, config);

        to = GetAndRemoveStringAttribute(config, "to", true);
        from = GetAndRemoveStringAttribute(config, "from", true);
        subjectPrefix = GetAndRemoveStringAttribute(config,
            "subjectPrefix", false);
    }
    public override void ProcessEvent(WebBaseEvent raisedEvent)
    {
        SendMail(raisedEvent);
    }

    private void SendMail(WebBaseEvent raisedEvent)
    {
        string subject = ComputeEmailSubject(raisedEvent);
        string body = ComputeEmailBody(raisedEvent);

        MailMessage msg = new MailMessage(from, to, subject, body);
        new SmtpClient().Send(msg);
    }

    private string ComputeEmailBody(WebBaseEvent raisedEvent)
    {
        WebRequestErrorEvent errorEvent =
            raisedEvent as WebRequestErrorEvent;
        if (null != errorEvent)
            return ErrorEventFormattingHelper.FormatRequestErrorEvent(errorEvent);
        else return raisedEvent.ToString();
    }

    private string ComputeEmailSubject(WebBaseEvent raisedEvent)
    {
        StringBuilder subjectBuilder = new StringBuilder();

        // surface some details in subject about error events
        WebBaseErrorEvent errorEvent = raisedEvent as WebBaseErrorEvent;
        if (null != errorEvent)
        {
            Exception unhandledException = errorEvent.ErrorException;

            // drill through reflection exceptions to show the root cause
            TargetInvocationException invocationException =
                unhandledException as TargetInvocationException;
            if (null != invocationException)
            {
                Exception innerException =
                    DrillIntoTargetInvocationException(invocationException);
                subjectBuilder.AppendFormat("{0}",
                    (innerException ?? invocationException).GetType().Name);
                if (null != innerException)
                    subjectBuilder.Append(" (via reflection)");
            }
            else subjectBuilder.Append(unhandledException.GetType().Name);
        }

        // if we've not got anything better
        // just show the event type in the subject
        if (0 == subjectBuilder.Length)
            subjectBuilder.AppendFormat("Event type: {0}",
                raisedEvent.GetType().Name);

        if (!string.IsNullOrEmpty(subjectPrefix)) {
            subjectBuilder.Insert(0, ' ');
            subjectBuilder.Insert(0, subjectPrefix);
        }
        return subjectBuilder.ToString();
    }

    /// <summary>
    /// Reflection often hides exception details, so we try to drill down
    /// through the plumbing exceptions to find a likely cause
    /// </summary>
    private Exception DrillIntoTargetInvocationException(
        TargetInvocationException outerException)
    {
        Exception innerException = outerException.InnerException;
        TargetInvocationException innerInvocationException =
            innerException as TargetInvocationException;
        if (null != innerInvocationException)
            return DrillIntoTargetInvocationException(innerInvocationException);
        else if (null != innerException)
            return innerException;
        else return null;
    }

    private static string GetAndRemoveStringAttribute(NameValueCollection config,
        string attributeName, bool required)
    {
        string value = config.Get(attributeName);
        if (required && string.IsNullOrEmpty(value))
            throw new ConfigurationErrorsException(string.Format(
                "Expected attribute {0}, which is missing or empty.",
                attributeName));
        config.Remove(attributeName);
        return value;
    }

    public override void Flush()
    {
        // nothing to do - this is not a buffering provider
    }

    public override void Shutdown()
    {
        // nothing to do here either
    }
}

Here's a helper class that formats the error messages the way I want to see them. Note that I've omitted some fields that I personally didn't care about, and I've reordered things a bit, so you might want to tweak this if you're going to use it in your own system.

internal static class ErrorEventFormattingHelper
{
    internal static string FormatRequestErrorEvent(
        WebRequestErrorEvent errorEvent)
    {
        CustomEventFormatter formatter = 
            new CustomEventFormatter();

        formatter.AppendLine(string.Format(
            "Unhandled Exception in {0}:",
            WebBaseEvent.ApplicationInformation
            .ApplicationVirtualPath));
        formatter.Indent();
        EmitExceptionAtAGlance(formatter, 
            errorEvent.ErrorException);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Exception stack trace(s):");
        EmitExceptionStackTrace(formatter, 
            errorEvent.ErrorException);

        formatter.AppendLine();
        formatter.AppendLine("Event information:");
        formatter.Indent();
        EmitEventInfo(formatter, errorEvent);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Application information:");
        formatter.Indent();
        EmitApplicationInfo(formatter, 
            WebBaseEvent.ApplicationInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Process/thread information:");
        formatter.Indent();
        EmitProcessInfo(formatter, 
            errorEvent.ProcessInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Request information:");
        formatter.Indent();
        EmitRequestInfo(formatter, 
            errorEvent.RequestInformation);
        formatter.RevertIndent();

        return formatter.ToString();
    }

    private static void EmitEventInfo(
        CustomEventFormatter formatter,
        WebBaseEvent theEvent)
    {
        formatter.AppendLine(string.Format(
            "Event code: {0}",
            theEvent.EventCode.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event message: {0}", 
            theEvent.Message));
        formatter.AppendLine(string.Format(
            "Event time: {0}", 
            theEvent.EventTime.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event ID: {0}", 
            theEvent.EventID.ToString("N", 
            CultureInfo.InvariantCulture)));
    }

    private static void EmitApplicationInfo(
        CustomEventFormatter formatter, 
        WebApplicationInformation appInfo)
    {
        formatter.AppendLine(string.Format(
            "Application domain: {0}", 
            appInfo.ApplicationDomain));
        formatter.AppendLine(string.Format(
            "Application Virtual Path: {0}", 
            appInfo.ApplicationVirtualPath));
        formatter.AppendLine(string.Format(
            "Application Physical Path: {0}", 
            appInfo.ApplicationPath));
    }

    private static void EmitProcessInfo(
        CustomEventFormatter formatter, 
        WebProcessInformation webProcessInfo)
    {
        formatter.AppendLine(string.Format(
            "Process ID: {0}", 
            webProcessInfo.ProcessID.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Process name: {0}", 
            webProcessInfo.ProcessName));
        formatter.AppendLine(string.Format(
            "Account name: {0}", 
            webProcessInfo.AccountName));
    }

    private static void EmitRequestInfo(
        CustomEventFormatter formatter, 
        WebRequestInformation webRequestInfo)
    {
        string name = null;
        if (webRequestInfo.Principal != null)
            name = webRequestInfo.Principal.Identity.Name;

        formatter.AppendLine(string.Format(
            "Request URL: {0}", 
            webRequestInfo.RequestUrl));
        formatter.AppendLine(string.Format(
            "Request path: {0}", 
            webRequestInfo.RequestPath));
        formatter.AppendLine(string.Format(
            "User name: {0}", 
            name ?? "[ANONYMOUS]"));
        formatter.AppendLine(string.Format(
            "User host address: {0}", 
            webRequestInfo.UserHostAddress));
    }

    private static void EmitExceptionAtAGlance(
        CustomEventFormatter formatter, 
        Exception exception)
    {
        formatter.AppendLine(string.Format(
            "Type: {0}", 
            exception.GetType().Name));
        formatter.AppendLine(string.Format(
            "Message: {0}", 
            exception.Message));
        if (null != exception.InnerException)
        {
            formatter.Indent();
            formatter.AppendLine("-->Inner Exception");
            EmitExceptionAtAGlance(formatter, 
                exception.InnerException);
            formatter.RevertIndent();
        }
    }

    private static void EmitExceptionStackTrace(
        CustomEventFormatter formatter, Exception exception)
    {
        formatter.AppendLine(exception.StackTrace);

        if (null != exception.InnerException)
        {
            // no point indenting
            // since stack traces typically wrap like crazy
            formatter.AppendLine();
            formatter.AppendLine("-->Inner exception stack trace:");
            EmitExceptionStackTrace(formatter, exception.InnerException);
        }
    }
}

And finally, here's a helper class that manages indentation levels for the output email message:

public class CustomEventFormatter
{
    const int TabSpaces = 4;

    StringBuilder sb = new StringBuilder();
    private int indentLevel;
    private bool startingNewLine = true;

    public void Indent()
    {
        ++indentLevel;
    }

    public void RevertIndent()
    {
        if (indentLevel > 0)
            --indentLevel;
    }

    public void Append(string text)
    {
        if (startingNewLine)
            EmitIndent();
        sb.Append(text);
        startingNewLine = false;
    }

    public void AppendLine(string lineOfText)
    {
        if (startingNewLine)
            EmitIndent();
        EmitIndent();
        sb.AppendLine(lineOfText);
        startingNewLine = true;
    }

    private void EmitIndent()
    {
        sb.Append(' ', TabSpaces * indentLevel);
    }

    public void AppendLine()
    {
        AppendLine(string.Empty);
    }

    public override string ToString()
    {
        return sb.ToString();
    }
}

Build this into a library application and reference it in your config file. Here's an example:

<healthMonitoring>
  <providers>
    <add name="mailWebEventProvider"
         type="MyMailWebEventProvider"
         to="web-fault@fabrikam.com"
         from="website@fabrikam.com"
         buffer="false"
         subjectPrefix="[WEB-ERROR]"
       />
  </providers>
  <rules>
    <add name="All Errors Email"
         eventName="All Errors"
         provider="mailWebEventProvider"
         profile="Default"
         minInstances="1"
         maxLimit="Infinite"
         minInterval="00:01:00"
         custom=""/>
  </rules>
</healthMonitoring>

What I found is that the two providers were using entirely different means to format the events! The email provider calls ToString(bool, bool) on the event to ask it to format itself. But the EventLogWebEventProvider does its own formatting of individual fields of the event. Indeed, its ProcessEvent method has a big list of checks:

if (eventRaised is WebBaseErrorEvent)
    AddErrorStuff();
if (eventRaised is WebAuthenticationSuccessAuditEvent)
    AddLogonStuff();

So it seemed like a better approach would be to write my own provider. I left the event log provider alone, and I wrote a custom email provider to display errors in a more useful way. This also allowed me to drop some fields from the event report that aren't useful for us. And I was able to construct a much more concise and useful subject line (the subject line that SimpleMailWebEventProvider uses is rather clunky since it assumes it might be spitting out a whole bunch of buffered events in one go).

Not only does my provider include the stack traces for all of the exceptions in the chain, but in the subject line, I display the type of error that is at the root of the problem. So if I am formatting a TargetInvocationException, I drill into its InnerException chain until I find a different exception type, and display that exception type instead.

Oh, one other benefit of building the custom provider instead of using a custom WebEvent was that I was then able to remove the Error handler from global.asax. All I had to do was replace the SimpleMailWebEventProvider with my own provider, and I got the behavior I wanted. Now my email notifications include detailed stack traces.

I'll post the code for this provider once it's run for a little while in production and I'm satisfied that it works reasonably well.