The first post described the problem: ASP.NET wasn't reporting inner exception stack traces.

The second post described my solution.

This post shows the code I used to solve the problem: a custom email provider for the Health Monitoring system in ASP.NET. Enjoy!

Here's the provider. Note that I opted *not* to build a buffering provider to keep things simple:

public class MyMailWebEventProvider : WebEventProvider
{
    string to;
    string from;
    string subjectPrefix;

    public override void Initialize(string name,
        NameValueCollection config)
    {
        base.Initialize(name, config);

        to = GetAndRemoveStringAttribute(config, "to", true);
        from = GetAndRemoveStringAttribute(config, "from", true);
        subjectPrefix = GetAndRemoveStringAttribute(config,
            "subjectPrefix", false);
    }
    public override void ProcessEvent(WebBaseEvent raisedEvent)
    {
        SendMail(raisedEvent);
    }

    private void SendMail(WebBaseEvent raisedEvent)
    {
        string subject = ComputeEmailSubject(raisedEvent);
        string body = ComputeEmailBody(raisedEvent);

        MailMessage msg = new MailMessage(from, to, subject, body);
        new SmtpClient().Send(msg);
    }

    private string ComputeEmailBody(WebBaseEvent raisedEvent)
    {
        WebRequestErrorEvent errorEvent =
            raisedEvent as WebRequestErrorEvent;
        if (null != errorEvent)
            return ErrorEventFormattingHelper.FormatRequestErrorEvent(errorEvent);
        else return raisedEvent.ToString();
    }

    private string ComputeEmailSubject(WebBaseEvent raisedEvent)
    {
        StringBuilder subjectBuilder = new StringBuilder();

        // surface some details in subject about error events
        WebBaseErrorEvent errorEvent = raisedEvent as WebBaseErrorEvent;
        if (null != errorEvent)
        {
            Exception unhandledException = errorEvent.ErrorException;

            // drill through reflection exceptions to show the root cause
            TargetInvocationException invocationException =
                unhandledException as TargetInvocationException;
            if (null != invocationException)
            {
                Exception innerException =
                    DrillIntoTargetInvocationException(invocationException);
                subjectBuilder.AppendFormat("{0}",
                    (innerException ?? invocationException).GetType().Name);
                if (null != innerException)
                    subjectBuilder.Append(" (via reflection)");
            }
            else subjectBuilder.Append(unhandledException.GetType().Name);
        }

        // if we've not got anything better
        // just show the event type in the subject
        if (0 == subjectBuilder.Length)
            subjectBuilder.AppendFormat("Event type: {0}",
                raisedEvent.GetType().Name);

        if (!string.IsNullOrEmpty(subjectPrefix)) {
            subjectBuilder.Insert(0, ' ');
            subjectBuilder.Insert(0, subjectPrefix);
        }
        return subjectBuilder.ToString();
    }

    /// <summary>
    /// Reflection often hides exception details, so we try to drill down
    /// through the plumbing exceptions to find a likely cause
    /// </summary>
    private Exception DrillIntoTargetInvocationException(
        TargetInvocationException outerException)
    {
        Exception innerException = outerException.InnerException;
        TargetInvocationException innerInvocationException =
            innerException as TargetInvocationException;
        if (null != innerInvocationException)
            return DrillIntoTargetInvocationException(innerInvocationException);
        else if (null != innerException)
            return innerException;
        else return null;
    }

    private static string GetAndRemoveStringAttribute(NameValueCollection config,
        string attributeName, bool required)
    {
        string value = config.Get(attributeName);
        if (required && string.IsNullOrEmpty(value))
            throw new ConfigurationErrorsException(string.Format(
                "Expected attribute {0}, which is missing or empty.",
                attributeName));
        config.Remove(attributeName);
        return value;
    }

    public override void Flush()
    {
        // nothing to do - this is not a buffering provider
    }

    public override void Shutdown()
    {
        // nothing to do here either
    }
}

Here's a helper class that formats the error messages the way I want to see them. Note that I've omitted some fields that I personally didn't care about, and I've reordered things a bit, so you might want to tweak this if you're going to use it in your own system.

internal static class ErrorEventFormattingHelper
{
    internal static string FormatRequestErrorEvent(
        WebRequestErrorEvent errorEvent)
    {
        CustomEventFormatter formatter = 
            new CustomEventFormatter();

        formatter.AppendLine(string.Format(
            "Unhandled Exception in {0}:",
            WebBaseEvent.ApplicationInformation
            .ApplicationVirtualPath));
        formatter.Indent();
        EmitExceptionAtAGlance(formatter, 
            errorEvent.ErrorException);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Exception stack trace(s):");
        EmitExceptionStackTrace(formatter, 
            errorEvent.ErrorException);

        formatter.AppendLine();
        formatter.AppendLine("Event information:");
        formatter.Indent();
        EmitEventInfo(formatter, errorEvent);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Application information:");
        formatter.Indent();
        EmitApplicationInfo(formatter, 
            WebBaseEvent.ApplicationInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Process/thread information:");
        formatter.Indent();
        EmitProcessInfo(formatter, 
            errorEvent.ProcessInformation);
        formatter.RevertIndent();

        formatter.AppendLine();
        formatter.AppendLine("Request information:");
        formatter.Indent();
        EmitRequestInfo(formatter, 
            errorEvent.RequestInformation);
        formatter.RevertIndent();

        return formatter.ToString();
    }

    private static void EmitEventInfo(
        CustomEventFormatter formatter,
        WebBaseEvent theEvent)
    {
        formatter.AppendLine(string.Format(
            "Event code: {0}",
            theEvent.EventCode.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event message: {0}", 
            theEvent.Message));
        formatter.AppendLine(string.Format(
            "Event time: {0}", 
            theEvent.EventTime.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Event ID: {0}", 
            theEvent.EventID.ToString("N", 
            CultureInfo.InvariantCulture)));
    }

    private static void EmitApplicationInfo(
        CustomEventFormatter formatter, 
        WebApplicationInformation appInfo)
    {
        formatter.AppendLine(string.Format(
            "Application domain: {0}", 
            appInfo.ApplicationDomain));
        formatter.AppendLine(string.Format(
            "Application Virtual Path: {0}", 
            appInfo.ApplicationVirtualPath));
        formatter.AppendLine(string.Format(
            "Application Physical Path: {0}", 
            appInfo.ApplicationPath));
    }

    private static void EmitProcessInfo(
        CustomEventFormatter formatter, 
        WebProcessInformation webProcessInfo)
    {
        formatter.AppendLine(string.Format(
            "Process ID: {0}", 
            webProcessInfo.ProcessID.ToString(
            CultureInfo.InvariantCulture)));
        formatter.AppendLine(string.Format(
            "Process name: {0}", 
            webProcessInfo.ProcessName));
        formatter.AppendLine(string.Format(
            "Account name: {0}", 
            webProcessInfo.AccountName));
    }

    private static void EmitRequestInfo(
        CustomEventFormatter formatter, 
        WebRequestInformation webRequestInfo)
    {
        string name = null;
        if (webRequestInfo.Principal != null)
            name = webRequestInfo.Principal.Identity.Name;

        formatter.AppendLine(string.Format(
            "Request URL: {0}", 
            webRequestInfo.RequestUrl));
        formatter.AppendLine(string.Format(
            "Request path: {0}", 
            webRequestInfo.RequestPath));
        formatter.AppendLine(string.Format(
            "User name: {0}", 
            name ?? "[ANONYMOUS]"));
        formatter.AppendLine(string.Format(
            "User host address: {0}", 
            webRequestInfo.UserHostAddress));
    }

    private static void EmitExceptionAtAGlance(
        CustomEventFormatter formatter, 
        Exception exception)
    {
        formatter.AppendLine(string.Format(
            "Type: {0}", 
            exception.GetType().Name));
        formatter.AppendLine(string.Format(
            "Message: {0}", 
            exception.Message));
        if (null != exception.InnerException)
        {
            formatter.Indent();
            formatter.AppendLine("-->Inner Exception");
            EmitExceptionAtAGlance(formatter, 
                exception.InnerException);
            formatter.RevertIndent();
        }
    }

    private static void EmitExceptionStackTrace(
        CustomEventFormatter formatter, Exception exception)
    {
        formatter.AppendLine(exception.StackTrace);

        if (null != exception.InnerException)
        {
            // no point indenting
            // since stack traces typically wrap like crazy
            formatter.AppendLine();
            formatter.AppendLine("-->Inner exception stack trace:");
            EmitExceptionStackTrace(formatter, exception.InnerException);
        }
    }
}

And finally, here's a helper class that manages indentation levels for the output email message:

public class CustomEventFormatter
{
    const int TabSpaces = 4;

    StringBuilder sb = new StringBuilder();
    private int indentLevel;
    private bool startingNewLine = true;

    public void Indent()
    {
        ++indentLevel;
    }

    public void RevertIndent()
    {
        if (indentLevel > 0)
            --indentLevel;
    }

    public void Append(string text)
    {
        if (startingNewLine)
            EmitIndent();
        sb.Append(text);
        startingNewLine = false;
    }

    public void AppendLine(string lineOfText)
    {
        if (startingNewLine)
            EmitIndent();
        EmitIndent();
        sb.AppendLine(lineOfText);
        startingNewLine = true;
    }

    private void EmitIndent()
    {
        sb.Append(' ', TabSpaces * indentLevel);
    }

    public void AppendLine()
    {
        AppendLine(string.Empty);
    }

    public override string ToString()
    {
        return sb.ToString();
    }
}

Build this into a library application and reference it in your config file. Here's an example:

<healthMonitoring>
  <providers>
    <add name="mailWebEventProvider"
         type="MyMailWebEventProvider"
         to="web-fault@fabrikam.com"
         from="website@fabrikam.com"
         buffer="false"
         subjectPrefix="[WEB-ERROR]"
       />
  </providers>
  <rules>
    <add name="All Errors Email"
         eventName="All Errors"
         provider="mailWebEventProvider"
         profile="Default"
         minInstances="1"
         maxLimit="Infinite"
         minInterval="00:01:00"
         custom=""/>
  </rules>
</healthMonitoring>

Blogger: Pete Lindstrom

On July 8th, Dan Kaminsky of IOActive announced a major DNS “vulnerability” in conjunction with a number of major DNS vendors. The announcement was off the charts in fanfare and attention, but what was the real impact on risk?

First, it is worth noting that this “bug” is more properly classified as a new attack technique invented by Dan. It combines two vulnerabilities that have been well-known for some time – the ability to guess non-random transaction IDs and the use of Additional RRs to insert new entries into the DNS cache. A fix against either of these vulnerabilities also negates the attack itself.

The fundamental question that determines the risk impact revolves around whether it is reasonable to expect fewer or more incidents that use this technique when comparing the period prior to disclosure -- or, more properly, before the date of Dan’s invention of the technique (this also assumes prior art) – with the period after invention/disclosure and into the future. If the disclosure reduces the number of those incidents, then risk is reduced; if the disclosure increases the number of those incidents, then risk is increased.

With that litmus test as our guideline, it is useful to break down the functional elements of risk and look at the impact on threats, vulnerabilities, and consequences (we will cover consequences, then vulnerabilities, and finally threat).

Consequences
Though the consequences are the same before and after disclosure, it is worth discussing the impact here, given that the implication was that the “entire web” could be taken down. The nature of the attack requires the following:

1. An attacker must convince/trick a user into making a DNS request for a domain that doesn’t already exist in their DNS server’s cache. The expectation here is that s/he can be easily tricked into doing this.
2. Then, the attacker must simultaneously attack the DNS server by guessing the transaction ID. According to Kaminsky, the request/attack phase can be done reliably in about 10 seconds.
3. The attack is DNS server-specific. Only users on the same DNS server are affected.
4. Propagation: once the cache is poisoned, anyone requesting that domain will be routed to a malicious server.

Without combining this attack with other attack techniques, there can be three results:

1. Spoofing of a single website for multiple, perhaps many, users using the same DNS server. Presumably, this would be followed by more traditional phishing and malware attacks.
2. Denial-of-service by rerouting traffic from a legitimate site thereby taking potential customers or “eyeballs” away.
3. Denial-of-service be rerouting traffic from a legitimate high volume site to a legitimate low-volume site thereby overloading the servers on the low-volume site.

Because of the point-to-point (user-to-website) nature of the attack, to do something that constitutes “taking over the entire web” is infeasible by a longshot.

The bottom line analysis for the effect on risk due to a change in consequences from pre-invention to post-invention: no change, and therefore no impact.

Vulnerabilities
These vulnerabilities have existed for years, and there have been workarounds for years. Along with this announcement, new patches were introduced in all major DNS server solutions. It is reasonable to assume that many DNS server implementations have been patched, though public accounts have suggested that number is in the 66%-75% range.

Bottom line analysis: the vulnerability level has been reduced, probably significantly, and the affect is positive for risk reduction. If 100% of DNS servers were patched, then overall risk would be reduced for this attack (assuming that there were actual attacks using this technique in the past.)

Threats
The real question regarding risk impact comes in the arena of the less-controllable manipulation of threat. The general threat equation revolves around an attacker’s willingness to attack, based on his/her own cost/benefit analysis that compares the cost to attack to the expected benefits, tempered by the potential for being caught and penalized.

Cost to attack – prior to disclosing the invention, there were likely few, if any attackers with “prior art” that mirrored this technique. It is anybody’s guess how many potential attackers might have figured it out eventually, but they would have had to come from the pool of folks with enough expertise to do so – I am going to guess 500,000 people.

After the disclosure, the hints provided in the press release, the podcast, the sorted stories, and the blog entries made it much easier to figure out. Let’s guess that 5 million people could execute the attack. With automated tools, that number goes up to 50 million.

These numbers are estimates that illustrate the nature of the exercise. You are welcome to fill in your own estimates and come to your own conclusions.

Bottom line analysis: a significant increase in threat and corresponding risk.

Net Effect
The risk manager's challenge is to weigh the decrease in vulnerable systems compared with the corresponding increase in threat, within the context of number of incidents and anticipated future incidents. Given the sheer size differential, it is difficult to conceive of a situation where risk is not increased.

Sometimes it "feels" like someone is taking action for the greater good, when that action actually creates a negative impact for all. For example, it is common for people to believe that raising prices of scarce resources during  times of trouble (e.g. gasoline in the hurricane Katrina aftermath) is unconscionable even though a majority of economists recognize that raising prices actually provides for the greater public good. Vulnerability discovery and disclosure, and attack inventions, might feel like the right thing to do, but the net result is almost always a negative impact.

Faith in the (UK) gov’s ability to securely manage personal data is out the window.

From Reuters:

The inquiries followed Britain’s biggest data loss scandal, when two discs containing child benefit records, including names, addresses and bank details, of some 25 million people, went missing after being put in the post by a junior employee.

The reports concluded that it wasn’t individuals who were to blame - some 30 were officials played some role in events leading to the loss of the discs - but institutional and systematic failures at Britain’s tax authority.

But the HMRC is not alone in such security breaches. A separate report into a stolen laptop containing the details of 600,000 potential recruits revealed similar failings at the Ministry of Defence. In all, four MoD computers had been stolen since 2004 and the report said the MoD was probably in breach of several principles set out in the Data Protection Act.

Well, where do you stand? Do you trust your respective government not to punt on data security?

Read on.

Article Link

Next, insert appropriate morbid jokes <here> for "IDS is dead", "NAC is dead", "GRC is dead", everybody is dead... WTF? Are we at the cemetery or what? Is "dead" dead? Yeah, but it came back as a zombie :-) So, "dead" is a "living dead" "dead" now. Ha*3.

Finally, think! Why were you thinking of buying a SIEM? 'Cause the big "G" in the sky said so? And while you are thinking, check these fun points out:

1. Does your SIEM require 17 beefy servers to operate? How many gallons of foreign oil have to go up in smoke to power that mammoth up? And you know what happened to mammoths, don't you?
2. If your "high-performance" SIEM appliance can only run 5 correlation rules at the same time, what "high" do they mean, really? Hold this thought....
3. Is five field engineers, two developers and CTO enough to install it? Who else needs to help? Ah, sorry, I missed the DBA :-)
4. Do you know when "If CustomVariable17 = Value5" condition matches? Will you still remember it in a year?
5. Can you tell "taxonomy" from "ontology"? You can now? Good for you. Are you more secure now? More efficient? Compliant?
6. How many shifts of security analysts do you have watching the shiny consoles 24/7? If zero, then why - oh - why those consoles are running in the first place? "If a tree falls..." - you know how this one ends. Correct! You get hit by the bough.
7. When was the last time you built a custom agent for parsing and normalizing, say, SAP logs? Did it work? What did you do after it didn't? Cried? And did it help? Then a burly vendor SE showed up, charged you $37,600 and left? Happy now?
8. Do you automatically correlate IDS/IPS alerts with vulnerability data ... for client-side attacks? Really? :-)
9. There are dozens of firewall, IDS/IPS, router, etc brands, each with its own log type. This is actually simple! But there are thousands upon thousands of applications in use today. Some have logs. All are different. Care to build rules for that? Now you finally know why SIEM vendors don't parse their own Java logs (no shit!)
10. Do you know what "threat x vulnerability x random()" equals to? Yup, it still equals random(). Automated prioritization, you say?
11. Do you know why some SIEM vendors are migrating to IT GRC now? So they can go and die there ... quietly.

All in all, I have to agree with Raffy to a large extent!  The world has evolved - and SIEM has not. It might not be dead (as old attacks and defenses never really die and large organization still build and man massive SOCs where SIEM is "a must"), but in this age of web application hacking, CSRF and XSS, phishing, PCI DSS, massive bot armies, client-side 0-days, stealth malware, etc, paying $x,000,000 for a pile of ugly Java code is insane ... As a result, SIEM has greatly diminished in importance and has become just one small thing you might do with logs and some other data. What made it so? Mostly implementation complexity - but a slew of other factors mentioned above as well.

So, consider this instead:

• Compliance? "Sorry, buddy, you need this for compliance, not that. "
• Want to simplify your incident response? Get log management and fly through all your logs, not crawl through some of them.
• Have a very real need to dig into your logs for troubleshooting or tracking that pesky user? Log management works.

Now, what if you have a latent and vague desire to "correlate something" and a million nice greenbacks to flush down the drain? OK, go get your SIEM toy for $780,000 + 20% maintenance/year ... a true bargain (price valid today only).

Finally, I would like to end this on an optimistic note. Do we need more intelligence to analyze the log data we have collected? Of course! Do we have a widest set of log use cases from today's security  to tomorrow's regulations? You bet. And, for you Raffy, I'd add "... we also have other data to analyze together with logs." So, can we "reinvent SIEM?" Yes, I think so! It just hasn't been done yet ... For now, just use log management.

Attackers insert SCRIPT and IFRAME tags into the content of trusted, legitimate web sites via a known SQL injection attack. Those tags redirect the user to the attacker’s server which hosts the Flash exploit. Tens of thousands of web sites are vulnerable to the SQL injection attack, meaning the distribution potential is high.

The vulnerability is not “zero-day”; however, these are the first known public exploits targeting it. The SecureWorks Counter Threat Unit (CTU) has analyzed 18 variants of the exploit, and all attempt to leverage the integer overflow vulnerability originally discovered by Mark Dowd (CVE-2007-0071), which was patched by Adobe with release of version 9.0.124.0 of the Flash Player. While some have reported that the latest version is vulnerable, the CTU was unable to duplicate these results with samples taken from known exploit sites. The only confirmed vulnerable version is (pre-patch) 9.0.115.0.

Insert pithy note about how much fun I’m having and how I enjoy the struggle of reading/collating/loving the links at 0-early-thirty in the frakkin morning. Thanks to all of our new subscribers that joined us yesterday. Welcome! And bunnies. Magic Bunnies!

Click here to subscribe to Liquidmatrix Security Digest!

And now, the news…

1. Man Allegedly takes a penny from the cup belonging to E-Trade and Schwab then gives the money back to Lumberg just before Milton burns the place to the ground.
2. Get Kraken on your botnet You want the original title or the funny title?
3. Haberdashery! Or, how to tell an Aitel fanboi from a mile away
4. HP SPIs SaaS appsec glory hey - if you think you’re so much smarter than me - comment! (not you CJ, you’re scary)
5. Singapore firm claims to own patent on clicking an image to go to a different site does prior art from 1993 count against a patent issued in 2004?
6. And the Gold goes to RFID - Olympic Tickets to contain details on legitimate holder What is the relevance of the Olympics these days anyways?
7. Flash Pants! - Flash 0day vuln pwns you
8. Consumer Alert - you’re keeping too much data in your phone Your drinking phone should look like you’re at a retro party
9. Prepare for Jesus-Net Family-friendly broadband - Nanny-state sez free-Wifi is walled garden

Tags: News, Daily Links, Security Blog, Information Security, Security News