• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, let me bet my ass that "Not knowing what to log" problem will be licked in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what  the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, for esoteric log sources (and custom applications) might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?"  for reference)

Next, the problem of "Log volume" will  definitely get worse, much worse.  One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is NOT a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL (!!!) get since standards are being developed. We will see people struggling with all sorts bizarro log data in the coming years. Virtualization, web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into dark voodooistic experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. BTW, watch my blog for some uber-fun info on that!

"Getting the logs"  has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing  files remotely via secure protocols made application log collection easier (syslog-NG with TCP transfer and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services (thanks Eric!). However, in the future it might resurface as we try to collect logs from "weird" places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them (Yo, my t-shirt-making colleagues... :-))

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in  an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and cursing under ones's breath. Only then we can attempt to reliable handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so."  For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. My next poll will address that :-)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?

First, I am proud to say that I was invited to Microsoft lunch for security bloggers. We definitely had some fun discussion on both blogging and security there and all sensed definite interest from the MS side. At times, though, it felt like talking to "spooks": information goes in, happy smile comes out :-) Other bloggers commented on that too.

Second, CEE logging standard work lives on. We had an informal meeting with people from MITRE, Microsoft, OpenGroup (home of OpenXDAS), Burton as well as relevant vendors and others involved (some people were sadly MIA though...). We did discuss - not too violently! - what to do next and what approach to take in regards to taxonomy (the most hotly debated part of CEE).

Finally, last by time but clearly FIRST by importance, 2nd Annual (?) RSA Security Bloggers Meet-up. OMG, this WAS an event of the century (at least until the next one, that is :-)) Not much more to say, I am still recovering :-) But if you are blogging on security - BE THERE next year!

BTW,  what is the overwhelming RSA theme this year? I think Rich is right, it is what he says.

Blogger: Dan Blum

There are two groups actively working to create a common event standard that allows event logs and audit records to be shared and understood across many products, and the good news is that they’re talking to each other:

• Common Event Expression (CEE) language, by Mitre 
• X/Open Distributed Audit Standard (XDAS), by Open Group 

The business benefits of creating a common event standard would be considerable:

• Reduced log management and security information event management (SIEM) system integration costs 
  • Reduced volume of event data and simplification of SIEM architecture 
  • Reduced need for (and increased effectiveness of) normalization 
• Reduced cost of integrating new solutions with security management infrastructures and frameworks 
• Lower cost of integrating event management and audit into cross-enterprise applications (such as federated identity management)
• Faster and simpler data exchange between organizations, vendors and incident response services supporting real time response to threats and attacks 
• Better forensics for a common defense 

Late last year, our Burton Group Security and Risk Management Strategies (SRMS) group decided to push the question of event standards with vendors, trade press, and standards groups. But we felt that we needed evidence of end user enterprise interest and involvement to start doing so. Happily, as we began researching the space, we found that Mitre’s CEE was being driven by the EU, NATO and DoD as well as log management and platform vendors. Burton Group held a conference call discussing common event standards and SIEM with members of the International Information Integrity Institute (I-4), and key stakeholders showed up. The Open Group reports that enterprises as well as vendors are getting involved with XDAS. Clearly, enterprises seem ready to focus on this topic.

Of course, there are challenges ahead. Not only is there no complete common event standard out in the field today, there are many partial standards or solutions, including Syslog; the IETF’s Intrusion Detection Message Exchange Format (IDMEF) and Incident Object Description and Exchange Format (IODEF); the Java Specification Request (JSR) 47 Logging API, WS-Management subscribe/publish APIs and so on. Any comprehensive standard released in the future should work with existing technologies like these as much as possible. Also, there are a number of complexities, including mapping event semantics between different systems, synchronizing time while managing clock drift, and maintaining dynamic event handling policies.

Fortunately, the Mitre and Open Group efforts are gaining traction. Mitre has put up a CEE web site and one can ask to subscribe to the CEE mailing list. Mitre has described its scope as covering standard event taxonomy/terminology, log syntax, log transport and recommendations on what types of events and data elements systems should log. Mitre’s specifications are in the draft stage, and publication for comment is “expected 2008” according to the website. That’s pretty indefinite. But we are told that while not complete, these draft documents will reflect a considerable amount for work that has already been done and can be built upon. It is positive that a CEE community representative says Mitre plans to begin by seeking comments on the underlying goals and requirements for event standards. But to establish a broadly accepted industry standard anytime soon, Mitre and the government/defense community it servers will have to accelerate overly lengthy document review cycles and possibly streamline handling procedures designed for classified information rather than open standards deliberation.

As my colleague Bob Blakley wrote in “An Auditing Standard: Has this rough beast's hour come round at last?” last July, Open Group revived prior work on a specification called “X/Open Distributed Audit Standard” (XDAS).  XDAS addresses the concerns necessary to build a robust distributed security auditing system in a mature and complete way, but its 1990s era C and UNIX interfaces need to be updated. Novell, whose Bandit Project incorporates XDAS, has contributed source code to a new open-source project called OpenXDAS (http://openxdas.sourceforge.net/) which makes an XDAS implementation widely available.

As these two standards efforts proceed, we hear mixed signals. There have been some indications of contention; for example, CEE representatives purport to have a strong emphasis on “simplicity,” while some observers have expressed concern that XDAS may be “too complex.” Of course, the other side of the argument could be that CEE will over-simplify issues, but it’s hard to have that discussion when specifications for CEE aren’t publicly available yet.

Fortunately, olive branches have been extended as well. During the Open Group meetings in January, 2008 Burton Group observed the XDAS and CEE leadership discuss ways they could coordinate and avoid overlaps. For example, CEE and XDAS could make sure that XDAS APIs become a CEE-compatible logging transport and, if both organizations produce data dictionaries for events, they could be perhaps formulated to use a common taxonomy and to avoid schema conflicts and overlaps. We’re also hoping that vendors such as Arcsight, Oracle and CA – who have been proactive about proposing specifications or encouraging the industry to create a common event standard – will be become part of the convergence on a common solution.

In the coming weeks and months, Burton Group will keep watching the event standards space and post more information on how matters develop. Please let us know by commenting on this blog if there are other standards efforts we should be watching, compatibility concerns to address, or other issues and questions you’re concerned about. We hope to continue being a voice for convergence and standardization that helps put the industry on the road to a common event standard by 2009.

Blogger: Dan Blum

There are two groups actively working to create a common event standard that allows event logs and audit records to be shared and understood across many products, and the good news is that they???re talking to each other:

• Common Event Expression (CEE) language, by Mitre 
• X/Open Distributed Audit Standard (XDAS), by Open Group 

The business benefits of creating a common event standard would be considerable:

• Reduced log management and security information event management (SIEM) system integration costs 
  • Reduced volume of event data and simplification of SIEM architecture 
  • Reduced need for (and increased effectiveness of) normalization 
• Reduced cost of integrating new solutions with security management infrastructures and frameworks 
• Lower cost of integrating event management and audit into cross-enterprise applications (such as federated identity management)
• Faster and simpler data exchange between organizations, vendors and incident response services supporting real time response to threats and attacks 
• Better forensics for a common defense 

Late last year, our Burton Group Security and Risk Management Strategies (SRMS) group decided to push the question of event standards with vendors, trade press, and standards groups. But we felt that we needed evidence of end user enterprise interest and involvement to start doing so. Happily, as we began researching the space, we found that Mitre???s CEE was being driven by the EU, NATO and DoD as well as log management and platform vendors. Burton Group held a conference call discussing common event standards and SIEM with members of the International Information Integrity Institute (I-4), and key stakeholders showed up. The Open Group reports that enterprises as well as vendors are getting involved with XDAS. Clearly, enterprises seem ready to focus on this topic.

Of course, there are challenges ahead. Not only is there no complete common event standard out in the field today, there are many partial standards or solutions, including Syslog; the IETF???s Intrusion Detection Message Exchange Format (IDMEF) and Incident Object Description and Exchange Format (IODEF); the Java Specification Request (JSR) 47 Logging API, WS-Management subscribe/publish APIs and so on. Any comprehensive standard released in the future should work with existing technologies like these as much as possible. Also, there are a number of complexities, including mapping event semantics between different systems, synchronizing time while managing clock drift, and maintaining dynamic event handling policies.

Fortunately, the Mitre and Open Group efforts are gaining traction. Mitre has put up a CEE web site and one can ask to subscribe to the CEE mailing list. Mitre has described its scope as covering standard event taxonomy/terminology, log syntax, log transport and recommendations on what types of events and data elements systems should log. Mitre???s specifications are in the draft stage, and publication for comment is ???expected 2008??? according to the website. That???s pretty indefinite. But we are told that while not complete, these draft documents will reflect a considerable amount for work that has already been done and can be built upon. It is positive that a CEE community representative says Mitre plans to begin by seeking comments on the underlying goals and requirements for event standards. But to establish a broadly accepted industry standard anytime soon, Mitre and the government/defense community it servers will have to accelerate overly lengthy document review cycles and possibly streamline handling procedures designed for classified information rather than open standards deliberation.

As my colleague Bob Blakley wrote in ???An Auditing Standard: Has this rough beast's hour come round at last???? last July, Open Group revived prior work on a specification called ???X/Open Distributed Audit Standard??? (XDAS).  XDAS addresses the concerns necessary to build a robust distributed security auditing system in a mature and complete way, but its 1990s era C and UNIX interfaces need to be updated. Novell, whose Bandit Project incorporates XDAS, has contributed source code to a new open-source project called OpenXDAS (http://openxdas.sourceforge.net/) which makes an XDAS implementation widely available.

As these two standards efforts proceed, we hear mixed signals. There have been some indications of contention; for example, CEE representatives purport to have a strong emphasis on ???simplicity,??? while some observers have expressed concern that XDAS may be ???too complex.??? Of course, the other side of the argument could be that CEE will over-simplify issues, but it???s hard to have that discussion when specifications for CEE aren???t publicly available yet.

Fortunately, olive branches have been extended as well. During the Open Group meetings in January, 2008 Burton Group observed the XDAS and CEE leadership discuss ways they could coordinate and avoid overlaps. For example, CEE and XDAS could make sure that XDAS APIs become a CEE-compatible logging transport and, if both organizations produce data dictionaries for events, they could be perhaps formulated to use a common taxonomy and to avoid schema conflicts and overlaps. We???re also hoping that vendors such as Arcsight, Oracle and CA ??? who have been proactive about proposing specifications or encouraging the industry to create a common event standard ??? will be become part of the convergence on a common solution.

In the coming weeks and months, Burton Group will keep watching the event standards space and post more information on how matters develop. Please let us know by commenting on this blog if there are other standards efforts we should be watching, compatibility concerns to address, or other issues and questions you???re concerned about. We hope to continue being a voice for convergence and standardization that helps put the industry on the road to a common event standard by 2009.