The Gartner Data Center Conference kicked off this morning in Las Vegas. Despite the completely packed plane coming out here, Vegas seems quieter and not so crowded. The bartender at Wolfgang Puck’s Bistro told me they were looking forward to the 1800 people coming to this show to fill the hotel up. As we’ve noted, the economic crisis is impacting business travel all around.

22% of the attendees at Data Center come from the public sector and government, with 44% coming from very large enterprises of 20K+ employees.

During the Gartner IOM conference in June, some of the most interesting info coming out of it was the quick polls of the audience on a variety of infrastructure and operations management topics. What are enterprises doing? Where are they headed? What’s important to them? Here are some quick takes from the opening session:

1) What is the largest data center challenge that you currently face?

• Smaller Budgets: 21%
• Power & Cooling: 20%
• Dealing with the Rate of Technology Change: 15%
• Aligning Activities with the Business: 15%
• Modernizing Legacy Applications: 10%
• Lack of Data Center Space because of Equipment Spread: 9%
• How to Source IT Services: 5%
• How to Find and Retain Talent: 5%

Well, it’s taken almost a year to be “official”, but the National Bureau of Economic Research just announced that the US has been in a recession since December of 2007. It should come as a surprise to no one that dealing with smaller budgets is top of mind, even for the predominantly larger enterprises attending here.

2) What projects will receive the most funding in 2009?

• Virtualization/Consolidation: 31%
• Data Center Facilities – new builds: 17%
• IT Operations Process Improvement: 12%
• IT Modernization: 7%
• Green IT: 5%

Virtualization and (server) consolidation projects are clearly a priority for larger enterprises in 2009. What’s interesting here is the relatively very low priority of Green IT projects – in spite of the importance to attendees of getting power and cooling costs under control. Perhaps there’s a gap here between what’s often the hype of Green IT and practical considerations for data center managers when it comes to power and cooling management.

3) Where are you with server consolidation projects?

• No Plans: 3%
• Looking at it now and will start in next 2 years: 13%
• In process now: 58%
• Have already completed server consolidation project: 26%

Larger enterprises are consolidating servers with a quarter of attendees already having gone through the process at least once. And according to poll #2, this trend will definitely continue.

• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, let me bet my ass that "Not knowing what to log" problem will be licked in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what  the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, for esoteric log sources (and custom applications) might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?"  for reference)

Next, the problem of "Log volume" will  definitely get worse, much worse.  One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is NOT a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL (!!!) get since standards are being developed. We will see people struggling with all sorts bizarro log data in the coming years. Virtualization, web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into dark voodooistic experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. BTW, watch my blog for some uber-fun info on that!

"Getting the logs"  has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing  files remotely via secure protocols made application log collection easier (syslog-NG with TCP transfer and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services (thanks Eric!). However, in the future it might resurface as we try to collect logs from "weird" places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them (Yo, my t-shirt-making colleagues... :-))

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in  an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and cursing under ones's breath. Only then we can attempt to reliable handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so."  For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. My next poll will address that :-)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?

• Implement a framework for risk assessment and mapping.
• Establish the responsibilities of risk managers with their areas of responsibility.
• Identify and define the risks to which the business is exposed and what constitutes a risk event or "near miss" so that incidents can be mapped to specific risks.
• Determine the threat level, and focus on those risks with the highest impact on performance.
• Establish levels of controls for processes commensurate with the perceived threat.
• Record and retain risk incident and near-miss information.
• Conduct periodic risk assessments to determine changes in the operations risk profile and assess control performance.

Blogger: Randall Gamby

As briefly mentioned in a Burton Group IdPS blog and a ZDNet Australia published article on July 3, 2008, HR data from Google was stolen from one of their previous HR outsource partners.  It seems that the partner, Colt Express Outsource Partners, had equipment stolen that contained HR data from some of its clients, including Google.  The data was unencrypted and stored on systems that were apparently portable.

So what does this mean for all of us? 

First, it shows that even large SaaS companies like Google can be bitten by a lack of security at their partners, just like many of us can.  Burton Group has been warning clients for a long time about the dangers of sending confidential information to outsource partners without proper security and audit processes in place. Of course this should also be backed by strong contractual language. 

Second, be prepared to pay.  Even if Google had breach mitigation terms in their contract, Colt Express announced that it was in financial difficulty. So Google has had to pay for financial reporting and other compensation to its own employees, even though Google did nothing wrong. 

Third, a Google representative stated "We take the security of our employees very seriously and require outside vendors to meet appropriate security standards. We review and update these standards on an on-going basis.”  Does this mean that Google doesn’t require encryption of its confidential information since encryption of the data was not deployed at Colt Express?  When working with third parties, whether it’s financial data or confidential personal data, this information needs to be protected from unauthorized access. One of the simplest ways is encrypting the data while at rest, regardless of where it’s located. 

Final, the Colt Express breach brings to mind a question Burton Group is always asking: “What is your exit strategy if the contract is terminated with your outsourcing partner?”  A lot of effort is expended in creating an outsourcing agreement around use and protection of data, but what happens when the contract is ended?  Do you obtain and retain the information the outsource partner maintained?  Do you have the outsource partner destroy the information and any archives of it (and verify this was done)?  Do you create a custodial contract with the outsourcing partner for them to maintain the information and archives on your behalf (ensuring the data is properly protected)?  As was found in this incident, after their contract with Google was terminated the outsourcing partner apparently retained the employee data unencrypted on their servers. This was the fatal mistake that allowed the breach to occur.

So as you work with your outsourcing and SaaS vendors, you should not only consider how day-to-day operations should be secured to maintain the confidentiality of your data. You should also think about how that data is being maintained over time, and what are your procedures should the unthinkable happen if your partner allows your data to be compromised.

Blogger: Randall Gamby

As briefly mentioned in a Burton Group IdPS blog and a ZDNet Australia published article on July 3, 2008, HR data from Google was stolen from one of their previous HR outsource partners.  It seems that the partner, Colt Express Outsource Partners, had equipment stolen that contained HR data from some of its clients, including Google.  The data was unencrypted and stored on systems that were apparently portable.

So what does this mean for all of us? 

First, it shows that even large SaaS companies like Google can be bitten by a lack of security at their partners, just like many of us can.  Burton Group has been warning clients for a long time about the dangers of sending confidential information to outsource partners without proper security and audit processes in place. Of course this should also be backed by strong contractual language. 

Second, be prepared to pay.  Even if Google had breach mitigation terms in their contract, Colt Express announced that it was in financial difficulty. So Google has had to pay for financial reporting and other compensation to its own employees, even though Google did nothing wrong. 

Third, a Google representative stated "We take the security of our employees very seriously and require outside vendors to meet appropriate security standards. We review and update these standards on an on-going basis.???  Does this mean that Google doesn???t require encryption of its confidential information since encryption of the data was not deployed at Colt Express?  When working with third parties, whether it???s financial data or confidential personal data, this information needs to be protected from unauthorized access. One of the simplest ways is encrypting the data while at rest, regardless of where it???s located. 

Final, the Colt Express breach brings to mind a question Burton Group is always asking: ???What is your exit strategy if the contract is terminated with your outsourcing partner????  A lot of effort is expended in creating an outsourcing agreement around use and protection of data, but what happens when the contract is ended?  Do you obtain and retain the information the outsource partner maintained?  Do you have the outsource partner destroy the information and any archives of it (and verify this was done)?  Do you create a custodial contract with the outsourcing partner for them to maintain the information and archives on your behalf (ensuring the data is properly protected)?  As was found in this incident, after their contract with Google was terminated the outsourcing partner apparently retained the employee data unencrypted on their servers. This was the fatal mistake that allowed the breach to occur.

So as you work with your outsourcing and SaaS vendors, you should not only consider how day-to-day operations should be secured to maintain the confidentiality of your data. You should also think about how that data is being maintained over time, and what are your procedures should the unthinkable happen if your partner allows your data to be compromised.

(ISC)2 is taking a hard stand on CPE submissions and coming down heavy-handed on audits. If you attend an event, even if the event host is an (ISC)2 CPE-Submitter, you need to retain your ‘proof of attendance’ and keep those documents on file. Generally an event host will provide a CPE audit retention sheet at the time of the event, or post-event.

I recently received an audit for my attendance at the RSA Conference in February. While they don’t specify a specific date or time frame you have to respond, I would recommend responding to an audit within 30 days.

Here are some quick tips…

• Provide your CISSP number to event hosts that are CPE-Submitters
• Keep and file any documentation, programs and badges from the event
• If provided, keep the ‘Proof of Attendance’ sheet from the event host
• Keep supporting docs for 12 months past the 3-year certification cycle you claimed the CPEs in

For more information, visit the official (ISC)2 site.

# # #