Hash functions are the most commonly used cryptographic primitive, and the most poorly understood. You can think of them as fingerprint functions: They take an arbitrary long data stream and return a fixed length, and effectively unique, string. The security comes from the fact that while it's easy to generate the fingerprint from a file, it's infeasible to go the other way and generate a file given a fingerprint.

Originally created to make digital signatures more efficient, hashes are now used to secure the very fundamentals of our information infrastructure: in password logins, secure web connections, encryption key management, virus and malware scanning, and almost every cryptographic protocol in current use. Without cryptographic hash functions, the internet would simply not work. At the same time, there isn't a good theory of hash functions. Unlike encryption algorithms, there are no secret keys involved; this makes it harder to mathematically define exactly what hash functions are.

The National Institute of Standards and Technology, NIST, is holding a competition to replace the SHA family of hash functions. "SHA" stands for "Secure Hash Algorithm." It was developed by the NSA in 1993 to replace the commercial MD4 and MD5 algorithms, and has been updated several times since then. All the SHA algorithms are very similar, and have been increasingly under attack, so NIST wants to replace them.

The competition is important because, unlike other technological standards, committee design — balancing the interests of diverse constituents — isn't conducive to good security. Security is best when it's designed by expert teams and then subjected to public review. And cryptography is best when it's chosen by competition.

In 1997, NIST held a competition for a block cipher to replace DES. Fifteen candidates and three-and-a-half years later, Rijndael became the new Advanced Encryption Standard — AES. NIST is doing the same thing for what it's calling SHA-3 (not, for some unexplained reason, the Advanced Hash Standard or AHS).

The deadline was October 31, and NIST received 64 submissions. This isn't surprising — I predicted 80 — as most of the 15 AES submitters were professors, whose students at the time have become professors themselves, with their own students. (If NIST does a stream cipher competition in another ten years, they should expect about 256 submissions.) These submissions came from academia, from industry, and from hobbyists. CIO magazine recently interviewed one of the submitters, who is 15. Twenty-eight submissions have been made public by the submitters, and six of those have been broken.

NIST is going through all the submissions right now, making sure they are complete and proper. Their goal is to publish all accepted submissions by the end of November, in advance of the First Hash Function Candidate Conference, to be held in Belgium right after the Fast Software Encryption workshop in February.

The group expects to quickly make a first cut of algorithms — hopefully to about a dozen — and give the community a year of cryptanalysis before making a second cut in 2010. After another year of cryptanalysis, NIST will choose a winner in 2011. Expect a final standard by 2012.

My advice for software developers is to let the process run its course. While it's tempting to use the new cool algorithms in your designs, it's far too soon to trust any of them. This process is likely to result in all sorts of new research results in hash function security, and some real cryptanalytic surprises. Give the community a few years to figure out which ones are good and which aren't.

I've previously called this sort of thing a cryptographic demolition derby: The last one left standing wins. But that's only partially true. Certainly all the groups will spend the next few years trying to cryptanalyze each other, but in the end there will be a bunch of unbroken algorithms. NIST will select one based on performance and features.

NIST has stated that the goal of this process is not to choose the best standard but to choose a good standard. I think that's smart; in this process, the best is the enemy of the good. While there's no rush to choose a new standard — the SHA-2 algorithms will remain secure for the foreseeable future — we don't want to analyze the candidates forever.

Personally, I was part of a group of eight cryptographers that submitted Skein to the competition. A decade ago, writing Twofish and participating in the AES process was the most fun I had ever had in cryptography. These next few years promise to be even more fun.

---

Bruce Schneier is chief security technology officer of BT. His new book is Schneier on Security.

Pouring over endless details of risks, regulations, taxonomies, and technologies can sometimes give us a narrow view of the world, so it seems worthwhile to take a minute to mark the 125th anniversary of the cataclysmic eruption of Krakatoa this week. For those of us that want to think big but can’t remember that far back, this week is also the 3rd anniversary of Hurricane Katrina’s devastating sweep across a wide stretch of the US Gulf Coast.

By now, I expect that most of you have read or are familiar with the 2007 book, The Black Swan, by Nassim Nicholas Taleb, which argues that these kinds of unpredictable, outlying occurrences are the ones that really shape businesses, countries, economies, and people. Taleb argues that although these “Black Swan” events are almost completely unforeseeable, we mistakenly try to explain the circumstances at the time and make predictions about similar events in the future.

In my ERM work with clients, and especially in the context of research I’ve been doing with my colleague Stephanie Balaouras on business continuity and resiliency, questions come up about how to plan for catastrophes... and they’re good questions. Were the CardSystems or TJX data breaches foreseeable? What about the Societe General debacle or the 2004 Indian Ocean tsunami? What’s next? Should these types of events be included in our risk assessments?

We’d like to get your opinion on these and other risks that may be on the very edge of the statistical tail. At what point do they belong in your risk register?

Of course, it’s possible to define mitigating controls for crises, disasters, or incidents without knowing for sure what they’re going to look like. That’s one of the hallmarks of a good crisis management plan. And that’s an important point, because trying to predict the next unforeseeable event can be a real challenge sometimes.

TechCrunchIT reported today that a Rackspace data center went down for several hours during the evening due to a power grid failure. Because Rackspace is a managed service provider (MSP), the downtime affected several businesses hosted in the data center.

When companies think of disaster recovery and downtime, they typically think of catastrophic events such as hurricanes, tornadoes, and earthquakes. What companies don't realize is that the most common cause of downtime is power failures. In a joint study by Forrester Research and The Disaster Recovery Journal of 250 disaster recovery decision-makers and influencers, 42% of respondents indicated that a power failure was the cause of their most significant disaster declaration or major business disruption.

To prevent power failures, businesses must ensure that they have multiple diverse connections to the power grid as well as install backup power generators and uninterruptible power supplies (UPS) at the data center. But it's not enough to have these preventative measures in place, businesses must test the ability to switch over to backup power must at least twice year. And if your business has a recovery data center, it's best if the recovery data center is on a different power grid and is also equipped with backup power generation.

But despite all these measures, failures might still happen, in the case of the Rackspace power failure, the company successfully failed over to its backup power generators but some of its chillers did not start up correctly.

In North America, the risk of power failures is likely to remain high for the foreseeable future. According to a 2007 report by the North American Electric Reliability Corporation (NERC), long-term capacity margins are still inadequate and significant investment in transmission is still required.

So businesses must not only invest in preventative measures such as backup power generators, they must think about where they locate their data centers. You must avoid areas that have clearly identified congestion issues and focus on areas that have access to cheap and abundant power. And, don't take it for granted that your service provider has effectively managed the risk of power failures.

Fred Wilson has an interesting blog up regarding the new Google Health service. Fred filled out his personal medical information and was disappointed that he was not able to publish this data and make it public.  Fred would like to have a sidebar widget for his blog with his health profile.  Many people wrote to Fred telling him why Google does not do this.  Many of them centered on the fact that insurance companies would use this information against you to deny or limit your coverage.  Some took shots at Fred's socio-economic status saying that he didn't care if the insurance companies used it against him because he could afford to pay whatever he had to.  Fred replies that he thinks withholding or being less than open about health issues to insurance companies, investors, etc. is problematic and in a perfect world insurance companies should not be able to use this against us. In fact Fred says:

Wouldn't we all be better off with an insurance system that wasn't able to discriminate between people based on pre-existing conditions? Wouldn't we be better off if we came together to insure everyone? Wouldn't we be better off if we knew everyone's medical conditions and what treatments worked and what did not? Wouldn't we be better off if we could search for others with the same conditions to share our experiences?

I don't believe Fred feels this way because of his socio-economic status. I think Fred thinks like this because he is I assume in good health.  I wonder if Fred were suffering from some medical condition, if his views on this would change.  This reminds me of the "nothing to hide" argument that some use to justify the government trampling on our privacy rights.  If you have nothing to hide, what do you care.  I care because it is wrong.  I care about not making health records public because it is wrong.  We don't live in a perfect world.  Even taking Hillary or Obama's health plans into account, we live in a world where insurance companies can discriminate against those with pre-existing conditions for the foreseeable future.  Think about if only healthy people published their records, what would that say about people who did not publish their records?

Fred's point about searching for others with the same condition is fine, if they wanted to be found. It is inherently a persons right not to be found.  In fact today if you want to share with a person who shares a medical condition with you, you can search and usually find a group and on line community of people. What is nice is some of these people can share in these groups without revealing their identity.  It is this ability to remain anonymous that I think make these types of communities successful.

Fred recognizes that not everyone would want to share their records. I say once we start dividing society by those who do and don't we really already have imposed a penalty on those who cherish their privacy.

Today I noticed that SL Corporation has revamped their website with a new page, Solutions for CEP Engine Users.    The page is well written, reinforcing some of my earlier posts on the value proposition for CEP; so I hope the folks at SL don’t mind if I repost their excellent thoughts on BAM and CEP here. 

Solutions for CEP Engine Users by SL Corporation

© 1999-2008 Sherrill-Lubinski Corporation. All rights reserved.

Complex Event Processing (CEP) is a relatively new technology that is used to help companies detect both opportunities and threats in real-time with minimal coding and reusable key performance indicators (KPIs) and business models. Just as services are shared and reused in a SOA, CEP permits the sharing and reuse of KPIs in business activity monitoring while efficiently processing events so businesses can act on situations that impact business and take advantage of real-time processing.

Business activity monitoring, often referred to as BAM, is the capability that Gartner and other distinguished analysts use to describe this visualization capability in the business world. BAM introduces a human element to CEP. It is well-established that the human mind is, today and for the foreseeable future, far superior to machine intelligence in making sense out of complicated situations and events. Therefore, BAM is critical to the success of any complex event processing (CEP) solution.

Depending on an organization’s mission, BAM can be used in various levels within an event processing solution to help users visualize and understand the dynamics behind rapidly changing situations and critical business events. In other words, BAM plays a key role wherever there is a need for better insight into the myriad events that effect your business operations.

BAM provides real-time visualization and alerting capabilities for users to better understand how business events impact their organization. BAM software permits users to quickly prototype, build and deploy event processing business solutions. For example, a telecommunications company would find BAM useful to achieve event-driven SLA monitoring and management; and a large retailer would find BAM important as they stay on top of business-critical events in their supply chain.

Insight gained from BAM, in concert with event processing solutions, enable organizations to make better and faster business decisions so they can rapidly sense and respond to threats, problems and opportunities. BAM solutions permit applications to be designed, deployed and modified rapidly with minimal or no coding resulting in significantly lower development costs. Therefore, a key benefit of BAM in real-time event processing solutions is that KPIs can be deployed, monitored, revised, reused and utilized, economically and rapidly.

Depending on the business application, BAM-enabled visualization is required at numerous levels in an event processing architecture. For example, events from across the enterprise are typically processed by a CEP software platforms from companies such as TIBCO, BEA (soon to be Oracle), Progress Apama, StreamBase, Aleri, and Coral8.

Long before KPIs are displayed to the business users, BAM tools can be configured to assist application developers to monitor and visualize the raw event stream. For the developer, their business is developing applications, and BAM can be very useful when designing KPIs for event processing applications.

Fine-tuned KPIs that have been derived from an event processing application are displayed to the business user. These KPIs can indicate risks, threats, problems, opportunities and other emerging business situations that impact the business.

BAM, in concert with state-of-the-art event processing software, provides the framework for a complete sense-and-respond capability for businesses. Processing raw events and event streams for business opportunities and threats requires robust and rapidly deployable visualization solutions. This is the reason that many distinguished analysts believe that BAM and CEP are complementary and critically interdependent core business capabilities. We at SL Corporation agree, and are pleased to be the leading BAM visualization platform in the event processing/CEP ecosystem today.

© 1999-2008 Sherrill-Lubinski Corporation. All rights reserved.