Hash functions are the most commonly used cryptographic primitive, and the most poorly understood. You can think of them as fingerprint functions: They take an arbitrary long data stream and return a fixed length, and effectively unique, string. The security comes from the fact that while it's easy to generate the fingerprint from a file, it's infeasible to go the other way and generate a file given a fingerprint.

Originally created to make digital signatures more efficient, hashes are now used to secure the very fundamentals of our information infrastructure: in password logins, secure web connections, encryption key management, virus and malware scanning, and almost every cryptographic protocol in current use. Without cryptographic hash functions, the internet would simply not work. At the same time, there isn't a good theory of hash functions. Unlike encryption algorithms, there are no secret keys involved; this makes it harder to mathematically define exactly what hash functions are.

The National Institute of Standards and Technology, NIST, is holding a competition to replace the SHA family of hash functions. "SHA" stands for "Secure Hash Algorithm." It was developed by the NSA in 1993 to replace the commercial MD4 and MD5 algorithms, and has been updated several times since then. All the SHA algorithms are very similar, and have been increasingly under attack, so NIST wants to replace them.

The competition is important because, unlike other technological standards, committee design — balancing the interests of diverse constituents — isn't conducive to good security. Security is best when it's designed by expert teams and then subjected to public review. And cryptography is best when it's chosen by competition.

In 1997, NIST held a competition for a block cipher to replace DES. Fifteen candidates and three-and-a-half years later, Rijndael became the new Advanced Encryption Standard — AES. NIST is doing the same thing for what it's calling SHA-3 (not, for some unexplained reason, the Advanced Hash Standard or AHS).

The deadline was October 31, and NIST received 64 submissions. This isn't surprising — I predicted 80 — as most of the 15 AES submitters were professors, whose students at the time have become professors themselves, with their own students. (If NIST does a stream cipher competition in another ten years, they should expect about 256 submissions.) These submissions came from academia, from industry, and from hobbyists. CIO magazine recently interviewed one of the submitters, who is 15. Twenty-eight submissions have been made public by the submitters, and six of those have been broken.

NIST is going through all the submissions right now, making sure they are complete and proper. Their goal is to publish all accepted submissions by the end of November, in advance of the First Hash Function Candidate Conference, to be held in Belgium right after the Fast Software Encryption workshop in February.

The group expects to quickly make a first cut of algorithms — hopefully to about a dozen — and give the community a year of cryptanalysis before making a second cut in 2010. After another year of cryptanalysis, NIST will choose a winner in 2011. Expect a final standard by 2012.

My advice for software developers is to let the process run its course. While it's tempting to use the new cool algorithms in your designs, it's far too soon to trust any of them. This process is likely to result in all sorts of new research results in hash function security, and some real cryptanalytic surprises. Give the community a few years to figure out which ones are good and which aren't.

I've previously called this sort of thing a cryptographic demolition derby: The last one left standing wins. But that's only partially true. Certainly all the groups will spend the next few years trying to cryptanalyze each other, but in the end there will be a bunch of unbroken algorithms. NIST will select one based on performance and features.

NIST has stated that the goal of this process is not to choose the best standard but to choose a good standard. I think that's smart; in this process, the best is the enemy of the good. While there's no rush to choose a new standard — the SHA-2 algorithms will remain secure for the foreseeable future — we don't want to analyze the candidates forever.

Personally, I was part of a group of eight cryptographers that submitted Skein to the competition. A decade ago, writing Twofish and participating in the AES process was the most fun I had ever had in cryptography. These next few years promise to be even more fun.

---

Bruce Schneier is chief security technology officer of BT. His new book is Schneier on Security.

The panelists were:
William P. Crowell – former Deputy Director of the National Security Agency
Michael P. Jackson – Deputy Secretary, Department of Homeland Security
Jose A. Rodriguez, Jr. – former Director CIA, National Clandestine Service & CIA, DCI Counterterrorist Center

Overall, it was a very nicely arranged event on a brisk fall evening with about 100 CXO attendees; mostly large but some small government contractors and a few product companies like ScienceLogic that conduct business with military, intelligence and the public sector.

No surprise, given the financial crisis the economy is suffering from that the panelists said we also have a crisis coming on the Federal budget front. This will put enormous pressure on the way Administration thinks, and how and where to spend the $$.

Obama’s tone regarding the issues he will be confronting in the world during the election was encouraging. Make the world more non-partisan and take on the threats that we have in front of us head-on!

The panel was very upfront about current threats. William Crowell said,

“It is highly imprudent to believe that there will not be another 9-11. We have to fund and support the work to stop other attacks. We can only mitigate risk but we can’t eliminate risk. We have to try to absorb the sense of urgency and wake up every day looking at the intelligence screens as if 9-11 happened within the last couple of months.”

He added,

“They (the intelligence community) need the innovation, sense of commitment and urgency that comes from the private sector – a sense of mutual commitment to that mission.”

Predicted Priorities for investment for DHS:

1. Cyber attack as the top issue
2. Nuclear threats including dirty bomb
3. Chemical and biological attacks
4. Explosive attacks against critical infrastructure with maximum # of lives and or financial disruption / loss.
5. Large scale natural disasters – hurricane + earthquakes
6. Border penetration - identity management and border management issues

An Obama administration will spend dollars around these threat vectors. They will want to spend $$ to help state and local governments. Grants to state and local governments should significantly increase with the Obama administration, so think about how you will increase your focus on the state and local government spending initiatives.

Secure border investments – the panelists believe that the new administration will feel compelled to invest here. Michael P. Jackson bluntly said, “You have to make investments in border tools to get meaningful immigration reform.”

Panelists agreed that the 1^st year will be an intense period of scrutiny about fundamental directions. We can’t afford it all at DHS; it is dramatically under budgeted. At TSA/DOT and then at DHS, we spent about $4 Billion on technology investments since 9-11; those investments are now reaching the end of the original service life.

One gripe from the panel that I found humorous: “We don’t have a group of people who think like entrepreneurs.” It is insane how long things last when you buy things in the government. As an example, we are still replacing vacuum tubes in some of the very old FAA gear… this is well beyond what any reasonable person would think these initial investments should/would last.

Final Thoughts:
I actually think that the Obama Administration will be quite favorable to COTS software products, SaaS offerings, and creative financing initiatives from the private sector. The government just won’t have the capital budget to do everything it wants to accomplish. I would say if you look at how intelligently and aggressively Obama used technology to assist his campaign, the odds are good that this new breed of IT talent (which is already really comfortable with SaaS products, blogs, wiki’s, hosted/outsourced Cloud solutions… this team really understands the latest technology trends) will quickly work to bring these new IT paradigms to the Federal marketplace. Clearly the private sector can help the Government achieve more with lower capital budgets – beginning to provide services rather than transaction-based selling. Another clear idea is to think about leasing as a better way to work with the government which going forward will have increased budgets restrictions.

They will likely be in confrontation with members of Congress that won’t change fast enough, however the future of our nation’s ability to fight terror lies in becoming more efficient and effective. It requires the government be flexible enough to figure out what jobs and IT functions to outsource in a nimble and smart way. My prediction: this is great news for Service Providers. Overall the next 4 years should be great for our business as well as the Managed Service Provider/SaaS industry!

Of course, virtualization can do its bit to help companies save on hardware, power and cooling costs. We have this straight from a customer’s mouth – see the video here. And having your data center be 100% wind-powered certainly does its bit as well.

But the winner – according to eWeek and and an EMA study in September – is to turn equipment off when they’re not in use. I would say this is a no-brainer, but apparently we all need to be reminded…

Who remembers that line from the Schoolhouse Rock video on Energy? “So don’t get cross, when momma says turn that extra light off.” The full video for your viewing pleasure here.

(image from freedomroadproject)

I’m just not sure how a customer is ever expected to become an expert in the product(s) they’ve purchased, and hope to successfully implement, when the manufacturer requires nearly 60 experts on a call to answer questions.  I understand that a customer generally won’t use all parts of the Tivoli suite of products, but how do you even figure out what to buy?

And let’s not forget…12 hours! That is one long support call.

Partial Disclosure - The Good

The responsible disclosure process tends to break down in rare occasions where the vendor doesn’t want to fix the issue. When this occurs, the researcher is put into a difficult position whereby full disclosure could put users’ systems at high risk of compromise. The other case where partial disclosure becomes an alternative is when the researcher has discovered a design flaw in a protocol or underlying multiple vendor component. Examples of this case include the DNS flaws published this past summer by Dan Kaminsky and the TCP denial of service condition discovered by Robert E. Lee and Jack Louis that is currently in the disclosure process. When the flaw affects a very large number of vendors and the actual problem is located within the underlying protocols that support the communications of the Internet as a whole, one possible solution is to follow a partial disclosure model where phasing the details to the general public can be used to encourage adoption and creation of patches throughout the enormous target audience.

Partial Disclosure - The Bad

What is driving the fear surrounding partial disclosure is the potential for abuse. When a major flaw is partially disclosed, a number of potential issues may occur. First and foremost, the further along the partial disclosure path we are, the more details will be released to the public, and the higher the probability that someone (either good or bad intentioned) will figure out the exploit and disclose the details. Second, when partially disclosing, the vendor’s hand is being forced into a situation that could speed up fixes, reduce testing, and cause ripple problems elsewhere within the infrastructure. It is difficult enough to dance the fine time line when doing responsible disclosure, but if we are escalated to the point of partial disclosure, additional fuel is added to the fire.

The Ugly

The real ugly part of partial disclosure is when we add to the equation the ability to spread fear, uncertainty, and doubt into the normal user community. It is generally well accepted that FUD can be used to drive additional revenue. If it is possible to increase the perceived magnitude of the “problem” that your product or service solves, it is possible to directly impact the demand for that product or service. That is the major fear imposed by the growing trend of partial disclosure. By releasing just enough information to trigger wide scale speculation into the flaw, it is possible to create buzz and garner media attention resulting in a lot of speculation and very little hard facts around the issue. The potential for abuse by the security industry at large is enormous.

The Fix

Some have suggested a group of security researchers be convened to vet the requirement of partial disclosure and to allow for independent peer review of any security research that requires the partial disclosure process. This suggestion leaves questions regarding who would stand on this group and who would be impartial enough to ensure that the right thing was always done regardless of profit potential. It also leaves open the opportunity for member researchers to utilize the information gathered during the vetting process to position themselves to profit from the data upon release. It might be wiser to rely on a higher level authority or government entity to manage this process and use the services of security researchers as required for subject matter expertise. While a group of this type wouldn’t ensure that all partial disclosure is appropriate, it would hopefully limit the potential for abuse and the ever present chance that people try to profit from the FUD that surrounds the current partial disclosure process.

Martin MC Brown at ComputerWorld has a great post on capacity planning and cloud computing. He discusses a new book “The Art of Capacity Planning”. The problem with the current model of data center management is that often a large number of machines may sit relatively idle while waiting for the traffic spike that causes them to be used. This is a problem because it’s simply a waste of time and resources on a whole number of levels. Enter the cloud – or at least the “hope of cloud computing”.

Numbers – what do they really mean? IDC released a statement with a whole bunch of them from their “Worldwide Quarterly Server Virtualization Tracker”. The most interesting stat: x86 Virtualization License Market Standings. VMware owns 44% of the market, but Microsoft, in its first quarter of general availability for Microsoft Hyper-V (plus Virtual Server 2005), has 23% of the market of new shipments.

These “take-down” companies are generally specialist offshoots of more general “brand protection” companies, and are hired by banks to handle removal of fake phishing websites.

When we examined our data more carefully we found that we were receiving “feeds” of phishing website URLs from several different sources — and the “take-down” companies that were passing the data to us were not passing the data to each other.

So it often occurs that take-down company A knows about a phishing website targeting a particular bank, but take-down company B is ignorant of its existence. If it is company B that has the contract for removing sites for that bank then, since they don’t know the website exists, they take no action and the site stays up.

Since we were receiving data feeds from both company A and company B, we knew the site existed and we measured its lifetime — which is much extended. In fact, it’s somewhat of a mystery why it is removed at all! Our best guess is that reports made directly to ISPs trigger removal.

The paper contains all the details, and gives all the figures to show that website lifetimes are extended by about 5 days when the take-down company is completely unaware of the site. On other occasions the company learns about the site some time after it is first detected by someone else; and this extends the lifetimes by an average of 2 days.

Since extended lifetimes equate to more unsuspecting visitors handing over their credentials and having their bank accounts cleaned out, these delays can also be expressed in monetary terms. Using the rough and ready model we developed last year, we estimate that an extra $326 million per annum is currently being put at risk by the lack of data sharing. This figure is from our analysis of just two companies’ feeds, and there are several more such companies in this business.

Not surprisingly, our paper suggests that the take-down companies should be sharing their data, so that when they learn about websites attacking banks they don’t have contracts with, they pass the details on to another company who can start to get the site removed.

We analyse the incentives to make this change (and the incentives the companies have not to do so) and contrast the current arrangements with the anti-virus/malware industry — where sample suspect code has been shared since the early 1990s.

In particular, we note that it is the banks who would benefit most from data sharing — and since they are paying the bills, we think that they may well be in a position to force through changes in policy. To best protect the public, we must hope that this happens soon.