Hash functions are the most commonly used cryptographic primitive, and the most poorly understood. You can think of them as fingerprint functions: They take an arbitrary long data stream and return a fixed length, and effectively unique, string. The security comes from the fact that while it's easy to generate the fingerprint from a file, it's infeasible to go the other way and generate a file given a fingerprint.

Originally created to make digital signatures more efficient, hashes are now used to secure the very fundamentals of our information infrastructure: in password logins, secure web connections, encryption key management, virus and malware scanning, and almost every cryptographic protocol in current use. Without cryptographic hash functions, the internet would simply not work. At the same time, there isn't a good theory of hash functions. Unlike encryption algorithms, there are no secret keys involved; this makes it harder to mathematically define exactly what hash functions are.

The National Institute of Standards and Technology, NIST, is holding a competition to replace the SHA family of hash functions. "SHA" stands for "Secure Hash Algorithm." It was developed by the NSA in 1993 to replace the commercial MD4 and MD5 algorithms, and has been updated several times since then. All the SHA algorithms are very similar, and have been increasingly under attack, so NIST wants to replace them.

The competition is important because, unlike other technological standards, committee design — balancing the interests of diverse constituents — isn't conducive to good security. Security is best when it's designed by expert teams and then subjected to public review. And cryptography is best when it's chosen by competition.

In 1997, NIST held a competition for a block cipher to replace DES. Fifteen candidates and three-and-a-half years later, Rijndael became the new Advanced Encryption Standard — AES. NIST is doing the same thing for what it's calling SHA-3 (not, for some unexplained reason, the Advanced Hash Standard or AHS).

The deadline was October 31, and NIST received 64 submissions. This isn't surprising — I predicted 80 — as most of the 15 AES submitters were professors, whose students at the time have become professors themselves, with their own students. (If NIST does a stream cipher competition in another ten years, they should expect about 256 submissions.) These submissions came from academia, from industry, and from hobbyists. CIO magazine recently interviewed one of the submitters, who is 15. Twenty-eight submissions have been made public by the submitters, and six of those have been broken.

NIST is going through all the submissions right now, making sure they are complete and proper. Their goal is to publish all accepted submissions by the end of November, in advance of the First Hash Function Candidate Conference, to be held in Belgium right after the Fast Software Encryption workshop in February.

The group expects to quickly make a first cut of algorithms — hopefully to about a dozen — and give the community a year of cryptanalysis before making a second cut in 2010. After another year of cryptanalysis, NIST will choose a winner in 2011. Expect a final standard by 2012.

My advice for software developers is to let the process run its course. While it's tempting to use the new cool algorithms in your designs, it's far too soon to trust any of them. This process is likely to result in all sorts of new research results in hash function security, and some real cryptanalytic surprises. Give the community a few years to figure out which ones are good and which aren't.

I've previously called this sort of thing a cryptographic demolition derby: The last one left standing wins. But that's only partially true. Certainly all the groups will spend the next few years trying to cryptanalyze each other, but in the end there will be a bunch of unbroken algorithms. NIST will select one based on performance and features.

NIST has stated that the goal of this process is not to choose the best standard but to choose a good standard. I think that's smart; in this process, the best is the enemy of the good. While there's no rush to choose a new standard — the SHA-2 algorithms will remain secure for the foreseeable future — we don't want to analyze the candidates forever.

Personally, I was part of a group of eight cryptographers that submitted Skein to the competition. A decade ago, writing Twofish and participating in the AES process was the most fun I had ever had in cryptography. These next few years promise to be even more fun.

---

Bruce Schneier is chief security technology officer of BT. His new book is Schneier on Security.

IBM’s presentation was cleverly titled 2mor0@Wrk - Tomororow work and Web 2.0.

Overview

Web 2.0 is delivering a whole different paradigm of communication. The slide is Lotus Symphony - NOT PPT. Over 2 million downloads.

There is an information overload that impacts individual productivity in the workplace. It has a profound effect on organizational productivity. A more complex organization entity provides more pressure and more inefficiencies in workplace. Up to 70% of time can be used looking for the WRONG information.

Collaboration mitigates information overload. It allows you to identify experts and opinions.

The collaboration agenda. Enterprises are at the onset of exploring these features. Web 2.0 is giving us the capacity to do more. Collaboration optimizes business outcomes - global, secure and dynamic.The most progressive companies are looking at UNIFIED COMMUNICATIONS. Making sure that directories and profiles are fully mobile.

Collaboration should be a contextual part of the workflow, going directly into applications.

IBM’s collaboration strategy is to deliver these services through online or offline services.

Demonstration

Executive IT architect Ron Sebastian provided a demonstration of IBM’s collaboration strategy. IBM’s Web 2.0 solutions span delivery platforms:

• Platform - web as  platform
• Application - development
• People - social computing

Lotus Connections - a family of social computing software that provides profile lookup and community capabilities. Think of Facebook, Yahoo Groups, and delicious combined in one portal.

Ron demonstrated these social services embedded into a healthcare provider application. Semantic tagging is available, contact information and commenting. Not only are we providing service to customers, you can integrate sync capability to directly call the person you want.

The biggest aspect of Lotus Connection? It’s all integrated.

A new service - Project Bluehouse. This is a SaaS delivery of these collaborated capabilities. The store and share can manage and share documents within and outside the company. Access control is no longer an issue.

Collaborative Web 2.0 services available as standalone products that also work in a mobile environment.

Case Study: Natural Disaster Management Mashup

Boeing came up with twenty different scenarios that they could handle through their systems. The problem was the one they didn’t count on. One example was Katrina - how to deliver supplies to the area: what airports were open? Where could they land? The problem was they could not find one list of public, private and military airports, nor what was open. The mashup took different feeds to allow the deacon maker to make a more rapid and intelligent decision based on information on where they could fly in the appropriate supplies. From open information sites like AirNav.com and personal contacts, users were able to mashup the information to make better decisions.

Conclusion

IBM announced the IBM Center for Social Software, proving their commitment to connect, collaborate, and innovate. Users and academics can work together to how these innovations can be applied to businesses and provide value to the market.

There has been some question of whether or not IBM can pull this off and move into the collaborative Web 2.0 market. Despite some criticism, it looks like IBM has really taken a step forward in advancing their products and services to meet market needs.

People drive better business outcomes. Connecting, collaboration, and innovation is key. Having the right tools and information to do that eases pressure that many organizations feel and brings Web 2.0 technologies to the heart of businesses.

The USB stick, outlining training for 70 soldiers from the 3rd Battalion, Yorkshire Regiment, was found on the floor of The Beach in Newquay in May.

Times, locations and travel and accommodation details for the troops were included in files on the device.

It's not the first time:

More than 120 USB memory sticks, some containing secret information, have been lost or stolen from the Ministry of Defence since 2004, it was reported earlier this year.

Some 26 of those disappeared this year == including three which contained information classified as "secret", and 19 which were "restricted".

I've written about this general problem before: we're storing ever more data in ever smaller devices.

The point is that it's now amazingly easy to lose an enormous amount of information. Twenty years ago, someone could break into my office and copy every customer file, every piece of correspondence, everything about my professional life. Today, all he has to do is steal my computer. Or my portable backup drive. Or my small stack of DVD backups. Furthermore, he could sneak into my office and copy all this data, and I'd never know it.

The solution? Encrypt them.

1. Our brilliant field engineer Dimitri McKay talks about the eternal topic of converting Windows event logs to syslog. Yes, Eric, we ALL know it is ugly, but that is the only way that actually works well across all systems ...
2. More on Windows and syslog: "Syslog ... 20 Years Later."  BTW, this is really not about syslog, but about Vista/2k8 finally getting an ability to natively centralize the event logs via event subscriptions ("It's only about twenty years behind schedule, if you're counting.")
3. Two fun pieces on correlation: 1 and 2. What often kills "a log correlation project"? "Whoever had worked on it had not had much time available to learn the way to properly configure the software" (from this)  and "correlation only really works when backed up by real data about what is the biggest problem in your environment, and how that problem manifests itself in the event logs." (from this) None of this is new, but a useful reminder nonetheless
4. Fun LogLogic podcast is here. The topic of this high-level discussion (CEO) is related to operational use for logs. I did one with them too; on logs and virtualization (will be up soon)
5. A couple of good posts on logging from Nemertes Research: "Sharpening Stones and Walking on Coals",  "Search or Destroy"
6. Reminder about a few useful Windows Vista and 2k8 events: 4802 (screensaver engaged) and 4803 (screensaver dismissed)
7. One person is wondering about the usefulness of logging after "experiencing" Linux auditd logging (kernel audit): "Logs are like a warm blanket; verbose logging means you can know what's happening on your systems if you keep up with the logs.  At the same time, logs become a burden very very easily, and they are easy to ignore." This post is a must read for us logging afficionados; producing too much log data is a sure way to make people hate you...
8. This also follows the same theme: people doubting the god-like power of logs :-) "So for an administrator to not care about logs was a shock." But would I argue that "log management is NOT a pain?" Now, would I? :-)
9. A classic about logging for application developers: "Building Secure Applications: Consistent Logging."  I am noticing a lot more discussions about logging in a developer community, e.g. see this and this (the latter, BTW, contains a lot of info on "why log" for developers). Overall, "getting logging right" is important (and will get more important in the future) and people need something NOW and cannot wait for the standards.  BTW, I am planning a mini-crusade on how to train application developers to include useful logging in their applications...
10. Finally, the "Is SIEM dead?" theme is continued in this fun post "Life after SIEM. Situational Awareness is next." Indeed, context is key for logs. BTW, if somebody mentions that I have "vendor bias", I will kick your ass! :-)

Enjoy!

Imagine you're in charge of infiltrating sleeper agents into the United States. The year is 1983, and the proliferation of identity databases is making it increasingly difficult to create fake credentials. Ten years ago, someone could have just shown up in the country and gotten a driver's license, Social Security card and bank account -- possibly using the identity of someone roughly the same age who died as a young child -- but it's getting harder. And you know that trend will only continue. So you decide to grow your own identities.

Call it "identity farming." You invent a handful of infants. You apply for Social Security numbers for them. Eventually, you open bank accounts for them, file tax returns for them, register them to vote, and apply for credit cards in their name. And now, 25 years later, you have a handful of identities ready and waiting for some real people to step into them.

There are some complications, of course. Maybe you need people to sign their name as parents -- or, at least, mothers. Maybe you need to doctors to fill out birth certificates. Maybe you need to fill out paperwork certifying that you're home-schooling these children. You'll certainly want to exercise their financial identity: depositing money into their bank accounts and withdrawing it from ATMs, using their credit cards and paying the bills, and so on. And you'll need to establish some sort of addresses for them, even if it is just a mail drop.

You won't be able to get driver's licenses or photo IDs on their name. That isn't critical, though; in the U.S., more than 20 million adult citizens don't have photo IDs. But other than that, I can't think of any reason why identity farming wouldn't work.

Here's the real question: Do you actually have to show up for any part of your life?

Again, I made this all up. I have no evidence that anyone is actually doing this. It's not something a criminal organization is likely to do; twenty-five years is too distant a payoff horizon. The same logic holds true for terrorist organizations; it's not worth it. It might have been worth it to the KGB -- although perhaps harder to justify after the Soviet Union broke up in 1991 -- and might be an attractive option to existing intelligence adversaries like China.

Immortals could also use this trick to self-perpetuate themselves, inventing their own children and gradually assuming their identity, then killing their parents off. They could even show up for their own driver's license photos, wearing a beard as the father and blue spiked hair as the son. I’m told this is a common idea in Highlander fan fiction.

The point isn't to create another movie plot threat, but to point out the central role that data has taken on in our lives. Previously, I've said that we all have a data shadow that follows us around, and that more and more institutions interact with our data shadows instead of with us. We only intersect with our data shadows once in a while -- when we apply for a driver's license or passport, for example -- and those interactions are authenticated by older, less-secure interactions. The rest of the world assumes that our photo IDs glue us to our data shadows, ignoring the rather flimsy connection between us and our plastic cards. (And, no, REAL-ID won't help.)

It seems to me that our data shadows are becoming increasingly distinct from us, almost with a life of their own. What's important now is our shadows; we're secondary. And as our society relies more and more on these shadows, we might even become unnecessary.

Our data shadows can live a perfectly normal life without us.

This essay previously appeared on Wired.com.

EDITED TO ADD (9/9): Interesting commentary.

Imagine you're in charge of infiltrating sleeper agents into the United States. The year is 1983, and the proliferation of identity databases is making it increasingly difficult to create fake credentials. Ten years ago, someone could have just shown up in the country and gotten a driver's license, Social Security card and bank account -- possibly using the identity of someone roughly the same age who died as a young child -- but it's getting harder. And you know that trend will only continue. So you decide to grow your own identities.

Call it "identity farming." You invent a handful of infants. You apply for Social Security numbers for them. Eventually, you open bank accounts for them, file tax returns for them, register them to vote, and apply for credit cards in their name. And now, 25 years later, you have a handful of identities ready and waiting for some real people to step into them.

There are some complications, of course. Maybe you need people to sign their name as parents -- or, at least, mothers. Maybe you need to doctors to fill out birth certificates. Maybe you need to fill out paperwork certifying that you're home-schooling these children. You'll certainly want to exercise their financial identity: depositing money into their bank accounts and withdrawing it from ATMs, using their credit cards and paying the bills, and so on. And you'll need to establish some sort of addresses for them, even if it is just a mail drop.

You won't be able to get driver's licenses or photo IDs on their name. That isn't critical, though; in the U.S., more than 20 million adult citizens don't have photo IDs. But other than that, I can't think of any reason why identity farming wouldn't work.

Here's the real question: Do you actually have to show up for any part of your life?

Again, I made this all up. I have no evidence that anyone is actually doing this. It's not something a criminal organization is likely to do; twenty-five years is too distant a payoff horizon. The same logic holds true for terrorist organizations; it's not worth it. It might have been worth it to the KGB -- although perhaps harder to justify after the Soviet Union broke up in 1991 -- and might be an attractive option to existing intelligence adversaries like China.

Immortals could also use this trick to self-perpetuate themselves, inventing their own children and gradually assuming their identity, then killing their parents off. They could even show up for their own driver's license photos, wearing a beard as the father and blue spiked hair as the son. I’m told this is a common idea in Highlander fan fiction.

The point isn't to create another movie plot threat, but to point out the central role that data has taken on in our lives. Previously, I've said that we all have a data shadow that follows us around, and that more and more institutions interact with our data shadows instead of with us. We only intersect with our data shadows once in a while -- when we apply for a driver's license or passport, for example -- and those interactions are authenticated by older, less-secure interactions. The rest of the world assumes that our photo IDs glue us to our data shadows, ignoring the rather flimsy connection between us and our plastic cards. (And, no, REAL-ID won't help.)

It seems to me that our data shadows are becoming increasingly distinct from us, almost with a life of their own. What's important now is our shadows; we're secondary. And as our society relies more and more on these shadows, we might even become unnecessary.

Our data shadows can live a perfectly normal life without us.

---

Bruce Schneier is Chief Security Technology Officer of BT, and author of Beyond Fear: Thinking Sensibly About Security in an Uncertain World.

“Event processing has been going on for more than fifty years.”

However, in On Event Processing as a Discipline and Some Subsets another colleague mistakenly blogs,

“… people who dealt in this area [network management and event correlation] have never investigated event processing in the larger sense (e.g. looking at additional patterns), and this area has also not spawned the event processing discipline.”

If you examine just one page from the CEP history at Stanford, researchers there outlined their view of the future applications for CEP, as follows:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer.
  • Analysing business processes
• Network Level Monitoring and Management
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring
• Analysis and Debugging of Distributed Systems

These applications areas mentioned by Stanford researchers, including Professor Luckham, support and validate our recent discussion Magic Quadrant for IT Event Correlation and Analysis, 2007 where we concluded that “event correlation and event analysis is Gartner’s closest magic quadrant (MQ)  [...] relates directly to complex event processing (and event processing in general).”  

If you take a detailed look at the 1999 CEP presentation, Defeating Large Scale Attacks: Technology and Strategies for Global Network Monitoring you will readily see that our colleagues are incorrect when they says that event correlational and network management folks have never investigated event processing in the “larger sense”.  For example, the 1999 slides above, Stanford, slide 6, is titled “Complex Event Processing,” defining CEP from the application perspective of event correlation;

Complex Event Processing

• Accept network ‘events’ from any source
  • CISCO NetFlow FlowCollector, tcpdump
• Correlates events based on content and temporal relationship between events
• Event Processing Agents (EPAs) connected in an Event Processing Network (EPNs)
• Both post-mortem and real-time processing

This single event correlational project example from David’s team at Stanford examined the challenging event correlation problems in the context of hierarchical events, maps, patterns, visualization tools, event processing models, patterns languages, network management abstraction layers, and more.  Those core event processing problems from this 1999 example, very large and complex then, still exist today and are much more large and complex - precisely why it is called “complex event processing.”

It is quite obvious, in just this one example, that many folks have been looking at event correlation as a motivating application for event processing, in a larger context, for a long time, contrary to what our colleagues write in their “history of event processing” posts.  

In a future post I will completely debuke these event processing “history revisionists.”   I will illustrate very clearly how the history of event processing goes back at least a decade, and perhaps two (twenty years) before the history outlined in posts like On Research and Practice in Event Processing and The History of Complex Event Processing. 

David Luckam stated that the art-and-science of event processing goes back around 50 years. 

I am not sure I will go all the way back to 1960 in my next post on the history of event processing.  However,  I will go back at least to the early days of Internet Protocol (IP) networking and illustrate why distributed IP networking, network management and network security, is one of the key  motivating factors for what we now call “event processing” and “complex event processing.”

Preface

The TCP/IP protocols were conceived during a time that was quite different from the hostile environment they operate in now. Yet a direct result of their effectiveness and widespread early adoption is that much of today's global economy remains dependent upon them.

While many textbooks and articles have created the myth that the Internet Protocols (IP) were designed for warfare environments, the top level goal for the DARPA Internet Program was the sharing of large service machines on the ARPANET. As a result, many protocol specifications focus only on the operational aspects of the protocols they specify and overlook their security implications.

Though Internet technology has evolved, the building blocks are basically the same core protocols adopted by the ARPANET more than two decades ago. During the last twenty years many vulnerabilities have been identified in the TCP/IP stacks of a number of systems. Some were flaws in protocol implementations which affect only a reduced number of systems. Others were flaws in the protocols themselves affecting virtually every existing implementation. Even in the last couple of years researchers were still working on security problems in the core protocols.

The discovery of vulnerabilities in the TCP/IP protocols led to reports being published by a number of CSIRTs (Computer Security Incident Response Teams) and vendors, which helped to raise awareness about the threats as well as the best mitigations known at the time the reports were published.

Much of the effort of the security community on the Internet protocols did not result in official documents (RFCs) being issued by the IETF (Internet Engineering Task Force) leading to a situation in which "known" security problems have not always been addressed by all vendors. In many cases vendors have implemented quick "fixes" to protocol flaws without a careful analysis of their effectiveness and their impact on interoperability.

As a result, any system built in the future according to the official TCP/IP specifications might reincarnate security flaws that have already hit our communication systems in the past.

Producing a secure TCP/IP implementation nowadays is a very difficult task partly because of no single document that can serve as a security roadmap for the protocols.

There is clearly a need for a companion document to the IETF specifications that discusses the security aspects and implications of the protocols, identifies the possible threats, proposes possible counter-measures, and analyses their respective effectiveness.

This document is the result of an assessment of the IETF specifications of the Internet Protocol from a security point of view. Possible threats were identified and, where possible, counter-measures were proposed. Additionally, many implementation flaws that have led to security vulnerabilities have been referenced in the hope that future implementations will not incur the same problems. This document does not limit itself to performing a security assessment of the relevant IETF specification but also offers an assessment of common implementation strategies.

Whilst not aiming to be the final word on the security of the IP, this document aims to raise awareness about the many security threats based on the IP protocol that have been faced in the past, those that we are currently facing, and those we may still have to deal with in the future. It provides advice for the secure implementation of the IP, and also insights about the security aspects of the IP that may be of help to the Internet operations community.

Feedback from the community is more than encouraged to help this document be as accurate as possible and to keep it updated as new threats are discovered.