Errata: Version 1.1 of the paper, reference, and optimized code corrects an error in which the length of the configuration string was passed in as the size of the internal block (256 bits for Skein-256, 512 for Skein-512, and 1024 for Skein-1024), instead of a constant 256 bits for all three sizes. This error has no cryptographic significance, but affected the test vectors and the initialization values. The revised code also fixes a bug in the MAC mode key processing. This bug does not affect the NIST submission in any way.

NIST has received 64 submissions. (This article interviews one of the submitters, who is fifteen.) Of those, 28 are public and six have been broken. NIST is going through the submissions right now, making sure they are complete and proper. Their goal is to publish the accepted submissions by the end of the month, in advance of the Third Cryptographic Hash Workshop to be held in Belgium right after FSE in February. They expect to quickly make a first cut of algorithms -- hopefully to about a dozen -- and then give the community about a year of cryptanalysis before making a second cut in 2010.

Lastly, this is a really nice article on Skein.

These submissions make some accommodation to the Core 2 processor. They operate in "little-endian" mode (a quirk of the Intel-like processors that reads some bytes in reverse order). They also allow a large file to be broken into chunks to split the work across multiple processors.

However, virtually all of the contest submissions share the performance problem mentioned above. The logic they use won't optimally fit within the constraints of a Intel Core 2 processor. Most will perform as bad or worse than the existing SHA-1 algorithm.

One exception to this is Skein, created by several well-known cryptographers and noted pundit Bruce Schneier. It was designed specifically to exploit all three of the Core 2 execution units and to run at a full 64-bits. This gives it roughly four to 10 times the logic density of competing submissions.

This is what I meant by the Matrix quote above. They didn't bend the spoon; they bent the crypto algorithm. They moved the logic operations around in a way that wouldn't weaken the crypto, but would strengthen its speed on the Intel Core 2.

In their paper (PDF), the authors of Skein express surprise that a custom silicon ASIC implementation is not any faster than the software implementation. They shouldn't be surprised. Every time you can redefine a problem to run optimally in software, you will reach the same speeds you get with optimized ASIC hardware. The reason software has a reputation of being slow is because people don't redefine the original problem.

That's exactly what we were trying to do.

Around the same time that Professor Luckham and his team was working on CEP applications in network management and security management, I was leading efforts to build network and security management control centers for the United States Air Force.  In the beginning, dating back to 1994, my Internet-related work was for Air Combat Command (ACC), working out of ACC headquarters at Langley Air Force Base.

In 1997, I lead a technical team that developed countermeasures against an actual distributed Internet-based attack on the Langley AFB SMTP email infrastructure.  This attack was documented in a technical paper, E-Mail Bombs and Countermeasures: Cyber Attacks on Availability and Brand Integrity, IEEE Network Magazine, Vol. 12, No. 2, pp. 10-17, March/April 1998.  In addition, this attackand countermeasures I designed was featured in Popular Science Magazine in an 1998 article, War.Com and other news channels.  I also published a number of related papers on this topic.

Our team used a rule-based approach for countermeasures against massive email bombs attacks on the Langley Air Force Base email infrastructure.   We called this rule-based system, BombShelter. and it was written in PERL.  I developed both the original software architecture and the original working prototype for BombShelter (in two days) and then we turned the software over to our team who used the rule-based approach for daily attack countermeasures.

I watched for days, and then weeks, as my team designed rules, and the attackers wrote new attacks that circumvented the rules.  Some folks in the Pentagon used to say that I “lead the effort to fight the first war in cyberspace”.   It might have have been the first cyberwar, I am not sure, but it was certainly the first publicly documented cyberwar.  There is no doubt about this.

Without getting into all the historical footnotes and significance of this cyberwar that was fought with experts and rule-based systems, I would like to jump to an important conclusion.

Rule-based systems are useful, but have limited functionality and scaleability in most complex event processing applications.

Rule-based systems are human resource intensive because rule-based systems cannot learn and adapt on their own, humans learn and then write new rules.  This is how rule-based systems work.

This is the motivation behind why I spend a lot of time to search for new, more efficient and adaptive methods as alternatives to rule-based systems.   After extensive research, I published a series of papers on the future of intrusion detection in the Internet.  Intrusion Detection Systems & Multisensor Data Fusion - Creating Cyberspace Situational Awareness [1], helped lead an evolution in Internet security, particularly in the area of network-based intrusion detection systems (IDS).

In my published research work, motivated by limitations with rule-based approaches, I used the same mature functional model that is used to process missile attacks, control global air traffic, and other complex event processing applications in physical space; but I applied these concepts to cyberspace.

Around the same time, Professor Luckham and others were working on similar problems, all related to real-time detection and response to threats in cyberspace.  They were also funded by the US government.

Sidebar: Stream processing of transaction- based systems (databases), another area of interest, was focused on a totally different problem, which was the low latency processing of straight-thru processing in databased-oriented systems.   These stream processing systems were, and remain however,  rule-based systems.  The problems we were trying to solve in cyberspace, however, cannot be efficiently and pragmatically solved by rule-based systems alone.  Only relatively simple scenarios can be efficiently detected by rule-based stream processing systems.

The vast majority of complex event processing classes of problems require rules plus advanced algorithms that can learn and adapt in real-time.    I know this, not from reading papers or taking university classes on rule-bases systems, but from working on some very challenging operational problems in real-time.    This is why I remain interested in complex event processing and why I continue to elaborate on why rule-based systems have limitations.

adaptive process management (n.) an element of resource and business process management, adaptive search and event processing. Sometimes referred to as “Level 4” event processing or process refinement.

application concept (n.) a definition of a set of properties that represent the data fields of an application entity. An application concept can describe relationships among themselves. For example, an order concept might have a parent/child relationship with an item concept. A department concept might be related to a purchase requisition concept based on the shared property, department_id. Application concepts can include an application state model.

application state modeler (n.) a UML-compliant application that allows you to model the life cycle of a concept instance — that is, for each instance of a given concept, you can define which states it will pass through and how it will transition from state to state. States have entry actions, exit actions, and conditions, providing precision control over the behavior of an event processing agent. Transitions between states also may have rules. Multiple types of states and transitions maximize the versatility and power of the application state modeler.

derived event (n.) an event that is created as a result of processing one or more other events.

complex event (n.) an event that is a situation-entity abstraction of two or more simple, derived or other complex events.

complex event processing (n.) CEP is a technology for extracting information from message-based systems. CEP is primarily an event processing concept that deals with the task of processing multiple events from an event cloud with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes.

event (n.) a instance of an event definition. It is an immutable object that represents a business activity that happened at a single point in time. Just as one cannot change the fact that a given activity occurred, one cannot change an event — events are immutable.

event aggregation (n.) the aggregation of simple, derived or complex events into higher levels of event abstractions.

event definition (n.) a set of properties related to a given activity that represents an important or interesting change of state in a human, system or computational activity. An event definition includes event properties such as event priority, event time to live (TTL), and a description of the payload, which is comprehensive information related to the activity that occurred. Events expire when the TTL has elapsed, unless the event processing agent has instructions to consume them prior to that time.

event channel (n.) a communications channel in which events are transmitted from event source to event receivers, typically received as electronic messages. Each channel can have multiple destination and. events can be configured to transmit to a default destination. JMS is an example of an event channel.

event cloud (n.) a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events. Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud.

event-driven (n.) the behavior of a human, system or computational entity whose execution or actuation is in response to events, typically received as electronic messages.

event-driven architecture (n.) an architectural style for distributed computing applications in which some of the components are event-driven and communicate by means of events.

event processing (n.) computing that performs operations on events, including modifying, creating and destroying events.

event-object (n.) an software object that represents an event, generally for the purpose of computer processing, that exhibits both encapsulation, inheritance and polymorphism.

event prediction (n.) computational activity where the impact of events, complex events, and situations caused by events identified, including both opportunity or threat. Sometimes referred to as “Level 2” event processing, impact assessment or predictive analytics.

event pre-processing (n.) computational activity where events are cleansed or normalized to produce semantically understandable data. Sometimes referred to as “Level 0” event processing.

event processing (n.) computational activities on events dealing with the association, correlation, and combination of event data and information from single and multiple event sources to achieve refined identity and situation estimates for observed event objects, and to achieve complete and timely assessments of opportunities, threats, and their significance. Event processing is characterized by continuous refinements of event estimates and assessments and by evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

event processing agent (n.) an EPA is a computational entity that performs event processing.

event processing network (n.) a set of event processing agents and a set of event channels connecting them.

event properties (n.) data representation of an event, typically by name-value pairs of type string, integer, real, boolean or a complex data type.

event refinement (n.) filter, identify and track events & make initial processing decisions based on association, correlation and state estimation. Sometimes referred to as “Level 1” event, or event-object, track and trace.

event stream (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

event stream processing (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

rule (n.) defines what triggers unusual, suspicious, problematic, or advantageous activity within an event processing agent and what the EPA does when it discovers these types of activities. Rules execute actions based on certain conditions on events, instances, or a combination of both. A rule includes a group of condition-rule statements and action-rule statements. The condition statements instruct the EPA what to look for in events, and action statements instruct the EPA how to respond when conditions are met. If all the conditions in a rule are satisfied by events or instances or both, the EPA fires the actions. The action might be to execute tasks, create an event instance, modify property values in an event instance, create and send an event, or something else.

rules engine (n.) a type of event processing agent that uses a declarative programming model to process events. Formally described as “an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet“. Informally, it can be any system that uses rules, in any form, that can be applied to data to produce outcomes.

rule language (n.) is an artificial language that is used to control the behavior of an event processing agent. Rules languages, like human languages, have syntactic and semantic rules to define meaning.

situation refinement (n.) identify situations, or complex events, based on event clustering, event-event relationships and relationship analysis and context. Sometimes referred to as “Level 2” event processing.

simple event (n.) an event that is not an abstraction or composition of other events.

virtual event (n.) an event that is imagined, modeled or simulated.

Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software.  He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.”  I think that sums up the problem pretty well.  Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think.  The same thinking is used by people who install social networking applets and give them access to their personal data.  Someone, somewhere is taking care of the software security so I don’t have to.  It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives.  I’m safe in a plane or car because the government is looking out for me with standards and testing requirements.  I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space.  I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process.  I ask them if they have made security quality part of the requirements of the project and they say no.  Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers.  They have no evidence.  Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for.  So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems.  The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source.  Surely the developers writing the software are trained in secure software engineering.  Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers.  The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog.  He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment.  Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software. He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.” I think that sums up the problem pretty well. Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think. The same thinking is used by people who install social networking applets and give them access to their personal data. Someone, somewhere is taking care of the software security so I don’t have to. It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives. I’m safe in a plane or car because the government is looking out for me with standards and testing requirements. I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space. I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process. I ask them if they have made security quality part of the requirements of the project and they say no. Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers. They have no evidence. Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for. So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems. The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source. Surely the developers writing the software are trained in secure software engineering. Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers. The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog. He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment. Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

Unsurprisingly, the vendors and their trade organization are spinning the results of these studies, as best they can, in an attempt to downplay their significance. Hopefully, legislators and election administrators are smart enough to grasp the vendors’ behavior for what it actually is and take appropriate steps to bolster our election integrity. Until then, the bottom line is that many jurisdictions in Texas and elsewhere in the country will be using e-voting equipment this November with known security vulnerabilities, and the procedures and controls they are using will not be sufficient to either prevent or detect sophisticated attacks on their e-voting equipment. While there are procedures with the capability to detect many of these attacks (e.g., post-election auditing of voter-verified paper records), Texas has not certified such equipment for use in the state. Texas’s DREs are simply vulnerable to and undefended against attacks.

- Take a risk based approach - Focus on business needs - Talk the language of the business - Don’t make wild statement about cost savings and ROI - Work to reduce costs - Put risk assessments into context - Present a decent set of meaningful security metrics