The report was written by a committee whose members include William Perry, a professor at Stanford University; Charles Vest, the former president of MIT; W. Earl Boebert, a retired senior scientist at Sandia National Laboratories; Cynthia Dwork of Microsoft Research; R. Gil Kerlikowske, Seattle's police chief; and Daryl Pregibon, a research scientist at Google.

They admit that far more Americans live their lives online, using everything from VoIP phones to Facebook to RFID tags in automobiles, than a decade ago, and the databases created by those activities are tempting targets for federal agencies. And they draw a distinction between subject-based data mining (starting with one individual and looking for connections) compared with pattern-based data mining (looking for anomalous activities that could show illegal activities).

But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so."

A summary of the recommendations:

• U.S. government agencies should be required to follow a systematic process to evaluate the effectiveness, lawfulness, and consistency with U.S. values of every information-based program, whether classified or unclassified, for detecting and countering terrorists before it can be deployed, and periodically thereafter.

• Periodically after a program has been operationally deployed, and in particular before a program enters a new phase in its life cycle, policy makers should (carefully review) the program before allowing it to continue operations or to proceed to the next phase.

• To protect the privacy of innocent people, the research and development of any information-based counterterrorism program should be conducted with synthetic population data... At all stages of a phased deployment, data about individuals should be rigorously subjected to the full safeguards of the framework.

• Any information-based counterterrorism program of the U.S. government should be subjected to robust, independent oversight of the operations of that program, a part of which would entail a practice of using the same data mining technologies to "mine the miners and track the trackers."

• Counterterrorism programs should provide meaningful redress to any individuals inappropriately harmed by their operation.

• The U.S. government should periodically review the nation's laws, policies, and procedures that protect individuals' private information for relevance and effectiveness in light of changing technologies and circumstances. In particular, Congress should re-examine existing law to consider how privacy should be protected in the context of information-based programs (e.g., data mining) for counterterrorism.

Here are more news articles on the report. I explained why data mining wouldn't find terrorists back in 2005.

The Maryland State Police classified 53 nonviolent activists as terrorists and entered their names and personal information into state and federal databases that track terrorism suspects, the state police chief acknowledged yesterday.

Why did they do that?

Both Hutchins and Sheridan said the activists' names were entered into the state police database as terrorists partly because the software offered limited options for classifying entries.

I know that once we had this "either you're with us or with the terrorists" mentality, but don't you think that -- just maybe -- the software should allow for a little bit more nuance?

Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

Conventional wisdom holds that terrorism is inherently political, and that people become terrorists for political reasons. This is the "strategic" model of terrorism, and it's basically an economic model. It posits that people resort to terrorism when they believe -- rightly or wrongly -- that terrorism is worth it; that is, when they believe the political gains of terrorism minus the political costs are greater than if they engaged in some other, more peaceful form of protest. It's assumed, for example, that people join Hamas to achieve a Palestinian state; that people join the PKK to attain a Kurdish national homeland; and that people join al-Qaida to, among other things, get the United States out of the Persian Gulf.

If you believe this model, the way to fight terrorism is to change that equation, and that's what most experts advocate. Governments tend to minimize the political gains of terrorism through a no-concessions policy; the international community tends to recommend reducing the political grievances of terrorists via appeasement, in hopes of getting them to renounce violence. Both advocate policies to provide effective nonviolent alternatives, like free elections.

Historically, none of these solutions has worked with any regularity. Max Abrahms, a predoctoral fellow at Stanford University's Center for International Security and Cooperation, has studied dozens of terrorist groups from all over the world. He argues that the model is wrong. In a paper published this year in International Security that -- sadly -- doesn't have the title "Seven Habits of Highly Ineffective Terrorists," he discusses, well, seven habits of highly ineffective terrorists. These seven tendencies are seen in terrorist organizations all over the world, and they directly contradict the theory that terrorists are political maximizers:

Terrorists, he writes, (1) attack civilians, a policy that has a lousy track record of convincing those civilians to give the terrorists what they want; (2) treat terrorism as a first resort, not a last resort, failing to embrace nonviolent alternatives like elections; (3) don't compromise with their target country, even when those compromises are in their best interest politically; (4) have protean political platforms, which regularly, and sometimes radically, change; (5) often engage in anonymous attacks, which precludes the target countries making political concessions to them; (6) regularly attack other terrorist groups with the same political platform; and (7) resist disbanding, even when they consistently fail to achieve their political objectives or when their stated political objectives have been achieved.

Abrahms has an alternative model to explain all this: People turn to terrorism for social solidarity. He theorizes that people join terrorist organizations worldwide in order to be part of a community, much like the reason inner-city youths join gangs in the United States.

The evidence supports this. Individual terrorists often have no prior involvement with a group's political agenda, and often join multiple terrorist groups with incompatible platforms. Individuals who join terrorist groups are frequently not oppressed in any way, and often can't describe the political goals of their organizations. People who join terrorist groups most often have friends or relatives who are members of the group, and the great majority of terrorist are socially isolated: unmarried young men or widowed women who weren't working prior to joining. These things are true for members of terrorist groups as diverse as the IRA and al-Qaida.

For example, several of the 9/11 hijackers planned to fight in Chechnya, but they didn't have the right paperwork so they attacked America instead. The mujahedeen had no idea whom they would attack after the Soviets withdrew from Afghanistan, so they sat around until they came up with a new enemy: America. Pakistani terrorists regularly defect to another terrorist group with a totally different political platform. Many new al-Qaida members say, unconvincingly, that they decided to become a jihadist after reading an extreme, anti-American blog, or after converting to Islam, sometimes just a few weeks before. These people know little about politics or Islam, and they frankly don't even seem to care much about learning more. The blogs they turn to don't have a lot of substance in these areas, even though more informative blogs do exist.

All of this explains the seven habits. It's not that they're ineffective; it's that they have a different goal. They might not be effective politically, but they are effective socially: They all help preserve the group's existence and cohesion.

This kind of analysis isn't just theoretical; it has practical implications for counterterrorism. Not only can we now better understand who is likely to become a terrorist, we can engage in strategies specifically designed to weaken the social bonds within terrorist organizations. Driving a wedge between group members -- commuting prison sentences in exchange for actionable intelligence, planting more double agents within terrorist groups -- will go a long way to weakening the social bonds within those groups.

We also need to pay more attention to the socially marginalized than to the politically downtrodden, like unassimilated communities in Western countries. We need to support vibrant, benign communities and organizations as alternative ways for potential terrorists to get the social cohesion they need. And finally, we need to minimize collateral damage in our counterterrorism operations, as well as clamping down on bigotry and hate crimes, which just creates more dislocation and social isolation, and the inevitable calls for revenge.

This essay previously appeared on Wired.com.

• Love, love, love this piece :-) Remember the "robotic gun rampage" stories from last year? How does this sound: "The gun can track 360 degress, but there is a software-driven safety zone that makes sure rounds don't blow the rotors off. If the Osprey has to maneuver away from the target and the crew chief can't hold the gun on the bad guys manually, the system slaves the gun to the point of the last shot, slewing it as the plane moves." (watch the fun video there too)
  
• "Security idiot" meme lives on - go here. BTW, the post is a follow-up to this
• A fun follow-up to my post on compliance approaches titled Is PCI DSS "Too Prescriptive"?
• Finally, my fave post: "Increase Your Logging." I am sooooo happy that logging evangelism is spreading far and wide! A quote from the paper: ”Logs are interesting, logs are fun, logs should be done by EVERYONE…..get to logging!!!” (I promise that specific case was not my quote, even though I do say that very thing all the time!)
  

Because then you get a Maginot Line problem. [After World War I, the French erected concrete fortifications to prevent Germany from invading again -- a response to the previous war, which proved ineffective for the next one.] You know, they make sure they solve that particular problem, the Germans will not invade from here. The thing you have to be aware of most obviously is scenario planning, because typically if you talk about scenarios, you'll overestimate the probability of these scenarios. If you examine them at the expense of those you don't examine, sometimes it has left a lot of people worse off, so scenario planning can be bad. I'll just take my track record. Those who did scenario planning have not fared better than those who did not do scenario planning. A lot of people have done some kind of "make-sense" type measures, and that has made them more vulnerable because they give the illusion of having done your job. This is the problem with risk management. I always come back to a classical question. Don't give a fool the illusion of risk management. Don't ask someone to guess the number of dentists in Manhattan after asking him the last four digits of his Social Security number. The numbers will always be correlated. I actually did some work on risk management, to show how stupid we are when it comes to risk.

Conventional wisdom holds that terrorism is inherently political, and that people become terrorists for political reasons. This is the "strategic" model of terrorism, and it's basically an economic model. It posits that people resort to terrorism when they believe -- rightly or wrongly -- that terrorism is worth it; that is, when they believe the political gains of terrorism minus the political costs are greater than if they engaged in some other, more peaceful form of protest. It's assumed, for example, that people join Hamas to achieve a Palestinian state; that people join the PKK to attain a Kurdish national homeland; and that people join al-Qaida to, among other things, get the United States out of the Persian Gulf.

If you believe this model, the way to fight terrorism is to change that equation, and that's what most experts advocate. Governments tend to minimize the political gains of terrorism through a no-concessions policy; the international community tends to recommend reducing the political grievances of terrorists via appeasement, in hopes of getting them to renounce violence. Both advocate policies to provide effective nonviolent alternatives, like free elections.

Historically, none of these solutions has worked with any regularity. Max Abrahms, a predoctoral fellow at Stanford University's Center for International Security and Cooperation, has studied dozens of terrorist groups from all over the world. He argues that the model is wrong. In a paper (.pdf) published this year in International Security that -- sadly -- doesn't have the title "Seven Habits of Highly Ineffective Terrorists," he discusses, well, seven habits of highly ineffective terrorists. These seven tendencies are seen in terrorist organizations all over the world, and they directly contradict the theory that terrorists are political maximizers:

Terrorists, he writes, (1) attack civilians, a policy that has a lousy track record of convincing those civilians to give the terrorists what they want; (2) treat terrorism as a first resort, not a last resort, failing to embrace nonviolent alternatives like elections; (3) don't compromise with their target country, even when those compromises are in their best interest politically; (4) have protean political platforms, which regularly, and sometimes radically, change; (5) often engage in anonymous attacks, which precludes the target countries making political concessions to them; (6) regularly attack other terrorist groups with the same political platform; and (7) resist disbanding, even when they consistently fail to achieve their political objectives or when their stated political objectives have been achieved.

Abrahms has an alternative model to explain all this: People turn to terrorism for social solidarity. He theorizes that people join terrorist organizations worldwide in order to be part of a community, much like the reason inner-city youths join gangs in the United States.

The evidence supports this. Individual terrorists often have no prior involvement with a group's political agenda, and often join multiple terrorist groups with incompatible platforms. Individuals who join terrorist groups are frequently not oppressed in any way, and often can't describe the political goals of their organizations. People who join terrorist groups most often have friends or relatives who are members of the group, and the great majority of terrorist are socially isolated: unmarried young men or widowed women who weren't working prior to joining. These things are true for members of terrorist groups as diverse as the IRA and al-Qaida.

For example, several of the 9/11 hijackers planned to fight in Chechnya, but they didn't have the right paperwork so they attacked America instead. The mujahedeen had no idea whom they would attack after the Soviets withdrew from Afghanistan, so they sat around until they came up with a new enemy: America. Pakistani terrorists regularly defect to another terrorist group with a totally different political platform. Many new al-Qaida members say, unconvincingly, that they decided to become a jihadist after reading an extreme, anti-American blog, or after converting to Islam, sometimes just a few weeks before. These people know little about politics or Islam, and they frankly don't even seem to care much about learning more. The blogs they turn to don't have a lot of substance in these areas, even though more informative blogs do exist.

All of this explains the seven habits. It's not that they're ineffective; it's that they have a different goal. They might not be effective politically, but they are effective socially: They all help preserve the group's existence and cohesion.

This kind of analysis isn't just theoretical; it has practical implications for counterterrorism. Not only can we now better understand who is likely to become a terrorist, we can engage in strategies specifically designed to weaken the social bonds within terrorist organizations. Driving a wedge between group members -- commuting prison sentences in exchange for actionable intelligence, planting more double agents within terrorist groups -- will go a long way to weakening the social bonds within those groups.

We also need to pay more attention to the socially marginalized than to the politically downtrodden, like unassimilated communities in Western countries. We need to support vibrant, benign communities and organizations as alternative ways for potential terrorists to get the social cohesion they need. And finally, we need to minimize collateral damage in our counterterrorism operations, as well as clamping down on bigotry and hate crimes, which just creates more dislocation and social isolation, and the inevitable calls for revenge.

---

Bruce Schneier is Chief Security Technology Officer of BT, and author of Beyond Fear: Thinking Sensibly About Security in an Uncertain World.

Originally, researchers in CEP were not trying to solve a problem of streaming data or streaming events.   Often we read this mischaracterization by folks in the database/streaming domain, as they were focused on the low latency processing of streaming events.   A natural extension of this research has been stream processing software (often called “engines”) that process streaming data with continuous queries, for example market data feeds for algo-trading or best market order execution.  This mischaracterization is partly responsible for why we see many order processing applications in market data stream processing mislabled as “complex event processing” applications.

The genesis of complex event processing was not the stream processing need for “feeds and speed” but the processing capability to solve what can be characterized as the “problem of asymmetric capabilties”.   The term “asymmetric” has been used in the military domain. For example we often hear the term “asymmetric warfare.”  However, in general the concept of “asymmetrical processing capablities” is the true genesis for CEP and related processing concepts and domains.   It is this genesis that distinguishes CEP from EDA, SOA, SOR, and so many other technology oriented concepts.

In order to illustrate what I mean by “asymmetrical processing capablities” we will take the example of the evolution of rocketry.    In the early days, scientists learned how to make rockets, I assume with gunpowder and similar chemical compounds to launch rockets.   Over many years the application of rocketry advanced much faster than the ability to understand the situations created in the sky.    In other words, folks could fill the skies with rockets long before they had the capability to track and identify (or sense and respond to)  the rockets in real time.

Therefore, the concept of “asymmetrical processing capablities” is the situation where there is a capability, such as “launch a rocket, sense-and-respond,” that is asymmetric in nature.    In other words, the capability to detect multiple rocket launches creates an asymmetric situation where it is easy to launch rockets, but hard to detect and defend against those launches.

The same concept can be applied to everyday air travel.   If we could only fly airplanes, but did not have the capability to track the planes, understand situations in airspace, and then respond to changing situations, air travel would be quite difficult.   Lucky for us, the global traveller, there is symmetry in the capabilities to build and fly aircraft and the capabilities to detect, track and follow the evolving situations in the sky.

The genesis of CEP was to solve the problem of asymmetry in cyberspace, or if you prefer, distributed data networks.   The folks who identified, early on,  the problems associated with asymmetry in cyberspace were folks working the the field of network and security management.    This is because there has been, and is currently, a great asymmetry between the capablities to “launch a process or transaction” in cyberspace and the capabilties to detect and track what is going on in the same domain.

In my next post on this topic, we will go into some details of this asymmetry and review the first CEP projects from Stanford University in the context of asymmetric processing capabilities in cyberspace.

One of the main problems in counterterrorism today is that there are so many people and vehicles, and so much data and material, moving through globalization's myriad networks that it seems virtually impossible to track it all effectively. Nowhere has this problem been more acute than on the high seas.

In 2006, Adm. Harry Ulrich, then U.S. commander of NATO Naval Forces Europe, decided to do something about it. Despite having virtually no resources, his dream was to transpose the global air-traffic control system onto sea traffic.

Worldwide, aircraft are transparent, because they're all required to carry an identification beacon that allows them to be tracked leaving and entering airports, and monitored between airports, by a global network of sensors. Act suspiciously and somebody's fighter aircraft will soon be on your tail.

No such pervasive system currently exists globally for maritime traffic. While bigger ships carry an ID beacon similar to aircraft, without a shared monitoring network, that's like tracking only selected commercial jets and giving everyone else a pass.
So Ulrich, upon taking command, asked a simple question: "If we can do that in the air, why can't we do it on the sea?" He made a point of pioneering his sea-traffic-control effort first inside the Mediterranean, where NATO's southern naval forces have historically been concentrated, but his real target was waters off Africa -- the most ungoverned maritime space in the world.

Ulrich knew the U. S. Navy couldn't do it alone, much less bring Africa's meager coast-guard-like navies up to snuff so they could do it on their own. So he quickly created a network of assets -- both public and private -- to manage that space, modeling his monitoring system on international air-traffic control.

Ulrich began stitching together a network of shore-based sensors ringing the Mediterranean. His naval command then began initial monitoring by tapping into the International Maritime Organization's existing Automated Identification System, transforming NATO's ability to track ship traffic in the Med.

Almost overnight, NATO went from tracking dozens of ships on the Mediterranean to thousands, and instead of getting the data sometimes up to 72 hours late, now the contacts were being tracked in one to five minutes -- to an accuracy within 50 feet on the earth's surface.

When the classic big-firm systems integrators told Ulrich it would be too costly to pull it off, the admiral turned to the Volpe Center in Cambridge, Massachusetts, a U.S. Department of Transportation research center. Instead of hundreds of millions of dollars, Ulrich's initial network cost $900,000. The shore-based receivers are small, roughly the size of a radar dish you might find on a pleasure craft.

The strength of the system is a function of its reach: the more countries join, the larger the shared operational picture. By the time Ulrich retired at the end of 2007, he had enlisted 32 countries throughout the Mediterranean, the North Atlantic, along the west coast of Africa, around the Black Sea, and in the Pacific. Today, the network continues to spread around the planet.

With Ulrich's system in place, local police, coast guards, and border patrols catch most bad guys, obviating American military responses. As Harry told me for an article I wrote about his work in a fall 2007 issue of Esquire, "I don't do defense; I do security. When you talk defense, you talk containment and mutually assured destruction. When you talk security, you talk collaboration and networking. This is the future."

The admiral's legacy program, the Maritime Safety and Security Information System, earned the Volpe Center a prestigious "Innovations in American Government" award this month from Harvard University's Ash Institute for Democratic Governance and Innovation.