• "Just A Thought on Compliance"

Turns out a Petabyte is 1000 Terbytes. So Google processes over 20,000 Terabytes of data per day. Supporting all of this impossibly massive data crunching is a huge network of proprietary servers and custom made storage. It's the mythical Google grid.

Google conceals the exact nature of the grid; it's one of their trade secrets.

So, what if I told you Google is abandoning its mythical, proprietary, custom-made processing and storage grid, and is moving to an off-the-shelf third party processing platform?

Any boffin would have choked on this scoop.

OK, relax. Google isn't ditching its proprietary grid. But its eCrime equivalent is certainly doing exactly that.

Defining Cloud Computing

Cloud computing is a very active community. The Google Group gets 600 posts per month and many bloggers are covering the space. However, “cloud computing” is impossible to define in a way that satisfies everyone (or even most). Cloud computing is not alone in this controversy, consider the definition and meaning of “Web 2.0″, “mashups” or “RESTful architecture”. All of these terms are relatively recent. According to Google Trends, these terms became popular to the general public sometime between 2005 and 2007:

• Web 2.0 - often confused with RIA, AKA Social Computing, Long-Tail Apps, Crowdware (2005 by O’Reilly Media)
• Mashup - made popular by Google Maps, AKA Composite/Situational Apps. (2005)
• REST - Has a strict definition, but many don’t understand it and abuse the term. (2006 by R. Fielding)
• Cloud computing - collides with many other terms, such as SaaS, Grid, Utility, PaaS, etc. (2007)

The definition of cloud computing is in progress:

There’s a Darwinian evolution of the exact definition of cloud computing running around. We’re about a country mile away from “knowing when I see it”, which is excellent progress. The cloud to everyone’s silver-lining has enough material to write a 3 volume desktop reference at this point. - Michael Cote, June 2008

Definition #1 - “Cloud computing is the realisation of Internet (”Cloud”) based development and use of computer technology (”Computing”) delivered by an ecosystem of providers. - Sam Johnston, July 2008

Definition #2 - “Cloud computing = network computing. I love the idea of cloud computing, the next evolution of the most network intensive architecture possible, but one that if it works well, is transparent. It’s all about the transparency.” - Douglas Gourlay, Cisco, May 2008

Definition #3 - “There seems to be a group myopia around so-called “cloud computing” and its definitions. What we’re really talking about are “cloud services” of which, “computing” is only a subset…Cloud services are not SaaS. They are far more akin to web services…” - Randy Bias, neoTactics, May 2008

(Anti-)Definition #4 - “Note that I refer to cloud services, not to the could. I am not interested in defining cloud as a term, because I don’t think it’s very useful. For those of us in the distributed computing’s pace

The Working Definition (Winner!):

“…the notion of providing easily accessible compute and storage resources on a pay-as-you-go, on-demand basis, from a virtually infinite infrastructure managed by someone else. As a customer, you don’t know where the resources are, and for the most part, you don’t care. What’s really important is the capability to access your application anywhere, move it freely and easily, and inexpensively add resources for instant scalability.” - Mitchell Crandell, Rightscale, June 2008

Taxonomies of the Cloud Space

Taxonomies are useful to provide insight into a market. It classifies a multitude of players into a smaller bucket.

Andreessen’s Platforms - September 2007

Provided an early taxonomy model for emerging cloud platforms

Platform being a system that can be programmed

• Access API - platform that provides web service endpoints
• Plug-In API - platform invokes your code, that you have deployed remotely
• Runtime Environment - your code runs inside the platform’s process space.

Mehta 11 Layer Stack, April 2008

1. Facilities (space, power, cooling)
2. Network
3. Hardware (e.g. servers Amazon EC2 runs)
4. Hardware virtualization (e.g. Xen for EC2) - optional
5. O/S (e.g. Linux)
6. Systems Management (e.g., tools to manage EC2 instances)
7. Application Middleware (e.g., MySQL on EC2)
8. Application Code
9. Application APIs / Web Services
10. GUI for Application
11. GUI for Application Development / Customization

Croll Cloud Stack, June 2008

7 layer stack within Turnkey app and Generic Platform.

Turnkey app

• SaaS
• Extensible app
• Generic IDE
• Constrained APIs
• App Cluster
• Virtual Data Center
• Virtual Servers

Generic Platform

The bottom of Alistair’s stack includes “root access “style compute clouds.

Robert Anderson, July 2008

3 layer stack

• Software (SaaS)
• Platform (PaaS)
• Infrastructure (IaaS)

This is the model taxonomy for this session.

Related Concepts and Terms

• Infrastructure as a Service (IaaS), Hardware as a Service (HaaS) are synonyms to cloud infrastructure.
• Virtualization
• Hosting
• Autonomic computing
• Distributed computing
• Grid computing

Cloud Applications

• SaaS
• S+S (Software+Services)
• Managed Service Provider (MSP)

It's not just terrorism; it's any rare risk in the news. The big fear in Canada right now, following a particularly gruesome incident, is random decapitations on intercity buses. In the US, fears of school shootings are much greater than the actual risks. In the UK, it's child predators. And people all over the world mistakenly fear flying more than driving. But the very definition of news is something that hardly ever happens. If an incident is in the news, we shouldn't worry about it. It's when something is so common that its no longer news - car crashes, domestic violence - that we should worry. But that's not the way people think.

Psychologically, this makes sense. We are a species of storytellers. We have good imaginations and we respond more emotionally to stories than to data. We also judge the probability of something by how easy it is to imagine, so stories that are in the news feel more probable - and ominous - than stories that are not. As a result, we overreact to the rare risks we hear stories about, and fear specific plots more than general threats.

The problem with building security around specific targets and tactics is that its only effective if we happen to guess the plot correctly. If we spend billions defending the Underground and terrorists bomb a school instead, we've wasted our money. If we focus on the World Cup and terrorists attack Wimbledon, we've wasted our money.

It's this fetish-like focus on tactics that results in the security follies at airports. We ban guns and knives, and terrorists use box-cutters. We take away box-cutters and corkscrews, so they put explosives in their shoes. We screen shoes, so they use liquids. We take away liquids, and they're going to do something else. Or they'll ignore airplanes entirely and attack a school, church, theatre, stadium, shopping mall, airport terminal outside the security area, or any of the other places where people pack together tightly.

These are stupid games, so let's stop playing. Some high-profile targets deserve special attention and some tactics are worse than others. Airplanes are particularly important targets because they are national symbols and because a small bomb can kill everyone aboard. Seats of government are also symbolic, and therefore attractive, targets. But targets and tactics are interchangeable.

The following three things are true about terrorism. One, the number of potential terrorist targets is infinite. Two, the odds of the terrorists going after any one target is zero. And three, the cost to the terrorist of switching targets is zero.

We need to defend against the broad threat of terrorism, not against specific movie plots. Security is most effective when it doesn't require us to guess. We need to focus resources on intelligence and investigation: identifying terrorists, cutting off their funding and stopping them regardless of what their plans are. We need to focus resources on emergency response: lessening the impact of a terrorist attack, regardless of what it is. And we need to face the geopolitical consequences of our foreign policy.

In 2006, UK police arrested the liquid bombers not through diligent airport security, but through intelligence and investigation. It didn't matter what the bombers' target was. It didn't matter what their tactic was. They would have been arrested regardless. That's smart security. Now we confiscate liquids at airports, just in case another group happens to attack the exact same target in exactly the same way. That's just illogical.

This essay originally appeared in The Guardian. Nothing I haven't already said elsewhere.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.   I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

1. Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.  There will be no physical access to the server. No exceptions.
2. Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
3. Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
4. Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
5. Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,  10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
6. Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
7. If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
8. It should send an email to one or more email addresses with a message, for example:  “New Bot Detected - Confidence 8″ with IP address, etc. in the message.  Another example would be an email, “Bot Blocked” - with details, etc.
9. You cannot automatically block any traffic that is not a bot.  Blocking one “non-bot” results in failure, no exceptions.
10. The Prize:  The winner will get their logo (w/link) on this site in a block called “Bot Hunter Winner” (or something like that.)

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!  Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.   So, you will need to work with both long and short time windows.  Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

I realize that security is a system level issue and it takes a long time to change things at that level, but what's more concerning to me is that the typical infosec mindset remains the same. Should we be surprised by rampant phishing and fraud? I am frankly surprised the numbers are so low given the opportunities that the attackers have via the glacial pace of security improvements. Its been three years since that list and I could write the same exact one today for SOAP, REST, SOA, Web 2.0 whatever. Maybe the main reason, beyond failure of imagination, why infosec is so far behind developers is that infosec lacks tools. Developers automate everything possible. Security doesn't. The most promising thing about static analysis is not the ability to find everything, its the ability to find many important things in an automated way. Infosec needs to stop giving people fish and teaching people to fish. Look at Fortify's vulncat site which has a Taxonomy of Coding Errors. Fortify's Seven (plus one) pernicious kingdoms are: