Learn how file integrity monitoring solutions work and the capabilities you should expect your solution to have. Then review a detailed checklist you should complete before purchasing your solution. Finally, discover how Tripwire Enterprise effectively combines file integrity monitoring with configuration assessment-a single configuration control solution that proactively assesses and monitors the IT infrastructure and enables organizations to achieve and maintain compliance with standards and regulations.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.   I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

1. Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.  There will be no physical access to the server. No exceptions.
2. Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
3. Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
4. Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
5. Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,  10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
6. Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
7. If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
8. It should send an email to one or more email addresses with a message, for example:  “New Bot Detected - Confidence 8″ with IP address, etc. in the message.  Another example would be an email, “Bot Blocked” - with details, etc.
9. You cannot automatically block any traffic that is not a bot.  Blocking one “non-bot” results in failure, no exceptions.
10. The Prize:  The winner will get their logo (w/link) on this site in a block called “Bot Hunter Winner” (or something like that.)

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!  Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.   So, you will need to work with both long and short time windows.  Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

Alert – do not upgrade to Virtual Infrastructure 3.5 Update 2.

Apparently there’s some problem with the license expiration time and the workaround suggested by a Virtualization.Info reader is to set the date back to August 10 – which of course messes up your logs and any monitoring that you may be doing. No immediate solution forthcoming from VMware and in fact, good luck getting in touch with the company.

“At the moment it seems that the entire VMware Knowledge Base collapsed. Calling the support line customers can just receive a brief message saying that the problem will be solved within 36 hours.
Additionally, VMware removed the capability to download any affected product.”

It just so happens that I have been experimenting with Amazon Elastic Computing Services (EC2), documented in Computing in the Clouds with AWS over at The CEP Blog.  The server over at The UNIX and Linux Forums has been experiencing some very hardware-limited, high load averages recently. We thought we should take a look at moving the forum server up to the clouds.   

Then, a fellow system admin over at the forums suggested that maybe some rogue bots were causing high server loads; so I wrote a one-line command to do a bit of real-time spider hunting in the Apache2 logfiles.  Surprise!  I found there were a number of rogue, hungry spiders that would not follow our robots.txt directive not to crawl the site.   One of the bots was from Russia, one was from China, and another one was from Korea.  There were spiders from places I never heard of, all consuming precious  resources and denying our users!

So, I did what any Linux admin would do. I used iptables to block the networks of these rogue, hungry, spiders (sorry I was not very kind to these cyber creatures).  It probally comes to no surprise at this point in the story that four of the spiders were from the Amazon EC2 cloud.  Here is a sample of the output from iptables -L:

root@www:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all — ec2-67-202-45-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-243-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-197-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-213-0.compute-1.amazonaws.com/24

Well, imagine a not-so-distant future dystopian world where criminals or terrorists want to launch a massive denial-of-service attack against some critical infrastructure, like the root DNS servers, or an attack against major financial institutions, military or e-commerce sites.   

First, the bad guys create an instance of powerful operating system with a malicious network application, they test it, and they place it the cloud (without invoking the instance, paying a very small storage fee, no computing time fee) and they wait.   Then, at the precise moment of their planned attack, they launch 128 instances each with the equivalence of whatever is the mega-platform at the time, and just blast away at their attack target(s).    Even more damaging, they do this from many cloud computing infrastructures.  (Note: The cost of the attack is minimal because the criminals are only charged a few pennies an hour for each running instance and the attack runs an hour or two.)

My experience with cloud computing, which is still maturing, is that cloud computing has great promise for both good and evil.  The very real example of the “spiders from the clouds” is a harmless enough story of folks using a cloud computing infrastructure for web crawling, perhaps hoping to be the next Google billionaires. 

One the other hand, cloud computing brings with it an emerging and growing danger for the misuse of the power of cloud computing infrastructures.   The misuse could be malicious, or accidental, but never-the-less, the danger is real.

What an interesting world we have created!  Would would have ever dreamed 10 years ago that we could be attacked by ……

#include <horror_movie_sounds.mp3>

…. Spiders from the Clouds.

Reprinted by permission from The Attack of the Spiders from the Clouds by Tim Bass, CISSP

Blogger: Randall Gamby

Across the different security technology presentations given this week at Catalyst, one common theme has been the important role of policy. As people hear about new and better technologies and how they can be integrated into their existing infrastructures, they should take the time to examine their policies to make sure they keep up with the solutions being considered.  Questions to ask:

• When did we review our policies last?
• Do we have not enough or too many?
• Will they still be valid?
• Are there other influencers on them?

But while changes will most likely be needed for many current policies, a question that often isn’t asked is, “Are they enforceable?”  As enterprises create policies based upon what users “should do,” can the security team validate that they “did do” what was asked?  For example, a common policy is, “All sensitive data at rest must be encrypted.”  So this means you must encrypt your Active Directory, your e-mail storage, every production database, yes? That's probably not happening.  So if the enterprise has no way to implement the policy, then it ultimately is not a valid policy and needs to either be modified or the enterprise needs money, resources and time to conform to the policy. 

The social effect on the user population also needs to be considered.  Essentially, the enterprise is teaching users that they don’t have to conform to this policy, so maybe they don’t have to be conformant to others on the books.  Not a good lesson to teach them.

So as the Catalyst attendees go back with “dreams of technology sugar plums dancing in their heads” don’t forget that good governance with valid processes should be skipping around the edge.

Blogger: Randall Gamby

Across the different security technology presentations given this week at Catalyst, one common theme has been the important role of policy. As people hear about new and better technologies and how they can be integrated into their existing infrastructures, they should take the time to examine their policies to make sure they keep up with the solutions being considered.  Questions to ask:

• When did we review our policies last?
• Do we have not enough or too many?
• Will they still be valid?
• Are there other influencers on them?

But while changes will most likely be needed for many current policies, a question that often isn???t asked is, ???Are they enforceable????  As enterprises create policies based upon what users ???should do,??? can the security team validate that they ???did do??? what was asked?  For example, a common policy is, ???All sensitive data at rest must be encrypted.???  So this means you must encrypt your Active Directory, your e-mail storage, every production database, yes? That's probably not happening.  So if the enterprise has no way to implement the policy, then it ultimately is not a valid policy and needs to either be modified or the enterprise needs money, resources and time to conform to the policy. 

The social effect on the user population also needs to be considered.  Essentially, the enterprise is teaching users that they don???t have to conform to this policy, so maybe they don???t have to be conformant to others on the books.  Not a good lesson to teach them.

So as the Catalyst attendees go back with ???dreams of technology sugar plums dancing in their heads??? don???t forget that good governance with valid processes should be skipping around the edge.