It just so happens that I have been experimenting with Amazon Elastic Computing Services (EC2), documented in Computing in the Clouds with AWS over at The CEP Blog.  The server over at The UNIX and Linux Forums has been experiencing some very hardware-limited, high load averages recently. We thought we should take a look at moving the forum server up to the clouds.   

Then, a fellow system admin over at the forums suggested that maybe some rogue bots were causing high server loads; so I wrote a one-line command to do a bit of real-time spider hunting in the Apache2 logfiles.  Surprise!  I found there were a number of rogue, hungry spiders that would not follow our robots.txt directive not to crawl the site.   One of the bots was from Russia, one was from China, and another one was from Korea.  There were spiders from places I never heard of, all consuming precious  resources and denying our users!

So, I did what any Linux admin would do. I used iptables to block the networks of these rogue, hungry, spiders (sorry I was not very kind to these cyber creatures).  It probally comes to no surprise at this point in the story that four of the spiders were from the Amazon EC2 cloud.  Here is a sample of the output from iptables -L:

root@www:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all — ec2-67-202-45-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-243-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-197-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-213-0.compute-1.amazonaws.com/24

Well, imagine a not-so-distant future dystopian world where criminals or terrorists want to launch a massive denial-of-service attack against some critical infrastructure, like the root DNS servers, or an attack against major financial institutions, military or e-commerce sites.   

First, the bad guys create an instance of powerful operating system with a malicious network application, they test it, and they place it the cloud (without invoking the instance, paying a very small storage fee, no computing time fee) and they wait.   Then, at the precise moment of their planned attack, they launch 128 instances each with the equivalence of whatever is the mega-platform at the time, and just blast away at their attack target(s).    Even more damaging, they do this from many cloud computing infrastructures.  (Note: The cost of the attack is minimal because the criminals are only charged a few pennies an hour for each running instance and the attack runs an hour or two.)

My experience with cloud computing, which is still maturing, is that cloud computing has great promise for both good and evil.  The very real example of the “spiders from the clouds” is a harmless enough story of folks using a cloud computing infrastructure for web crawling, perhaps hoping to be the next Google billionaires. 

One the other hand, cloud computing brings with it an emerging and growing danger for the misuse of the power of cloud computing infrastructures.   The misuse could be malicious, or accidental, but never-the-less, the danger is real.

What an interesting world we have created!  Would would have ever dreamed 10 years ago that we could be attacked by ……

#include <horror_movie_sounds.mp3>

…. Spiders from the Clouds.

Reprinted by permission from The Attack of the Spiders from the Clouds by Tim Bass, CISSP

We've probably all heard that you can't secure what you can't see and that statement is even more profound when it comes to virtual environments.  This is because it is extremely challenging to see what is going on at a micro vs. macro level within a virtual environments network.  The virtualization vendors such as VMWare and Citrix have provided embedded tools into their management consoles that show a macro level of visibility but its not enough to identify security events in the environment.  Take a look at the attached picture.  It simply shows VMWare's ability to monitor virtual network performance statistics from a bits per second perspective.

<-Click To Enlarge

With only this level of detail how can one determine which network applications are causing spikes.  Is it FTP traffic that is occuring at a high volume at an unuseal time of day?  If that were occuring, could that be indicative of either a breach or some sort of problem? What if FTP isn't even an authorized service in the virtual environment but there is a high volume of it?  Did someone install a rouge FTP service so they could steal information from the server at will?

These types of questions can't really be answered without a micro level of detail into the packets flowing in, out and within the virtual environment.  Now, what I am highlighting is not security in the traditional sense of prevention but using visibility as a means to first identify, then pin point the source of an issue so that it can properly be mitigated.  Having constant visibility can also ensure that other security products in the environment are performing as expected.  What if a Montego HyperSwitch with firewalling enabled is configured with many policies but someone forgot to create an FTP block policy.  One could think they are protected from rouge FTP services transmiting data out of the network, but without constant visibility monitoring, can you be certain?

Some vendors, namely Reflex Security will get you to believe that their IPS / IDS solution that is inline and running in the virtual environment is the right and only approach.  Or they will tell you to hang a virtual IDS off a span port in the virtual environment and you will at least have visibility into the attacks that are taking place.  Well, sure... You now have attack visibility but at the performance cost of your virtual environment.  Signature matching technologies are great, I'm a huge believer; however they don't scale very well in shared computing environments such as virtual ones.  IDS systems also don't typically track protocol and network service (FTP, HTTP, etc.) utilizations; which is another important part of visibility.

So, what do we do to gain visibility without the performance headache?  Well, for starters its probably best to put your IDS/IPS solutions in the physical environment where performance will be less of a concern.  In fact, you can span a virtual switch's traffic out to a physical NIC as easy as you can to a virtual one.  So why do it virtual and have to pay a 60% CPU utilization tax?  Another solution is to IDS inspect only the things you care about.  Why IDS inspect SSL traffic if you know your solution can't unencrypt SSL.  Its just a waste of compute cycles isnt it?  Policy based switching helps you with directing only the things you care about to an IDS (attack visualization product).  Montego's HyperSwitch also can help you with the traffic redirection of only the things you care about.

Another method of visibility which I tend to be a fan of is one of packet analysis (aka NetFlow).  NetFlow was invented by Cisco some time ago and has gained popularity in the physical world and definately has a use in the virtual world.  NetFlow is lightweight.  Let me say that again, its light weight!  It only sends a summation of packet detail to an analytical engine which can do some number crunching, packet comparison, etc. etc. to make some sense out of whats going on.  Lancope, an Atlanta based visibility company that provides Network Visibility, Security Visibility and User Visibility has this tool on their website that is a Netflow Bandwidth calculator.  You'll see from playing with this ( http://www.lancope.com/netflowcalculator.aspx ) calculator that it doesn't consume a lot of network bandwidth to transmit these network accounting records.  It also doesn't cause a lot of CPU overhead to send these records to an analytical engine sitting somewhere in the network.

Lancope's analytical engines have the ability to do the following for you within your virtual environment:

...and probably a slew of other things I'm not aware of.  A screen shot of their product is bellow:

<- Click to enlarge

You'll notice from the screenshot that you are able to visualize who is talking to who, how much traffic they have sent and received and something called a concern index (not seen on this screenshot).

Now, a concern index is a number that increases as Lancopes analytical engines monitor suspicious activity on a session.  A high counter can be indicative of a security problem.  Its another way of identifying (visualizing) compromised hosts (virtual machines) without having to do signature matching like a heavy weight IPS engine.  Example:  Lets say you have a VM that has a BOT on it and is "owned".  The Lancope product is monitoring this long life session.  Let's say that session is established for several hours or maybe even days or months.  Lets also say that the conversation appears to be mostly unidirectional from a public ip address not belonging to your enterprise.  Lancope would increase a the concern index on this since this server hasn't typically had this type of behavior.  Once the concern index reached a certain level it could then fire off an email, send you a text message or something saying:  Warning, Warning, Danger, Danger Will Robinson!!! You're virtual server may be infected with a BOT, please investigate immediately!!!

This example is VISIBILITY which helps you with SECURITY.  There are a number of other things you can do with NetFlow and Lancope products that have less to do with security and more to do with operational efficiencies.  Things like, helping you answer questions of:  How do I know what network applications are taking up the most bandwidth?  When should I move those applications over to a server with more horsepower?  When did these VM's vmotion over here and was there a traffic condition / CPU condition that caused that to occur?  I could go on and on but thats a topic for another blog entry.

So, my suggestion is to take a look at what NetFlow has to offer.  Montego Networks supports NetFlow transmission and Lancope supports NetFlow analytics and with both you can regain what was lost visibility.

I hope this was helpful to you all!

-John Peterson

We've probably all heard that you can't secure what you can't see and that statement is even more profound when it comes to virtual environments.  This is because it is extremely challenging to see what is going on at a micro vs. macro level within a virtual environments network.  The virtualization vendors such as VMWare and Citrix have provided embedded tools into their management consoles that show a macro level of visibility but its not enough to identify security events in the environment.  Take a look at the attached picture.  It simply shows VMWare's ability to monitor virtual network performance statistics from a bits per second perspective.

<-Click To Enlarge

With only this level of detail how can one determine which network applications are causing spikes.  Is it FTP traffic that is occuring at a high volume at an unuseal time of day?  If that were occuring, could that be indicative of either a breach or some sort of problem? What if FTP isn't even an authorized service in the virtual environment but there is a high volume of it?  Did someone install a rouge FTP service so they could steal information from the server at will?

These types of questions can't really be answered without a micro level of detail into the packets flowing in, out and within the virtual environment.  Now, what I am highlighting is not security in the traditional sense of prevention but using visibility as a means to first identify, then pin point the source of an issue so that it can properly be mitigated.  Having constant visibility can also ensure that other security products in the environment are performing as expected.  What if a Montego HyperSwitch with firewalling enabled is configured with many policies but someone forgot to create an FTP block policy.  One could think they are protected from rouge FTP services transmiting data out of the network, but without constant visibility monitoring, can you be certain?

Some vendors, namely Reflex Security will get you to believe that their IPS / IDS solution that is inline and running in the virtual environment is the right and only approach.  Or they will tell you to hang a virtual IDS off a span port in the virtual environment and you will at least have visibility into the attacks that are taking place.  Well, sure... You now have attack visibility but at the performance cost of your virtual environment.  Signature matching technologies are great, I'm a huge believer; however they don't scale very well in shared computing environments such as virtual ones.  IDS systems also don't typically track protocol and network service (FTP, HTTP, etc.) utilizations; which is another important part of visibility.

So, what do we do to gain visibility without the performance headache?  Well, for starters its probably best to put your IDS/IPS solutions in the physical environment where performance will be less of a concern.  In fact, you can span a virtual switch's traffic out to a physical NIC as easy as you can to a virtual one.  So why do it virtual and have to pay a 60% CPU utilization tax?  Another solution is to IDS inspect only the things you care about.  Why IDS inspect SSL traffic if you know your solution can't unencrypt SSL.  Its just a waste of compute cycles isnt it?  Policy based switching helps you with directing only the things you care about to an IDS (attack visualization product).  Montego's HyperSwitch also can help you with the traffic redirection of only the things you care about.

Another method of visibility which I tend to be a fan of is one of packet analysis (aka NetFlow).  NetFlow was invented by Cisco some time ago and has gained popularity in the physical world and definately has a use in the virtual world.  NetFlow is lightweight.  Let me say that again, its light weight!  It only sends a summation of packet detail to an analytical engine which can do some number crunching, packet comparison, etc. etc. to make some sense out of whats going on.  Lancope, an Atlanta based visibility company that provides Network Visibility, Security Visibility and User Visibility has this tool on their website that is a Netflow Bandwidth calculator.  You'll see from playing with this ( http://www.lancope.com/netflowcalculator.aspx ) calculator that it doesn't consume a lot of network bandwidth to transmit these network accounting records.  It also doesn't cause a lot of CPU overhead to send these records to an analytical engine sitting somewhere in the network.

Lancope's analytical engines have the ability to do the following for you within your virtual environment:

...and probably a slew of other things I'm not aware of.  A screen shot of their product is bellow:

<- Click to enlarge

You'll notice from the screenshot that you are able to visualize who is talking to who, how much traffic they have sent and received and something called a concern index (not seen on this screenshot).

Now, a concern index is a number that increases as Lancopes analytical engines monitor suspicious activity on a session.  A high counter can be indicative of a security problem.  Its another way of identifying (visualizing) compromised hosts (virtual machines) without having to do signature matching like a heavy weight IPS engine.  Example:  Lets say you have a VM that has a BOT on it and is "owned".  The Lancope product is monitoring this long life session.  Let's say that session is established for several hours or maybe even days or months.  Lets also say that the conversation appears to be mostly unidirectional from a public ip address not belonging to your enterprise.  Lancope would increase a the concern index on this since this server hasn't typically had this type of behavior.  Once the concern index reached a certain level it could then fire off an email, send you a text message or something saying:  Warning, Warning, Danger, Danger Will Robinson!!! You're virtual server may be infected with a BOT, please investigate immediately!!!

This example is VISIBILITY which helps you with SECURITY.  There are a number of other things you can do with NetFlow and Lancope products that have less to do with security and more to do with operational efficiencies.  Things like, helping you answer questions of:  How do I know what network applications are taking up the most bandwidth?  When should I move those applications over to a server with more horsepower?  When did these VM's vmotion over here and was there a traffic condition / CPU condition that caused that to occur?  I could go on and on but thats a topic for another blog entry.

So, my suggestion is to take a look at what NetFlow has to offer.  Montego Networks supports NetFlow transmission and Lancope supports NetFlow analytics and with both you can regain what was lost visibility.

I hope this was helpful to you all!

-John Peterson

With all this talk and reporting about security concerns, lets change the channel for a moment and assess the potential security benefits of Cloud Computing.

In my view, there are some strong technical security arguments in favour of Cloud Computing - assuming we can find ways to manage the risks.

With this new paradigm come challenges and opportunities.  The challenges are getting plenty of attention - I’m regularly afforded the opportunity to comment on them, plus obviously I cover them on this blog.  However, lets not lose sight of the potential upside.

In this post, I walk through seven technical security benefits.  Some are immediate, others may arise over time and have conditions attached (some unstated for the sake of brevity).  However, I’m including the longer-range benefits now to raise awareness.  Some of the outcomes listed are available today without the Cloud, but they are either complex and slow to implement (and thus less likely to happen) or prohibitive for capital cost reasons.  I don’t claim this is a definitive list - it reflects where my thinking is today.

Some benefits depend on the Cloud service used and therefore do not apply across the board.  For example; I see no solid forensic benefits with SaaS.  Also, for space reasons, I’m purposely not including the ‘flip side’ to these benefits, however if you read this blog regularly you should recognise some.

On a sidenote, I believe the Cloud offers Small and Medium Businesses major potential security benefits.  Frequently SMBs struggle with limited or non-existent in-house INFOSEC resources and budgets.  The caveat is that the Cloud market is still very new - security offerings are somewhat foggy - making selection tricky.  Clearly, not all Cloud providers will offer the same security.

Seven Technical Security Benefits of the Cloud

1. Centralised Data

• Reduced Data Leakage: this is the benefit I hear most from Cloud providers - and in my view they are right.  How many laptops do we need to lose before we get this?  How many backup tapes?  The data “landmines” of today could be greatly reduced by the Cloud as thin client technology becomes prevalent.  Small, temporary caches on handheld devices or Netbook computers pose less risk than transporting data buckets in the form of laptops.  Ask the CISO of any large company if all laptops have company ‘mandated’ controls consistently applied; e.g. full disk encryption.  You’ll see the answer by looking at the whites of their eyes.  Despite best efforts around asset management and endpoint security we continue to see embarrassing and disturbing misses.  And what about SMBs?  How many use encryption for sensitive data, or even have a data classification policy in place?
• Monitoring benefits: central storage is easier to control and monitor.  The flipside is the nightmare scenario of comprehensive data theft.  However, I would rather spend my time as a security professional figuring out smart ways to protect and monitor access to data stored in one place (with the benefit of situational advantage) than trying to figure out all the places where the company data resides across a myriad of thick clients!  You can get the benefits of Thin Clients today but Cloud Storage provides a way to centralise the data faster and potentially cheaper.  The logistical challenge today is getting Terabytes of data to the Cloud in the first place.

2. Incident Response / Forensics

• Forensic readiness: with Infrastructure as a Service (IaaS) providers, I can build a dedicated forensic server in the same Cloud as my company and place it offline, ready for use when needed.  I would only need pay for storage until an incident happens and I need to bring it online.  I don’t need to call someone to bring it online or install some kind of remote boot software - I just click a button in the Cloud Providers web interface.  If I have multiple incident responders, I can give them a copy of the VM so we can distribute the forensic workload based on the job at hand or as new sources of evidence arise and need analysis.  To fully realise this benefit, commercial forensic software vendors would need to move away from archaic, physical dongle based licensing schemes to a network licensing model.
• Decrease evidence acquisition time: if a server in the Cloud gets compromised (i.e. broken into), I can now clone that server at the click of a mouse and make the cloned disks instantly available to my Cloud Forensics server.  I didn’t need to “find” storage or have it “ready, waiting and unused” - its just there.
• Eliminate or reduce service downtime: Note that in the above scenario I didn’t have to go tell the COO that the system needs to be taken offline for hours whilst I dig around in the RAID Array hoping that my physical acqusition toolkit is compatible (and that the version of RAID firmware isn’t supported by my forensic software).  Abstracting the hardware removes a barrier to even doing forensics in some situations.
• Decrease evidence transfer time: In the same Cloud, bit fot bit copies are super fast - made faster by that replicated, distributed filesystem my Cloud provider engineered for me.  From a network traffic perspective, it may even be free to make the copy in the same Cloud.  Without the Cloud, I would have to a lot of time consuming and expensive provisioning of physical devices.  I only pay for the storage as long as I need the evidence.
• Eliminate forensic image verification time: Some Cloud Storage implementations expose a cryptographic checksum or hash.  For example, Amazon S3 generates an MD5 hash automagically when you store an object.  In theory you no longer need to generate time-consuming MD5 checksums using external tools - its already there.
• Decrease time to access protected documents: Immense CPU power opens some doors.  Did the suspect password protect a document that is relevant to the investigation?  You can now test a wider range of candidate passwords in less time to speed investigations.

3. Password assurance testing (aka cracking)

• Decrease password cracking time: if your organisation regularly tests password strength by running password crackers you can use Cloud Compute to decrease crack time and you only pay for what you use.  Ironically, your cracking costs go up as people choose better passwords ;-).
• Keep cracking activities to dedicated machines: if today you use a distributed password cracker to spread the load across non-production machines, you can now put those agents in dedicated Compute instances - and thus stop mixing sensitive credentials with other workloads.

4. Logging

• “Unlimited”, pay per drink storage: logging is often an afterthought, consequently insufficient disk space is allocated and logging is either non-existant or minimal.  Cloud Storage changes all this - no more ‘guessing’ how much storage you need for standard logs.
• Improve log indexing and search: with your logs in the Cloud you can leverage Cloud Compute to index those logs in real-time and get the benefit of instant search results. What is different here?  The Compute instances can be plumbed in and scale as needed based on the logging load - meaning a true real-time view.
• Getting compliant with Extended logging: most modern operating systems offer extended logging in the form of a C2 audit trail.  This is rarely enabled for fear of performance degradation and log size.  Now you can ‘opt-in’ easily - if you are willing to pay for the enhanced logging, you can do so.  Granular logging makes compliance and investigations easier.

5. Improve the state of security software (performance)

• Drive vendors to create more efficient security software: Billable CPU cycles get noticed.  More attention will be paid to inefficient processes; e.g. poorly tuned security agents.  Process accounting will make a comeback as customers target ‘expensive’ processes.  Security vendors that understand how to squeeze the most performance from their software will win.

6. Secure builds

• Pre-hardened, change control builds: this is primarily a benefit of virtualization based Cloud Computing.  Now you get a chance to start ’secure’ (by your own definition) - you create your Gold Image VM and clone away.  There are ways to do this today with bare-metal OS installs but frequently these require additional 3rd party tools, are time consuming to clone or add yet another agent to each endpoint.
• Reduce exposure through patching offline: Gold images can be kept up securely kept up to date.  Offline VMs can be conveniently patched “off” the network.
• Easier to test impact of security changes: this is a big one.  Spin up a copy of your production environment, implement a security change and test the impact at low cost, with minimal startup time.  This is a big deal and removes a major barrier to ‘doing’ security in production environments.

7. Security Testing

• Reduce cost of testing security: a SaaS provider only passes on a portion of their security testing costs.  By sharing the same application as a service, you don’t foot the expensive security code review and/or penetration test.  Even with Platform as a Service (PaaS) where your developers get to write code, there are potential cost economies of scale (particularly around use of code scanning tools that sweep source code for security weaknesses).

Your Thoughts?

What benefits do you see that I haven’t included in the above list?  Where do you agree/disagree and importantly, why?

1. What product(s) & version(s) should I use?
2. How much should I plan to spend?

The simplest answer of course is “it depends”. I’ve seen implementations range from a thousand bucks to over several million. Ideally, your virtualization project needs & goals should drive your product selection. The bells & whistles you chose will determine your spending.

10 Basic questions that will help you determine product & cost:

1. Will your Virtual Infrastructure (VI) host production Virtual Machines (VM)?
2. What servers do you already have that can be used as hosts (32bit, 64bit, Mem, Disk, Network)?
3. Do you have a need for High Availability (HA)?
4. Do you have the need to manage SLA’s on your VMs?
5. What will a typical VM in your VI look like (OS, Disk, Mem, Network, CPU)?
6. What other IT resources do you have that can be used (SAN, NAS, Switches, etc…)?
7. What level of comfort does your existing staff have with the various IT resources?
8. Do you have existing hardware/software support agreements with Vendors you could leverage?
9. What tools do you already own that are “virtualization aware” and what new tools will you need?
10. How many VM’s do you plan to scale to?

Please, please, please, don’t make the mistake of implementing features that you don’t need and over-engineering just because the product lets you do so.

If you plan it right your product & cost, questions will be answered with no unpleasant surprises.

ShareThis

These days every one wants to be seen as green.  Larry Seltzer over on PC Mag has an interesting story from McAfee Avert Labs that using anti-virus on your computer is green. The reasoning goes that by keeping your computer free of malware, your CPU usage stays lower, thereby using less energy and lowering your carbon footprint.  OK, I get it.  My question is what about all of the extra CPU cycles that some of the bloated endpoint security suites use on all of these machines they are installed on.  I would bet that they far outweigh any energy savings from clean machines. 

I guess in place of wrapping yourself in the flag, the thing to do now is wrap yourself in the green thing. How long will it be until some company hires Al Gore to hawk thier technology. In the meantime I would beware of Jolly Green Giants.