So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

The trick to doing this is not to hardcode things like host, port, etc. for your SMTP server when you use System.Net.Mail to send mail. Instead, use the default ctor for SmtpClient as I've done in the code below.

static void Main(string[] args)
{
    // note the use of the MailAddress class
    // this allows me to specify display names as well as email addresses
    MailAddress from = new MailAddress("admin@fabrikam.com", "Fabrikam Website");
    MailAddress to = new MailAddress("mari@fabrikam.com", "Mari Joyce");

    MailMessage msg = new MailMessage(from, to);
    msg.Subject  = "Testing 123";
    msg.Body = "This is only a test!";

    // note use of default ctor
    // this looks in config to figure out how to send mail
    new SmtpClient().Send(msg);
}

What you're telling .NET by using the default ctor for SmtpClient is, "please use my config file to figure out how to send mail". Now you can use the system.net/mailSettings/smtp section in config to specify the details of your mail server, and all of the code in your app that is written to use the default SmtpClient ctor will inherit these settings. Here's an example of what the config on a production server might look like (if you put passwords in your config files, be sure to encrypt those sections):

<configuration>
  <system.net>
    <mailSettings>
      <smtp deliveryMethod="Network">
        <network host="mail.fabrikam.com"
                 port="25"
                 userName="WebsiteMailAccount"
                 password="whatever"/>
      </smtp>
    </mailSettings>
  </system.net>
</configuration>

During development, I use different settings because I don't usually want to deal with the hassle of installing an SMTP server on my development box. Instead, I want email messages delivered as individual files in a directory on my hard drive (I always have a c:\mail directory on my development box for just this purpose):

<configuration>
  <system.net>
    <mailSettings>
      <smtp deliveryMethod="SpecifiedPickupDirectory">
        <specifiedPickupDirectory pickupDirectoryLocation="c:\mail"/>
      </smtp>
    </mailSettings>
  </system.net>
</configuration>

Now when I run the program above, I get a .EML file in my c:\mail directory:

Outlook Express is normally registered as the viewer for .EML files, so double-click the file to view it:

If you've never seen this method of simulating email before, I hope you find it as useful as I have. Happy coding!

We've probably all heard that you can't secure what you can't see and that statement is even more profound when it comes to virtual environments.  This is because it is extremely challenging to see what is going on at a micro vs. macro level within a virtual environments network.  The virtualization vendors such as VMWare and Citrix have provided embedded tools into their management consoles that show a macro level of visibility but its not enough to identify security events in the environment.  Take a look at the attached picture.  It simply shows VMWare's ability to monitor virtual network performance statistics from a bits per second perspective.

<-Click To Enlarge

With only this level of detail how can one determine which network applications are causing spikes.  Is it FTP traffic that is occuring at a high volume at an unuseal time of day?  If that were occuring, could that be indicative of either a breach or some sort of problem? What if FTP isn't even an authorized service in the virtual environment but there is a high volume of it?  Did someone install a rouge FTP service so they could steal information from the server at will?

These types of questions can't really be answered without a micro level of detail into the packets flowing in, out and within the virtual environment.  Now, what I am highlighting is not security in the traditional sense of prevention but using visibility as a means to first identify, then pin point the source of an issue so that it can properly be mitigated.  Having constant visibility can also ensure that other security products in the environment are performing as expected.  What if a Montego HyperSwitch with firewalling enabled is configured with many policies but someone forgot to create an FTP block policy.  One could think they are protected from rouge FTP services transmiting data out of the network, but without constant visibility monitoring, can you be certain?

Some vendors, namely Reflex Security will get you to believe that their IPS / IDS solution that is inline and running in the virtual environment is the right and only approach.  Or they will tell you to hang a virtual IDS off a span port in the virtual environment and you will at least have visibility into the attacks that are taking place.  Well, sure... You now have attack visibility but at the performance cost of your virtual environment.  Signature matching technologies are great, I'm a huge believer; however they don't scale very well in shared computing environments such as virtual ones.  IDS systems also don't typically track protocol and network service (FTP, HTTP, etc.) utilizations; which is another important part of visibility.

So, what do we do to gain visibility without the performance headache?  Well, for starters its probably best to put your IDS/IPS solutions in the physical environment where performance will be less of a concern.  In fact, you can span a virtual switch's traffic out to a physical NIC as easy as you can to a virtual one.  So why do it virtual and have to pay a 60% CPU utilization tax?  Another solution is to IDS inspect only the things you care about.  Why IDS inspect SSL traffic if you know your solution can't unencrypt SSL.  Its just a waste of compute cycles isnt it?  Policy based switching helps you with directing only the things you care about to an IDS (attack visualization product).  Montego's HyperSwitch also can help you with the traffic redirection of only the things you care about.

Another method of visibility which I tend to be a fan of is one of packet analysis (aka NetFlow).  NetFlow was invented by Cisco some time ago and has gained popularity in the physical world and definately has a use in the virtual world.  NetFlow is lightweight.  Let me say that again, its light weight!  It only sends a summation of packet detail to an analytical engine which can do some number crunching, packet comparison, etc. etc. to make some sense out of whats going on.  Lancope, an Atlanta based visibility company that provides Network Visibility, Security Visibility and User Visibility has this tool on their website that is a Netflow Bandwidth calculator.  You'll see from playing with this ( http://www.lancope.com/netflowcalculator.aspx ) calculator that it doesn't consume a lot of network bandwidth to transmit these network accounting records.  It also doesn't cause a lot of CPU overhead to send these records to an analytical engine sitting somewhere in the network.

Lancope's analytical engines have the ability to do the following for you within your virtual environment:

...and probably a slew of other things I'm not aware of.  A screen shot of their product is bellow:

<- Click to enlarge

You'll notice from the screenshot that you are able to visualize who is talking to who, how much traffic they have sent and received and something called a concern index (not seen on this screenshot).

Now, a concern index is a number that increases as Lancopes analytical engines monitor suspicious activity on a session.  A high counter can be indicative of a security problem.  Its another way of identifying (visualizing) compromised hosts (virtual machines) without having to do signature matching like a heavy weight IPS engine.  Example:  Lets say you have a VM that has a BOT on it and is "owned".  The Lancope product is monitoring this long life session.  Let's say that session is established for several hours or maybe even days or months.  Lets also say that the conversation appears to be mostly unidirectional from a public ip address not belonging to your enterprise.  Lancope would increase a the concern index on this since this server hasn't typically had this type of behavior.  Once the concern index reached a certain level it could then fire off an email, send you a text message or something saying:  Warning, Warning, Danger, Danger Will Robinson!!! You're virtual server may be infected with a BOT, please investigate immediately!!!

This example is VISIBILITY which helps you with SECURITY.  There are a number of other things you can do with NetFlow and Lancope products that have less to do with security and more to do with operational efficiencies.  Things like, helping you answer questions of:  How do I know what network applications are taking up the most bandwidth?  When should I move those applications over to a server with more horsepower?  When did these VM's vmotion over here and was there a traffic condition / CPU condition that caused that to occur?  I could go on and on but thats a topic for another blog entry.

So, my suggestion is to take a look at what NetFlow has to offer.  Montego Networks supports NetFlow transmission and Lancope supports NetFlow analytics and with both you can regain what was lost visibility.

I hope this was helpful to you all!

-John Peterson

We've probably all heard that you can't secure what you can't see and that statement is even more profound when it comes to virtual environments.  This is because it is extremely challenging to see what is going on at a micro vs. macro level within a virtual environments network.  The virtualization vendors such as VMWare and Citrix have provided embedded tools into their management consoles that show a macro level of visibility but its not enough to identify security events in the environment.  Take a look at the attached picture.  It simply shows VMWare's ability to monitor virtual network performance statistics from a bits per second perspective.

<-Click To Enlarge

With only this level of detail how can one determine which network applications are causing spikes.  Is it FTP traffic that is occuring at a high volume at an unuseal time of day?  If that were occuring, could that be indicative of either a breach or some sort of problem? What if FTP isn't even an authorized service in the virtual environment but there is a high volume of it?  Did someone install a rouge FTP service so they could steal information from the server at will?

These types of questions can't really be answered without a micro level of detail into the packets flowing in, out and within the virtual environment.  Now, what I am highlighting is not security in the traditional sense of prevention but using visibility as a means to first identify, then pin point the source of an issue so that it can properly be mitigated.  Having constant visibility can also ensure that other security products in the environment are performing as expected.  What if a Montego HyperSwitch with firewalling enabled is configured with many policies but someone forgot to create an FTP block policy.  One could think they are protected from rouge FTP services transmiting data out of the network, but without constant visibility monitoring, can you be certain?

Some vendors, namely Reflex Security will get you to believe that their IPS / IDS solution that is inline and running in the virtual environment is the right and only approach.  Or they will tell you to hang a virtual IDS off a span port in the virtual environment and you will at least have visibility into the attacks that are taking place.  Well, sure... You now have attack visibility but at the performance cost of your virtual environment.  Signature matching technologies are great, I'm a huge believer; however they don't scale very well in shared computing environments such as virtual ones.  IDS systems also don't typically track protocol and network service (FTP, HTTP, etc.) utilizations; which is another important part of visibility.

So, what do we do to gain visibility without the performance headache?  Well, for starters its probably best to put your IDS/IPS solutions in the physical environment where performance will be less of a concern.  In fact, you can span a virtual switch's traffic out to a physical NIC as easy as you can to a virtual one.  So why do it virtual and have to pay a 60% CPU utilization tax?  Another solution is to IDS inspect only the things you care about.  Why IDS inspect SSL traffic if you know your solution can't unencrypt SSL.  Its just a waste of compute cycles isnt it?  Policy based switching helps you with directing only the things you care about to an IDS (attack visualization product).  Montego's HyperSwitch also can help you with the traffic redirection of only the things you care about.

Another method of visibility which I tend to be a fan of is one of packet analysis (aka NetFlow).  NetFlow was invented by Cisco some time ago and has gained popularity in the physical world and definately has a use in the virtual world.  NetFlow is lightweight.  Let me say that again, its light weight!  It only sends a summation of packet detail to an analytical engine which can do some number crunching, packet comparison, etc. etc. to make some sense out of whats going on.  Lancope, an Atlanta based visibility company that provides Network Visibility, Security Visibility and User Visibility has this tool on their website that is a Netflow Bandwidth calculator.  You'll see from playing with this ( http://www.lancope.com/netflowcalculator.aspx ) calculator that it doesn't consume a lot of network bandwidth to transmit these network accounting records.  It also doesn't cause a lot of CPU overhead to send these records to an analytical engine sitting somewhere in the network.

Lancope's analytical engines have the ability to do the following for you within your virtual environment:

...and probably a slew of other things I'm not aware of.  A screen shot of their product is bellow:

<- Click to enlarge

You'll notice from the screenshot that you are able to visualize who is talking to who, how much traffic they have sent and received and something called a concern index (not seen on this screenshot).

Now, a concern index is a number that increases as Lancopes analytical engines monitor suspicious activity on a session.  A high counter can be indicative of a security problem.  Its another way of identifying (visualizing) compromised hosts (virtual machines) without having to do signature matching like a heavy weight IPS engine.  Example:  Lets say you have a VM that has a BOT on it and is "owned".  The Lancope product is monitoring this long life session.  Let's say that session is established for several hours or maybe even days or months.  Lets also say that the conversation appears to be mostly unidirectional from a public ip address not belonging to your enterprise.  Lancope would increase a the concern index on this since this server hasn't typically had this type of behavior.  Once the concern index reached a certain level it could then fire off an email, send you a text message or something saying:  Warning, Warning, Danger, Danger Will Robinson!!! You're virtual server may be infected with a BOT, please investigate immediately!!!

This example is VISIBILITY which helps you with SECURITY.  There are a number of other things you can do with NetFlow and Lancope products that have less to do with security and more to do with operational efficiencies.  Things like, helping you answer questions of:  How do I know what network applications are taking up the most bandwidth?  When should I move those applications over to a server with more horsepower?  When did these VM's vmotion over here and was there a traffic condition / CPU condition that caused that to occur?  I could go on and on but thats a topic for another blog entry.

So, my suggestion is to take a look at what NetFlow has to offer.  Montego Networks supports NetFlow transmission and Lancope supports NetFlow analytics and with both you can regain what was lost visibility.

I hope this was helpful to you all!

-John Peterson

The details of the vulnerability aren't important, but basically it's a form of DNS cache poisoning. The DNS system is what translates domain names people understand, like www.schneier.com, to IP addresses computers understand: 204.11.246.1. There is a whole family of vulnerabilities where the DNS system on your computer is fooled into thinking that the IP address for www.badsite.com is really the IP address for www.goodsite.com -- there's no way for you to tell the difference -- and that allows the criminals at www.badsite.com to trick you into doing all sorts of things, like giving up your bank account details. Kaminsky discovered a particularly nasty variant of this cache-poisoning attack.

Here's the way the timeline was supposed to work: Kaminsky discovered the vulnerability about six months ago, and quietly worked with vendors to patch it. (There's a fairly straightforward fix, although the implementation nuances are complicated.) Of course, this meant describing the vulnerability to them; why would companies like Microsoft and Cisco believe him otherwise? On July 8, he held a press conference to announce the vulnerability -- but not the details -- and reveal that a patch was available from a long list of vendors. We would all have a month to patch, and Kaminsky would release details of the vulnerability at the BlackHat conference early next month.

Of course, the details leaked. How isn't important; it could have leaked a zillion different ways. Too many people knew about it for it to remain secret. Others who knew the general idea were too smart not to speculate on the details. I'm kind of amazed the details remained secret for this long; undoubtedly it had leaked into the underground community before the public leak two days ago. So now everyone who back-burnered the problem is rushing to patch, while the hacker community is racing to produce working exploits.

What's the moral here? It's easy to condemn Kaminsky: If he had shut up about the problem, we wouldn't be in this mess. But that's just wrong. Kaminsky found the vulnerability by accident. There's no reason to believe he was the first one to find it, and it's ridiculous to believe he would be the last. Don't shoot the messenger. The problem is with the DNS protocol; it's insecure.

The real lesson is that the patch treadmill doesn't work, and it hasn't for years. This cycle of finding security holes and rushing to patch them before the bad guys exploit those vulnerabilities is expensive, inefficient and incomplete. We need to design security into our systems right from the beginning. We need assurance. We need security engineers involved in system design. This process won't prevent every vulnerability, but it's much more secure -- and cheaper -- than the patch treadmill we're all on now.

What a security engineer brings to the problem is a particular mindset. He thinks about systems from a security perspective. It's not that he discovers all possible attacks before the bad guys do; it's more that he anticipates potential types of attacks, and defends against them even if he doesn't know their details. I see this all the time in good cryptographic designs. It's over-engineering based on intuition, but if the security engineer has good intuition, it generally works.

Kaminsky's vulnerability is a perfect example of this. Years ago, cryptographer Daniel J. Bernstein looked at DNS security and decided that Source Port Randomization was a smart design choice. That's exactly the work-around being rolled out now following Kaminsky's discovery. Bernstein didn't discover Kaminsky's attack; instead, he saw a general class of attacks and realized that this enhancement could protect against them. Consequently, the DNS program he wrote in 2000, djbdns, doesn't need to be patched; it's already immune to Kaminsky's attack.

That's what a good design looks like. It's not just secure against known attacks; it's also secure against unknown attacks. We need more of this, not just on the internet but in voting machines, ID cards, transportation payment cards ... everywhere. Stop assuming that systems are secure unless demonstrated insecure; start assuming that systems are insecure unless designed securely.

This essay previously appeared on Wired.com.

The details of the vulnerability aren't important, but basically it's a form of DNS cache poisoning. The DNS system is what translates domain names people understand, like www.schneier.com, to IP addresses computers understand: 204.11.246.1. There is a whole family of vulnerabilities where the DNS system on your computer is fooled into thinking that the IP address for www.badsite.com is really the IP address for www.goodsite.com -- there's no way for you to tell the difference -- and that allows the criminals at www.badsite.com to trick you into doing all sorts of things, like giving up your bank account details. Kaminsky discovered a particularly nasty variant of this cache-poisoning attack.

Here's the way the timeline was supposed to work: Kaminsky discovered the vulnerability about six months ago, and quietly worked with vendors to patch it. (There's a fairly straightforward fix, although the implementation nuances are complicated.) Of course, this meant describing the vulnerability to them; why would companies like Microsoft and Cisco believe him otherwise? On July 8, he held a press conference to announce the vulnerability -- but not the details -- and reveal that a patch was available from a long list of vendors. We would all have a month to patch, and Kaminsky would release details of the vulnerability at the BlackHat conference early next month.

Of course, the details leaked. How isn't important; it could have leaked a zillion different ways. Too many people knew about it for it to remain secret. Others who knew the general idea were too smart not to speculate on the details. I'm kind of amazed the details remained secret for this long; undoubtedly it had leaked into the underground community before the public leak two days ago. So now everyone who back-burnered the problem is rushing to patch, while the hacker community is racing to produce working exploits.

What's the moral here? It's easy to condemn Kaminsky: If he had shut up about the problem, we wouldn't be in this mess. But that's just wrong. Kaminsky found the vulnerability by accident. There's no reason to believe he was the first one to find it, and it's ridiculous to believe he would be the last. Don't shoot the messenger. The problem is with the DNS protocol; it's insecure.

The real lesson is that the patch treadmill doesn't work, and it hasn't for years. This cycle of finding security holes and rushing to patch them before the bad guys exploit those vulnerabilities is expensive, inefficient and incomplete. We need to design security into our systems right from the beginning. We need assurance. We need security engineers involved in system design. This process won't prevent every vulnerability, but it's much more secure -- and cheaper -- than the patch treadmill we're all on now.

What a security engineer brings to the problem is a particular mindset. He thinks about systems from a security perspective. It's not that he discovers all possible attacks before the bad guys do; it's more that he anticipates potential types of attacks, and defends against them even if he doesn't know their details. I see this all the time in good cryptographic designs. It's over-engineering based on intuition, but if the security engineer has good intuition, it generally works.

Kaminsky's vulnerability is a perfect example of this. Years ago, cryptographer Daniel J. Bernstein looked at DNS security and decided that Source Port Randomization was a smart design choice. That's exactly the work-around being rolled out now following Kaminsky's discovery. Bernstein didn't discover Kaminsky's attack; instead, he saw a general class of attacks and realized that this enhancement could protect against them. Consequently, the DNS program he wrote in 2000, djbdns, doesn't need to be patched; it's already immune to Kaminsky's attack.

That's what a good design looks like. It's not just secure against known attacks; it's also secure against unknown attacks. We need more of this, not just on the internet but in voting machines, ID cards, transportation payment cards ... everywhere. Stop assuming that systems are secure unless demonstrated insecure; start assuming that systems are insecure unless designed securely.

---

Bruce Schneier is chief security technology officer of BT, and author of Beyond Fear: Thinking Sensibly About Security in an Uncertain World.