So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

Scrawlr quickly came under fire on the Web Security mailing list for having some pretty major limitations. Billy Hoffman et al have been quick to point out that the tool was designed to address a very specific subset of SQL Injection vulnerability — the type affected by the mass attacks — and is not designed to be a general purpose replacement for existing SQL Injection scanners. Let’s look at the limitations, as outlined on the HP page, one by one.

Limitation: Will only crawl up to 1500 pages

Depends on what they mean by 1500 pages. For example, if I have these links on my front page, is that one URL or three?

• http://www.veracode.com/blog/?p=111&foo=1
• http://www.veracode.com/blog/?p=111&foo=2
• http://www.veracode.com/blog/?p=111&foo=3

Or, does it mean that it will really only crawl 1500 pages total, so if I have the same link 1500 times on the front page, it won’t go any further? Either way, for most smaller websites this is probably fine. If you need more than 1500 you could give it different starting URLs in an attempt to improve coverage. It would be nice to have a clearer definition of what it means to “crawl up to 1500 pages” though.

Limitation: Does not support sites requiring authentication

Well, this will render it useless for the majority of enterprise apps. But there are still a lot of sites out there that don’t require authentication, including some of the ones that got hit during the mass attacks, such as the United Nations, UK government, etc.

Limitation: Does not perform Blind SQL injection

They have taken a lot of flack for this but Billy describes it as a conscious choice:

An early version of the tool checked for blind SQL injection, but the final verison of Scrawlr did not. … The biggest feedback we got from early testing was developers wanted to “see” the vulnerability. Differential analysis is kind of difficult to visualize in a way that is helpful for the average dev, and pulling the table names through blind was too much of a performance issue.

I can sort of understand this rationale. Blind SQL Injection testing is much more susceptible to false positives. As users of any commercial web scanner or source code analyzer will attest, the more time you spend chasing down FPs, the less likely you are to put any faith in future results. It’d be nice if there was a way to toggle Blind SQL Injection testing on and off, though (could be off by default so nobody gets confused).

Limitation: Cannot retrieve database contents

Who cares? Find and fix the vulnerability. Pulling down the entire database “because you can” is a total ego move.

Limitation: Does not support JavaScript or flash parsing

Nobody does this very well anyway, particularly the JavaScript part. Writing a great crawler is probably the hardest part of writing an automated web scanner and it’s one of the biggest differentiators from one product to the next. You’re not going to get that for free.

Limitation: Will not test forms for SQL Injection (POST Parameters)

This is probably the toughest one to swallow. It’s not that difficult to parse out forms from HTML, and form POSTs can represent a major chunk of the attack surface. Granted, the Chinese tool associated with the mass attacks did operate solely on GET requests (i.e. parameters in the query string) so HP can defend this again by saying the tool is really aimed at the sites being targeted by the mass attacks. I think it’s a little short-sighted though; chances are that the mass attacks will evolve and it’s better to be proactive about it than reactive.

Conclusion

It’s tough to bash someone for releasing a free tool. I personally think HP should add an option for enabling Blind SQL Injection testing, and that they should consider supporting POSTs as well as GETs. You’re basically getting a (massively) stripped-down WebInspect for free, so take it for what it is. No single tool is a panacea.

The jury is still out on how effective Scrawlr is against the things it does claim support for. Keep watching the Web Security list; the reviews are filtering in.

photo credit: нσвσ

1. Be A Security Cool Cat: Place penguin stickers on every surface in your cubicle. Stick at least 3 on the dual boot company issued laptop (that hasn’t had a kernel upgrade in 6 months). Use BlackHat stickers for bonus points.
2. Be An Undercover Open Source Evangelist: Unfailingly, recommend open source solutions as more secure. Be sure to quote ‘more eyes, less vulnerabilities’. Recite frequently . Always forward security advisories about commercial products to your boss.
3. Walk the Tech Talk: Learn at Least 10 Bash Keyboard Shortcuts. Treat this as a party trick. Perform rapidly in sequence whenever anyone watches your screen. Giggle and pass the keyboard over and say ‘Your turn!’.
4. Be All Knowing, Jedi Warrior!: Say ‘Trust but verify’ whenever you are asked a question you do not understand. Make it clear in meetings that you trust no-one and “verify” solely through a Google/Secunia search.
5. Impress with a Penetration Test!: Download Metasploit, spend 7 hours modifying the web interface: create custom graphics and hack up the CSS files. Start Metasploit running before you leave for the day. Use Camtasia to capture all screen activity so you can review in the morning. If all went well upload to YouTube and link out via facebook.
6. Practice Defense In Depth’: When you are asked ‘What is the Risk?’, grin inanely and say ‘I’ll tell you after I break out the vulnerability scanners’. Run at least 3 vulnerability scanners to get ‘defense in depth’.
7. Latest *Is* Greatest!: Clipboard stealing attacks are *always* a bigger issue than the CISCO infrastructure with default passwords (how did they get there?!).
8. Educate The Great Unwashed with a Deep Dive Security Awareness Program. Educate end-users about Cross Site Scripting and SQL injection attacks. Don’t invite the outsourced developers - they already know this stuff and have deadlines to meet.
9. Impress Your Peers - Perfect the RFC Shoutout: Pick at least 10 common protocols and learn the associated RFC numbers. Intimidate IT colleagues by shouting out the RFC numbers whenever they mention the protocol.
10. Start A Security Blog: What Can I Say?

Its no secret that over the past year it has been quite fashionable to bash NAC.  It has not lived up to the hype.  It is not the promised silver bullet.  Some companies in the market went belly up.  Yes, yes and true.  But as I have said all along this was I think just the natural evolution of a technology as it matures.  There was no way it could live up to the over hype that it was saddled with.  Those who spoke about it realistically always said it was not the next "great white hope" of security, just another arrow in the quiver. However, the reason that people got excited about NAC was that at a rather simple level it was very easy to describe the problem it was trying to solve.  As it turns out, solving that simple problem takes a rather complex solution, no matter how you slice it.

In the end though what we have seen in the NAC market is textbook hype cycle.  The technology triggers for NAC were unseen before numbers of guests having legitimate reasons to access the network.  The spread of malware not through downloading via the Internet, but by introduction via devices logging on and the need for compliance or otherwise to enforce access policies with the network technologies to make it happen.  With Cisco announcing their Network Admission Control program in December, 2003 and Microsoft announcing NAP that summer (interesting that it would be years before either one was actually available) NAC buzz went through a big bang expansion to the very height of inflated expectations. What goes up, must come down and NAC certainly has been dragged into the trough of disillusionment. However, the inherent appeal of the problems it can solve continue to drive customers and interest.  Now we are seeing real signs of NAC emerging into the slope of enlightenment on the way to the plateau of productivity.

What has got me so optimistic?  It is a variety of things.  Let me list them:

1. Network Computing's 3rd annual NAC survey which while it shows demand is down for NAC from past years, it is still substantial and appears to be deeper if not as wide. It also has several other metrics that show people are being more realistic in what they want to accomplish with NAC and have more confidence that it will work.

2. Forrester's new report that shows that customers think NAC is mature enough to be ready for more wide scale deployments. Remember this is the same Forrester who said that NAC as we know it would fail last year. Has NAC changed so much in a year or has Forrester?

3. That Ebenezer Scrooge of NAC, Mike Rothman, actually admits that maybe we are seeing some progress with less inflated expectations with NAC. What could be next, the NAC Grinch, Richard Stiennon admitting it might be OK as well. Here is my prediction: When Rich's new MSSP can make money offering a managed NAC service, Richard will jump on the NAC bandwagon with bells on.

4. My own observations at Interop, RSA, SANS and other events where I spoke to real live potential customers.  I have personally seen a marked upturn in the amount of real NAC projects that we see coming into both our partners and our sales pipelines. I assume that other NAC products are seeing the same pick up.

All of this is very gratifying to see after the bashing NAC has taken.  Now it is onwards and upwards to the plateau of productivity.   See you there!

Thursday was a little different, I got up early and got a few ‘real’ work things done (you know, those things) before heading off to meet Mike Fratto for a project he’s working on. More on that later.

I made it back to the show around lunch-ish but didn’t stop for lunch yet, since the show floor was closing at 4:00pm- I still had some browsing and chatting to do. Starting around 3:45, I took a ‘Last 15 on the Floor’ series of shots from the expo floor.

At some point Thursday or Wednesday, I did stop by the Security Smackdown challenge they had running- pretty neato- bunch of hackers beatin’ each other down for the ultimate Smackdown Title. WWCF: World Wide Crypto Fighting…. or… something like that. There was a guy sporting an overtly over-sized gold WWF-style belt… hence the joke… nevermind.

Anyway, I also stopped by the ‘official’ RSA Bookstore and picked up a little book on 802.1X. When I say little, I mean little… and it was $60. Yes, seriously. To top it off, it’s probably the most poorly-written book I’ve ever read. You’ll see a book review on that later. I want to give it a fair shake and read the whole thing, but I’m not entirely sure I can submit myself to much more of the torture… we’ll see.

Thursday evening was the big RSA Codebreakers Bash and they really did it up right! There were several rooms full of fun, regardless of your taste. One room had a really good cover band and lots of music and dancing, another room had a huge bar area and light display I could have watched for hours. In one area, they had Guitar Hero full band playoffs, and in another yet bubble-head karaoke. Across the hall was a little more subdued, with more quiet sitting areas, perfect for chatting over a glass of wine. They also had crazy looking costumed ladies applying barcode tattoos to whomever was drunk enough to let them paste them on their forehead or face…. yeah… I have no clue about that one. I stopped in for about an hour before calling it a night. Thursday was day 6 in San Fran for me and I was exhausted. I did get some photos for you to try and capture the chaos. View photos from the Bash.

That pretty much sums up my day, and I left the hooplah on a Friday morning flight back to the East Coast. That’s about all I have from RSA 2008, but you’ll be hearing about some fun new projects and events that have grown out of this trip.

Next stop: Interop Las Vegas (yee-haw!)

# # #

It is already the end of February and the buzz is in full swing for this years RSA Conference. I usually know that it is RSA time because it takes place around my wedding anniversary.  However, this past Monday was my anniversary and no RSA.  That is because this year RSA is a little later, taking place the 2nd week of April in San Fransisco.

Over the years I have come to really enjoy RSA as a chance to catch up on the industry, friends and of course, parties!  Some of my favorites are the SC Magazine Awards show and the RSA conference party itself.  Last year one of my favorite events was the bloggers meet up that I had a hand in putting together along with Martin McKeay and a few others put together and was sponsored by Microsoft and Fortinet. That party has become legendary with posts about it here, here, here and here among other places. We had a similar event at Black Hat last year and that was fun too.  There is something about getting together with all of the folks you virtually talk to all the time via the blogosphere and put a real face and voice to a name.  We try to keep these blogging parties confined to blogger and media types, so the that everyone is comfortable sharing and conversing without the "general public" there. 

For this years RSA conference we wanted to do a similar type of event. However, the blogroll of security bloggers attending has grown quite a bit and of course most security media types are blogging now as well.  So we wound up getting about 100 of the top security blogging crowd together and got Fortinet, Microsoft and StillSecure to sponsor.  It is shaping up to be the bash of RSA, for me anyway.  The buzz around it was so loud that before we knew it we had a logo, our own official blog on the RSA conference site and a full committee running invites, food, drink and logistics (OK so Jennifer Leggio does most of the work)!  I am just totally pumped to meet a bunch of the folks on the RSVP list and have a great time. Truth be told I am also proud as a peacock that I played a role in putting this thing together from the beginning.

If you have a security blog or podcast, are going to be at RSA and want to attend there is information on the RSA blog page on how to get an invite. For many of you reading this, I know you are saying to yourself, "great sounds like a cool party, free drinks and I can't get an invite because I don't blog".  Well you don't have to fire up that old free blogger page you started but never finished months ago.  Through the magic of modern technology you can party along with us virtually!

We are going to have live video streaming, live audio podcasting and a live Twitter feed.  The RSA site has more details on signing up for the Twitter channel we have set up to follow on the pre-party chatter (or is it twitter) you can follow that at @RSABloggers2008. Hey it will be almost like being there.  Anyway, hope to see as many of you as possible at the party and as many of you as possible virtually if you can't make it!