First, the most obvious conclusion: people still don't care much about log security; I am saying that since this was BY FAR the least popular of my polls. Only 24 people responded, so everything below is pretty unscientific :-)  A good way to explain it: look at the recent media? Do these people care about their key business data and their customer data security? Nope. So, how on Earth do you make them care about securing their log data?

Second,  it is entirely unsurprising that 83% of respondents want "Authenticated access to log server." In fact, I'd opine that 100% of people want authenticated access to any of their servers :-) But, this was my "red herring" to set the baselines for the rest of the questions... 

However, this is where the buck stops: other security measures are notably less popular.

Third, "Logging all access to logs" is my favorite and I am happy to see it reported as popular. But do you really do it?  Do you log access to log server OR access to actual logs? Think about it... I think a lot of people who do the latter still answered "yes" to this one.

Fourth,  "Reliable / acknowledged network transfer of log data" and "Encryption of log data in transit " are two true "no-brainer" security features; they took the next spot at 45% and 50% of those who answered. They are simple, they are easy, they make  sense - and, obviously, they don't make logs entirely secure so you need to do more. Why only 50%? Where is THE OTHER 50%?!

Fifth, "all things crypto" are below 40%. "Cryptographic hashing of stored logs", "Cryptographic signing of stored log data" and "Encryption of stored log data" all hover at around 30%. I attribute them to general disregard of log security AND reliance on "system security" (separate server, etc) over "data security" measures for log protection.

Finally, I am embarrassed to say that I missed  the obvious security measure "Separate server for logging, not accessible from the Internet;" one of my readers added this using "Other security measures" choice. Indeed, this is a good point - and a good idea to do it. Another option mention there was "Destroy old logs." Amen to that too!

Possibly related posts:

• Top 11 Reasons to Secure and Protect Logs
• All other polls and their analysis

So, here is my next monthly "Security Warrior" blog round-up of top 5 popular posts and topics.

1. In a bizarre twist of fate (maybe driven by my latest poll), the "Top 11 Reasons to Secure and Protect Your Logs" came up as #1 most popular post in August.  The analysis of said log security poll is coming up tomorrow. BTW, see my other logging polls:  poll #8 that covered context data for log analysis is analyzed here and a controversial Windows Log Collection Poll (which is a poll #7)  and poll #6 about logs that people actually review and poll #5 about logging challenges.
2. Next up is my post "Log Management - Day 1," which talks about the very first thing you do when embarking on a journey to log management.
3. Still burning hot is a post with my irreverent comments on a Terry Childs saga. Namely, "On Doomsaying (Terry Childs case)", "So ... Am I? Maybe I Am!" and "Admins , Good Guys or "I am NOT an Idiot!""
4. Somewhat predictably, PCI compliance is all the rage again with 1.2 coming out soon. So, MUST-DO Logging for PCI? post was again propelled to a place in my monthly Top5 list. It discusses the fact that there is no "easy list" of what you MUST do to comply.
5. Finally, my post "11 Signs That Your SIEM Is A Dog or "Raffy, You Killed SIM!"". It is both humorous and sadly true (and backed up by other sources)

See you in September,  when .... ah, come on! I will tell you later :-)

Possibly related posts / past monthly popular blog round-ups:

• Monthly Blog Round-Up - July 2008
• Monthly Blog Round-Up - June 2008
• Monthly Blog Round-Up - May 2008  
• Monthly Blog Round-Up - April 2008  
• Monthly Blog Round-Up - March 2008  
• Monthly Blog Round-Up - February 2008  
• Monthly Blog Round-Up - January 2008  
• Monthly Blog Round-Up - December 2007  
• Monthly Blog Round-Up - November 2007  
• Monthly Blog Round-Up - October 2007  
• Monthly Blog Round-Up - September 2007
• Monthly Blog Round-Up - August 2007

1. Sad, but VERY insightful story of Alan Shimmel getting 0wned (1,2,3,4, others on his blog)
2. A very good essay on security industry/market/community "Evolution is Punctuated Equilibria" ("Right now, Internet security is due for another period of rapid change.")
3. As I like to say, most everybody in out industry is confused about risk (myself included, in fact) - here is some nice reading about the subject: "Quant love", "What is Risk?" ("The probability of a threat overcoming security controls resistance to exploit a vulnerability that results in a loss.") While you are at it, check this blurb about risk and CVSS (BTW, CVSS is about "V" - vulnerability, not "R" for risk!)
4. Solid gold on "running IT as business" (and where it hits the wall) - Richard, the original CIO.com piece ("If you've tried managing an internal IT department as a bona fide business you already know that you can't take that very far, for the obvious reason that your IT department isn't a business.")
5. More fun stuff from Richard on insiders and why NOT look for them (sadly, same logic applies to not looking for owned boxes in your environment...).
6. Analyst firms shocking discovery: wireless MAY have security issues (I guess count it as humor...)
7. Fun read: "Challenges of Enterprise Cloud Computing" ("By moving the data into the cloud, enterprise, for now, will lose some capabilities to govern their own data set.")
8. Raffy on visualization. ("One of the dangerous things is if you don't understand the log file itself, don't assume you'll understand the visualization of it or even generate a visualization that makes sense") Amen to that! BTW, Raffy's book is finally out.
9. Compliance and checkbox mentality: fun pickup from my original "DLP and Compliance" post - Rich and TechTarget. Good stuff! ("Don’t Sell ‘Compliance’ If It Isn’t A Checkbox ")
10. RedHat is nicely 0wned (more info)
11. BGP hole to dwarf the DNS hole?
12. Chris continues the virtualization and PCI DSS theme here. The jury is still out on this one, even though the common sense approach (that virtualization is OK in regards to PCI) will probably win.
13. NEWS FLASH! Privacy dies. The date of death? 1967. While reading it, think just how visionary some folks are...
14. Finally, just for laughs: How to Spin Bad News

Enjoy!

BTW, I am saving some fun reading for dedicated posts soon :-)

HANSEI - WHAT IS “RELENTLESS REFLECTION?” - And why we’re talking about it in the context of Risk Analysis.

Recall from yesterday’s post about how I got to thinking about the concept of Hansei-Kaizen, “relentless reflection” and “continuous improvement” and how we might apply that to risk management.  It’s a concept born of Toyota and is, in some way, the foundation for “Lean” production.

Call me biased, but I think that Hansei - the act of ‘relentless reflection’ made structured is the analytical function.  And I hate to debate (post-mortem) the father of Toyota quality success when he says that Hansei is the “check” in Plan/Do/Check/Act, but I think that Hansei also applies to the “Plan” of the P/D/C/A or Deming cycle.

You’ll recall the P/D/C/A cycle can be thought of even as an implementation of Scientific Method, in that it is Observation & Hypothesis Creation (P), Experiment (D), Analysis (Check), and Act (Revise/New Hypothesis, etc…).  Well then as such, the Hypothesis creation involves creating a model or creating an expected outcome for data using the currently accepted model.

So in our industry there is an opportunity for Relentless Reflection in both the Observation and Hypothesis (Plan) creation steps, and the Check step.  We create an estimate for control strength, or probable losses in the context of risk- then we go to Experiment step.  That hypothesis can be put it into production, have an audit, have a penetration test, whatever, in the context of the Do step.  BTW - using Hansei/Analytics in Plan is one way that strong analytical functions can really make penetration testing more useful - as a means to test the estimates and inputs into a model.  It’s Penetration Testing 2.0!  (<- tongue fully in cheek, yes)

Those who are versed in the reasons to merge Six Sigma and Lean together are probably already seeing where I’m going with this today.  But before you think that a simple DMAIC function is all that is needed to create proper “Hansei”, let me encourage you to keep reading.

Now if the analytical function can said to be “reflection”, why must it be relentless?

One word.  Change. There are essentially four separate “landscapes” or sources of change that we face (more on those tomorrow).  But anyone who has tried to manage system compliance, log management or policy exceptions knows that change is possibly the most difficult thing we security professionals must manage.  And when you think about it, there aren’t too many other business functions like information security where significant visibility and insight about the environment is needed for “complete” information (get bullish on Log Management is my recommendation).

HANSEI STEPS ADAPTED TO INFORMATION SECURITY

This is one of those quality control concepts that we can mangle adopt.  At Toyota, Hansei-Kaizen includes the following basic steps:

1. Initial problem perception
2. Clarify the problem
3. Locate area/point of cause
4. Investigate root cause (using an ask why 5 times approach)
5. Countermeasure
6. Evaluate
7. Standardize

Now it’s important to note that part of this includes the concept of Go See For Yourself, called “Gemba“.  Gemba can be translated as “the actual place” or “the place where virtue or truth is found.” At Toyota this might mean going to the shop floor to see the issue at hand in the production line.  But for us, that’s a problem because we live in the virtual world.  There’s usually not much use in hanging out in the wiring closets to try to see the problems.

But if you combine the concept of Gemba with the concept of “Nemawashi” –the process of discussing problems and potential solutions with all those affected- we can forge a similar concept using risk analysis.  That is discussing the issue and the risk associated with an issue (what some people would call “risk management”) with the business/LOB/data owner and let them accept authority and the risk decision.  We, the risk analyst, our goal is simply to perform items 1-5 (presenting countermeasure options that include transferring or accepting risk).  By going to the line of business and involving them, responsibility is shared.  Also, if you structure organizational behavior right, personal risk is transferred!

This sort of approach is also in harmony with concepts like “mutual ownership of problems,” or “genchi genbutsu,” (solving problems at the source instead of behind desks), and the “kaizen mind,” (an unending sense of crisis behind the company’s constant drive to improve).

One of the criticisms I have with the way most people try to implement DMAIC into “Lean”

REQUIREMENTS

Now to get this done, I really see three significant requirements.

1.)  A change in political structure.

2.)  Models that provide consistent, defensible analysis.

3.)  A Quantitative approach.  This means using actual units of measurement (not just amorphous percents, ordinal scales, etc.)  for risk and it’s subsequent factors.  Sure there are times when Q&D qualitative approaches are acceptable, but policy should be to have quantitative analysis whenever and wherever possible.

That last item - the quantitative approach - is really quite important.  And the reasons why will be discussed further in tomorrow’s post:

“What should we be reflecting about? & What is needed for reflection?”

P.S.  Your comments and suggestions, as always, are welcome.

P.P.S  Those who may be familiar with Lean/SixSigma/Kaizen sorts of mashups may be thinking - “hey, an Analytical step is built into SixSigma”.  Well, yes there is some prevision for analytical functions based on statistics, but I find SixSigma geared towards creating a State of Knowledge about operational processes, not towards creating a State of Wisdom for CISO’s around security & risks “big questions”.  In otherwords, the analytical function in DMAIC is in the context of Kaizen, and a different step than “reflective” analytics.

"consumers hate tokens."

"Everyone wants to be the single federation platform for everyone else."

"Federation will never work. It won’t work because the single most important consumer Web applications in the world are scared of it. Banks hate the concept because it becomes a weakest link in the chain problem."

"Tokens don’t actually solve most security problems, like man-in-the-middle, phishing, and keystroke-logging malware."

"Oh yes, and finally, consumers are going to have to carry around 13 of them just to make sure they can log into whatever they need to log into since no one will federate."

"Global federation is nowhere near a solid concept in the consumer space, despite what the vendors will try to sell you."

"This is a long term problem. If you work on it and make any progress against it, you'll find yourself much smarter at the far end, than you were at the near end.

When I was in Norway about 5 years ago, I was there very close to the summer solstice. I was wandering around town at 2 o'clock in the morning and there was plenty of light out. You come to a sign that says New Minsk about 60 km and it points south.

And I ask the lady "what country is this?"

She scratched her head for a bit, and said "well I think its Norway"

I said "well who plows the roads?"

"well Norway does, but he have to pay them."

There is a triple boundary in this town that I was in between Norway, Finland and Russia.

But what I did there, was, I had a card about wallet size, I stuck it into a machine, I punched in four digits, and it gave me about 2,000 krone, whatever the hell that is.

Now there are a lot of participants in that transaction. When I put a card into that machine, punch in a pin, and it gurgles for awhile, and finally gives me, a fairly large amount of money. There are a lot of participants in that transaction. The bank that owned the machine that gave me the money, it gave some money away -- that bank wants it back. The pin is necessary to convince my own bank that I'm me. But I don't want my pin to be broadcast all over the world. My bank in the us, it hasn't really given out or taken in any money, really. But there is a lot of credits involved here. Somebody needs to charge somebody else for having more money available. Even though there was actually no cash transfer.

And the problem that I have in mind is
- who are all the participants in an ATM transaction?
- what do those participants need to satisfy their problems?
- how is that in fact done?

In a general way, does the atm system actually work in some reasonable sense? To which the answer is by the way: yes. The atm system damn well works. With extremely high reliability and accuracy. It surprises me. Its quite a bit different than voting machines.

• Good news: router is now officially a firewall (it has been for a while, but many people are still stuck in "security device" vs "network device" cloud) - see Req 1
• From the "WTH dept": anti-virus is a MUST on ALL platforms - Req 5. Please ship me some of the stuff they are smoking; I want it! BTW, I am going to Amsterdam soon :-)
• WAF or code review for web application security is still a stupid "OR" - Req 6.6. OMG, please, software security folks, teach them the truth.
• Can we kill "plain text passwords" once and for all? Req 8 tries to achieve that noble goal (good thing!)
• Visit your offsite data storage - good (if costly) idea - added to Req 9. Requirements to secure electronic AND  paper media  are solid too.
• Love it, love it! Req 10 explains that logs needs to be actually available: 'three months of audit trail history must be “immediately available for analysis” or quickly accessible' (bye-bye, silly log dumps...)
• Some vulnerability stuff clarified in Req 11, mostly about ASVs and pentesting.
• Scope of security policy is expanded to "employee-facing technologies" (what a term!) - Req 12
• All over: more references to wireless  (WEP, access points, hidden SSIDs, etc) - indeed, recent data losses are often due to insecure wireless.

Overall, a minor change that, sadly, doesn't touch a few KEY areas, such as virtualization, for one.

So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.