One of the main problems in counterterrorism today is that there are so many people and vehicles, and so much data and material, moving through globalization's myriad networks that it seems virtually impossible to track it all effectively. Nowhere has this problem been more acute than on the high seas.

In 2006, Adm. Harry Ulrich, then U.S. commander of NATO Naval Forces Europe, decided to do something about it. Despite having virtually no resources, his dream was to transpose the global air-traffic control system onto sea traffic.

Worldwide, aircraft are transparent, because they're all required to carry an identification beacon that allows them to be tracked leaving and entering airports, and monitored between airports, by a global network of sensors. Act suspiciously and somebody's fighter aircraft will soon be on your tail.

No such pervasive system currently exists globally for maritime traffic. While bigger ships carry an ID beacon similar to aircraft, without a shared monitoring network, that's like tracking only selected commercial jets and giving everyone else a pass.
So Ulrich, upon taking command, asked a simple question: "If we can do that in the air, why can't we do it on the sea?" He made a point of pioneering his sea-traffic-control effort first inside the Mediterranean, where NATO's southern naval forces have historically been concentrated, but his real target was waters off Africa -- the most ungoverned maritime space in the world.

Ulrich knew the U. S. Navy couldn't do it alone, much less bring Africa's meager coast-guard-like navies up to snuff so they could do it on their own. So he quickly created a network of assets -- both public and private -- to manage that space, modeling his monitoring system on international air-traffic control.

Ulrich began stitching together a network of shore-based sensors ringing the Mediterranean. His naval command then began initial monitoring by tapping into the International Maritime Organization's existing Automated Identification System, transforming NATO's ability to track ship traffic in the Med.

Almost overnight, NATO went from tracking dozens of ships on the Mediterranean to thousands, and instead of getting the data sometimes up to 72 hours late, now the contacts were being tracked in one to five minutes -- to an accuracy within 50 feet on the earth's surface.

When the classic big-firm systems integrators told Ulrich it would be too costly to pull it off, the admiral turned to the Volpe Center in Cambridge, Massachusetts, a U.S. Department of Transportation research center. Instead of hundreds of millions of dollars, Ulrich's initial network cost $900,000. The shore-based receivers are small, roughly the size of a radar dish you might find on a pleasure craft.

The strength of the system is a function of its reach: the more countries join, the larger the shared operational picture. By the time Ulrich retired at the end of 2007, he had enlisted 32 countries throughout the Mediterranean, the North Atlantic, along the west coast of Africa, around the Black Sea, and in the Pacific. Today, the network continues to spread around the planet.

With Ulrich's system in place, local police, coast guards, and border patrols catch most bad guys, obviating American military responses. As Harry told me for an article I wrote about his work in a fall 2007 issue of Esquire, "I don't do defense; I do security. When you talk defense, you talk containment and mutually assured destruction. When you talk security, you talk collaboration and networking. This is the future."

The admiral's legacy program, the Maritime Safety and Security Information System, earned the Volpe Center a prestigious "Innovations in American Government" award this month from Harvard University's Ash Institute for Democratic Governance and Innovation.

1. Our brilliant field engineer Dimitri McKay talks about the eternal topic of converting Windows event logs to syslog. Yes, Eric, we ALL know it is ugly, but that is the only way that actually works well across all systems ...
2. More on Windows and syslog: "Syslog ... 20 Years Later."  BTW, this is really not about syslog, but about Vista/2k8 finally getting an ability to natively centralize the event logs via event subscriptions ("It's only about twenty years behind schedule, if you're counting.")
3. Two fun pieces on correlation: 1 and 2. What often kills "a log correlation project"? "Whoever had worked on it had not had much time available to learn the way to properly configure the software" (from this)  and "correlation only really works when backed up by real data about what is the biggest problem in your environment, and how that problem manifests itself in the event logs." (from this) None of this is new, but a useful reminder nonetheless
4. Fun LogLogic podcast is here. The topic of this high-level discussion (CEO) is related to operational use for logs. I did one with them too; on logs and virtualization (will be up soon)
5. A couple of good posts on logging from Nemertes Research: "Sharpening Stones and Walking on Coals",  "Search or Destroy"
6. Reminder about a few useful Windows Vista and 2k8 events: 4802 (screensaver engaged) and 4803 (screensaver dismissed)
7. One person is wondering about the usefulness of logging after "experiencing" Linux auditd logging (kernel audit): "Logs are like a warm blanket; verbose logging means you can know what's happening on your systems if you keep up with the logs.  At the same time, logs become a burden very very easily, and they are easy to ignore." This post is a must read for us logging afficionados; producing too much log data is a sure way to make people hate you...
8. This also follows the same theme: people doubting the god-like power of logs :-) "So for an administrator to not care about logs was a shock." But would I argue that "log management is NOT a pain?" Now, would I? :-)
9. A classic about logging for application developers: "Building Secure Applications: Consistent Logging."  I am noticing a lot more discussions about logging in a developer community, e.g. see this and this (the latter, BTW, contains a lot of info on "why log" for developers). Overall, "getting logging right" is important (and will get more important in the future) and people need something NOW and cannot wait for the standards.  BTW, I am planning a mini-crusade on how to train application developers to include useful logging in their applications...
10. Finally, the "Is SIEM dead?" theme is continued in this fun post "Life after SIEM. Situational Awareness is next." Indeed, context is key for logs. BTW, if somebody mentions that I have "vendor bias", I will kick your ass! :-)

Enjoy!

Examples include:

• GSM mobile phones have an identifier for the phone (separate from the identifier for the user) that can be blacklisted when the phone is stolen.
• Some car radios will stop working when the battery is disconnected, and only start working again when a numeric code is entered. This is intended to deter theft of the radio.
• In Windows Vista, Bitlocker can be used to encrypt files. One of the intended applications for this is that if someone steals your laptop, it will be difficult for them to gain access to your encrypted files.

Ross told a story of what happened when he needed to disconnect the battery on his car: the radio stopped working, and the code he had been given to reactivate it didn’t work - it was the wrong code.
Ross argues that these reactivation codes are unecessary, because other measures taken by the car manufacturers - such as making radios non-standard sizes, and hence not refittable in other car models - have made them redundant.

I described how the motherboard on a laptop had needed to be replaced recently. The motherboard contains the TPM chip, which contains the encryption keys needed to decrypt files protected with Bitlocker. If you replace the motherboard, the files on your hard disk will become unreadable, even if the disk is physically OK. Domain-joined Vista machines can be configured so that a sysadmin somewhere within your organization is able to recover the keys when this happens.

Both of these situations suffer from classic usability problems: the recovery procedures are invoked rarely (so users may not know what they’re supposed to do), and, if your system is configured incorrectly, you only find out when it is too late: you key in the code to your radio and it remains a doorstop; the admin you hoped was escrowing your keys turns out not to have the private key corresponding to the public key you were encrypting under (or, more subtly: the person with the authority to ask for your laptop’s key to be recovered is not you, because the appropriate admin has the wrong name for the laptop’s owner in their database).

I also described what happens when an XBox 360 is stolen. When you buy XBox downloadable content, you buy two licenses: one that’s valid on any XBox, as long as you’re logged in to XBox live; and one that’s valid on just your XBox, regardless of who’s logged in. If a burglar steals your Xbox, and you buy a new one, you need to get another license of the second type (for all the other people in your household who make use of it). The software makes this awkward, because it knows that you already have a license of the first type, and assumes that you couldn’t possibly want to buy it again. The work-around is to get a new email address, a new Microsoft Live Account, and a new Gamer Tag, and use these to repurchase the license. You can’t just change the gamertag, because XBox live doesn’t let the same Microsoft Live account have two gamertags. And yes, I know, your buddies in the MMORPG you were playing know you by your gamertag, so you don’t want to change it.

It's become a big deal in IT security, too. Many corporate customers are demanding ROI models to demonstrate that a particular security investment pays off. And in response, vendors are providing ROI models that demonstrate how their particular security solution provides the best return on investment.

It's a good idea in theory, but it's mostly bunk in practice.

Before I get into the details, there's one point I have to make. "ROI" as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It's an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn't make sense in this context.

But as anyone who has lived through a company's vicious end-of-year budget-slashing exercises knows, when you're trying to make your numbers, cutting costs is the same as increasing revenues. So while security can't produce ROI, loss prevention most certainly affects a company's bottom line.

And a company should implement only security countermeasures that affect its bottom line positively. It shouldn't spend more on a security problem than the problem is worth. Conversely, it shouldn't ignore problems that are costing it money when there are cheaper mitigation alternatives. A smart company needs to approach security as it would any other business decision: costs versus benefits.

The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money.

Of course, that $1,000 has to reduce the chance of being robbed to zero in order to be cost-effective. If a security measure cuts the chance of robbery by 40 percent -- to 6 percent a year -- then you should spend no more than $400 on it. If another security measure reduces it by 80 percent, it's worth $800. And if two security measures both reduce the chance of being robbed by 50 percent and one costs $300 and the other $700, the first one is worth it and the second isn't.

The Data Imperative

The key to making this work is good data; the term of art is "actuarial tail." If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one.

Cybersecurity is considerably harder, because there just isn't enough good data. There aren't good crime rates for cyberspace, and we have a lot less data about how individual security countermeasures -- or specific configurations of countermeasures -- mitigate those risks. We don't even have data on incident costs.

One problem is that the threat moves too quickly. The characteristics of the things we're trying to prevent change so quickly that we can't accumulate data fast enough. By the time we get some data, there's a new threat model for which we don't have enough data. So we can't create ALE models.

But there's another problem, and it's that the math quickly falls apart when it comes to rare and expensive events. Imagine you calculate the cost -- reputational costs, loss of customers, etc. -- of having your company's name in the newspaper after an embarrassing cybersecurity event to be $20 million. Also assume that the odds are 1 in 10,000 of that happening in any one year. ALE says you should spend no more than $2,000 mitigating that risk.

So far, so good. But maybe your CFO thinks an incident would cost only $10 million. You can't argue, since we're just estimating. But he just cut your security budget in half. A vendor trying to sell you a product finds a Web analysis claiming that the odds of this happening are actually 1 in 1,000. Accept this new number, and suddenly a product costing 10 times as much is still a good investment.

It gets worse when you deal with even more rare and expensive events. Imagine you're in charge of terrorism mitigation at a chlorine plant. What's the cost to your company, in money and reputation, of a large and very deadly explosion? $100 million? $1 billion? $10 billion? And the odds: 1 in a hundred thousand, 1 in a million, 1 in 10 million? Depending on how you answer those two questions -- and any answer is really just a guess -- you can justify spending anywhere from $10 to $100,000 annually to mitigate that risk.

Or take another example: airport security. Assume that all the new airport security measures increase the waiting time at airports by -- and I'm making this up -- 30 minutes per passenger. There were 760 million passenger boardings in the United States in 2007. This means that the extra waiting time at airports has cost us a collective 43,000 years of extra waiting time. Assume a 70-year life expectancy, and the increased waiting time has "killed" 620 people per year -- 930 if you calculate the numbers based on 16 hours of awake time per day. So the question is: If we did away with increased airport security, would the result be more people dead from terrorism or fewer?

Caveat Emptor

This kind of thing is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They've jiggered the numbers so that they do.

This doesn't mean that ALE is useless, but it does mean you should 1) mistrust any analyses that come from people with an agenda and 2) use any results as a general guideline only. So when you get an ROI model from your vendor, take its framework and plug in your own numbers. Don't even show the vendor your improvements; it won't consider any changes that make its product or service less cost-effective to be an "improvement." And use those results as a general guide, along with risk management and compliance analyses, when you're deciding what security products and services to buy.

This essay previously appeared in CSO Magazine.

So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.