It's become a big deal in IT security, too. Many corporate customers are demanding ROI models to demonstrate that a particular security investment pays off. And in response, vendors are providing ROI models that demonstrate how their particular security solution provides the best return on investment.

It's a good idea in theory, but it's mostly bunk in practice.

Before I get into the details, there's one point I have to make. "ROI" as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It's an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn't make sense in this context.

But as anyone who has lived through a company's vicious end-of-year budget-slashing exercises knows, when you're trying to make your numbers, cutting costs is the same as increasing revenues. So while security can't produce ROI, loss prevention most certainly affects a company's bottom line.

And a company should implement only security countermeasures that affect its bottom line positively. It shouldn't spend more on a security problem than the problem is worth. Conversely, it shouldn't ignore problems that are costing it money when there are cheaper mitigation alternatives. A smart company needs to approach security as it would any other business decision: costs versus benefits.

The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money.

Of course, that $1,000 has to reduce the chance of being robbed to zero in order to be cost-effective. If a security measure cuts the chance of robbery by 40 percent -- to 6 percent a year -- then you should spend no more than $400 on it. If another security measure reduces it by 80 percent, it's worth $800. And if two security measures both reduce the chance of being robbed by 50 percent and one costs $300 and the other $700, the first one is worth it and the second isn't.

The Data Imperative

The key to making this work is good data; the term of art is "actuarial tail." If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one.

Cybersecurity is considerably harder, because there just isn't enough good data. There aren't good crime rates for cyberspace, and we have a lot less data about how individual security countermeasures -- or specific configurations of countermeasures -- mitigate those risks. We don't even have data on incident costs.

One problem is that the threat moves too quickly. The characteristics of the things we're trying to prevent change so quickly that we can't accumulate data fast enough. By the time we get some data, there's a new threat model for which we don't have enough data. So we can't create ALE models.

But there's another problem, and it's that the math quickly falls apart when it comes to rare and expensive events. Imagine you calculate the cost -- reputational costs, loss of customers, etc. -- of having your company's name in the newspaper after an embarrassing cybersecurity event to be $20 million. Also assume that the odds are 1 in 10,000 of that happening in any one year. ALE says you should spend no more than $2,000 mitigating that risk.

So far, so good. But maybe your CFO thinks an incident would cost only $10 million. You can't argue, since we're just estimating. But he just cut your security budget in half. A vendor trying to sell you a product finds a Web analysis claiming that the odds of this happening are actually 1 in 1,000. Accept this new number, and suddenly a product costing 10 times as much is still a good investment.

It gets worse when you deal with even more rare and expensive events. Imagine you're in charge of terrorism mitigation at a chlorine plant. What's the cost to your company, in money and reputation, of a large and very deadly explosion? $100 million? $1 billion? $10 billion? And the odds: 1 in a hundred thousand, 1 in a million, 1 in 10 million? Depending on how you answer those two questions -- and any answer is really just a guess -- you can justify spending anywhere from $10 to $100,000 annually to mitigate that risk.

Or take another example: airport security. Assume that all the new airport security measures increase the waiting time at airports by -- and I'm making this up -- 30 minutes per passenger. There were 760 million passenger boardings in the United States in 2007. This means that the extra waiting time at airports has cost us a collective 43,000 years of extra waiting time. Assume a 70-year life expectancy, and the increased waiting time has "killed" 620 people per year -- 930 if you calculate the numbers based on 16 hours of awake time per day. So the question is: If we did away with increased airport security, would the result be more people dead from terrorism or fewer?

Caveat Emptor

This kind of thing is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They've jiggered the numbers so that they do.

This doesn't mean that ALE is useless, but it does mean you should 1) mistrust any analyses that come from people with an agenda and 2) use any results as a general guideline only. So when you get an ROI model from your vendor, take its framework and plug in your own numbers. Don't even show the vendor your improvements; it won't consider any changes that make its product or service less cost-effective to be an "improvement." And use those results as a general guide, along with risk management and compliance analyses, when you're deciding what security products and services to buy.

This essay previously appeared in CSO Magazine.

Parlez-vous open source? While wide-spread open source usage is still debated in many companies, the French have been advocating for all open source all the time in government and education. French President Nicolas Sarkozy set up an economic commission that recommended tax benefits to stimulate more open source development. Lesson learned from France: start ‘em early. “All students in France use open source.”

Just in time for Labor Day, John Edwards (no, not that one) comes out with an informative guide on “Who provides what in the cloud”. No doubt, this will be a rapidly expanding list, but what’s really interesting is the comment on the article. People have very strong opinions on the cloud…

Research firm Aberdeen Group reports that network costs will increase slightly more than 5 percent over 2007. Contributing factors: “need for speed”, shift from standard to mobile PCs (more end points of connectivity), and the ever-expanding network. And of course the hidden costs of multiple tools with multiple management consoles – if you’re not smart enough to choose say a comprehensive network management solution that is vendor agnostic…One tool to monitor them all…

And just because I miss the Olympics already, here’s an irreverent take on what it’s like to lose to Michael Phelps. http://www.thetechstop.net/?p=1503

Enjoy your long Labor Day Weekend!

For those of you who aren’t familiar with XSS Filter, a brief summary is that it is a client-side defense against reflected cross-site scripting (XSS) attacks.  It works by recognizing that reflected XSS attacks inject script into the string that the browser sends to the targeted web server.  If the server doesn’t neuter or strip out the injected script, it gets sent back to the browser and executed in the context of the target web page.  Bad things then happen.  At a high level, XSS Filter remembers the string that the browser sent to the server, and looks at the server’s response to see if any of the script was actually in that string.  If it was, then XSS Filter decides that it got there because it was injected by an XSS attack and blocks the script from executing.  The rest of the web page renders as usual.  This is a vastly oversimplified sketch of XSS Filter – for details, see the post by David Ross, inventor of XSS Filter on the Security Vulnerability Research and Defense blog.

So what does XSS Filter have to do with the SDL?  Well, for almost nine years, since XSS was first discovered at Microsoft, we’ve been trying to figure out effective ways to reduce vulnerability to XSS attacks.  Our focus has been on improving the ways that web page developers code their pages, and we’ve developed a lot of tools and techniques for making web content safer from XSS attacks and for detecting XSS vulnerabilities in live pages.  The SDL requires the use of many of these tools and techniques, and we’re sure we’ve prevented a lot of XSS vulnerabilities from being introduced into Microsoft web pages as a result. 

But while we identify (and the SDL requires) measures that allow developers to avoid classes of vulnerabilities, we also look to identify more sweeping solutions that can either 1) eliminate classes of vulnerabilities, 2) reduce their severity, or 3) reduce the likelihood of attacks being successful.  The process usually starts from deep understanding of a class of vulnerabilities and attacks, and then we broaden defenses from there.  In the case of XSS Filter, David’s years of work researching XSS led him to come up with an approach that blocks many of the most common vulnerabilities to reflected attacks found on the web today.  The solution is compatible with existing web pages (doesn’t “break the web”) and thus we were able to enable it by default for users of Internet Explorer 8.  Because it’s a client-side mitigation, it will help protect users from attacks even though the sites they visit may be vulnerable to XSS. 

Our work on buffer overrun defenses follows a somewhat similar pattern – we started by prescribing coding techniques, banning the use of some APIs, and building tools that detect coding constructs that look like buffer overruns.  As we gained a deeper understanding of how buffer overruns can be exploited, we enhanced the /GS compiler flag and added ASLR in a quest to cause classes of exploits to fail even if a buffer overrun remains.  We’re not yet close to eliminating the SDL requirements for use of tools and coding techniques, but the SDL also requires the use of the mitigations to reduce the severity of vulnerabilities that slip past.  Will we ever get to the point where the mitigating technologies are so strong that we can relax the coding requirements?  Maybe not, but we will continue to introduce technologies that reduce the chances of a successful attack.

Similarly, in the case of XSS, even after IE8 ships, the SDL will continue to require the use of safe web site coding practices and tools such as the Anti-XSS library both to protect users of browsers other than IE8 and to provide protection in recognition of the fact that XSS Filter is a mitigation or defense in depth rather than a complete solution.  But we’ll also be keeping our eyes open (and doing active research) in the quest for an even more effective defense – whether client or server side – that eliminates XSS for good.

This post is a little far afield from the normal content of the SDL blog, but I thought it was important to provide a picture of the role of security science and security research in defining SDL requirements and in making major improvements in software security.  You can read more about our work in security science in the Security Vulnerability Research and Defense blog.

Learn how file integrity monitoring solutions work and the capabilities you should expect your solution to have. Then review a detailed checklist you should complete before purchasing your solution. Finally, discover how Tripwire Enterprise effectively combines file integrity monitoring with configuration assessment-a single configuration control solution that proactively assesses and monitors the IT infrastructure and enables organizations to achieve and maintain compliance with standards and regulations.

"The rigorous studies clearly show red-light cameras don't work," said lead author Barbara Langland-Orban, professor and chair of health policy and management at the USF College of Public Health. "Instead, they increase crashes and injuries as drivers attempt to abruptly stop at camera intersections."

Comprehensive studies from North Carolina, Virginia, and Ontario have all reported cameras are associated with increases in crashes. The study by the Virginia Transportation Research Council also found that cameras were linked to increased crash costs. The only studies that conclude cameras reduced crashes or injuries contained "major research design flaws," such as incomplete data or inadequate analyses, and were always conducted by researchers with links to the Insurance Institute for Highway Safety. The IIHS, funded by automobile insurance companies, is the leading advocate for red-light cameras since insurance companies can profit from red-light cameras by way of higher premiums due to increased crashes and citations.

And, of course, the agenda of the government is to increase revenue due to fines:

A 2001 paper by the Office of the Majority Leader of the U.S. House of Representatives reported that red-light cameras are "a hidden tax levied on motorists." The report came to the same conclusions that all of the other valid studies have, that red-light cameras are associated with increased crashes and that the timings at yellow lights are often set too short to increase tickets for red-light running. That's right, the state actually tampers with the yellow light settings to make them shorter, and more likely to turn red as you're driving through them.

In fact, six U.S. cities have been found guilty of shortening the yellow light cycles below what is allowed by law on intersections equipped with cameras meant to catch red-light runners. Those local governments have completely ignored the safety benefit of increasing the yellow light time and decided to install red-light cameras, shorten the yellow light duration, and collect the profits instead.

The cities in question include Union City, CA, Dallas and Lubbock, TX, Nashville and Chattanooga, TN, and Springfield, MO, according to Motorists.org, which collected information from reports from around the country.