It's become a big deal in IT security, too. Many corporate customers are demanding ROI models to demonstrate that a particular security investment pays off. And in response, vendors are providing ROI models that demonstrate how their particular security solution provides the best return on investment.

It's a good idea in theory, but it's mostly bunk in practice.

Before I get into the details, there's one point I have to make. "ROI" as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It's an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn't make sense in this context.

But as anyone who has lived through a company's vicious end-of-year budget-slashing exercises knows, when you're trying to make your numbers, cutting costs is the same as increasing revenues. So while security can't produce ROI, loss prevention most certainly affects a company's bottom line.

And a company should implement only security countermeasures that affect its bottom line positively. It shouldn't spend more on a security problem than the problem is worth. Conversely, it shouldn't ignore problems that are costing it money when there are cheaper mitigation alternatives. A smart company needs to approach security as it would any other business decision: costs versus benefits.

The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money.

Of course, that $1,000 has to reduce the chance of being robbed to zero in order to be cost-effective. If a security measure cuts the chance of robbery by 40 percent -- to 6 percent a year -- then you should spend no more than $400 on it. If another security measure reduces it by 80 percent, it's worth $800. And if two security measures both reduce the chance of being robbed by 50 percent and one costs $300 and the other $700, the first one is worth it and the second isn't.

The Data Imperative

The key to making this work is good data; the term of art is "actuarial tail." If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one.

Cybersecurity is considerably harder, because there just isn't enough good data. There aren't good crime rates for cyberspace, and we have a lot less data about how individual security countermeasures -- or specific configurations of countermeasures -- mitigate those risks. We don't even have data on incident costs.

One problem is that the threat moves too quickly. The characteristics of the things we're trying to prevent change so quickly that we can't accumulate data fast enough. By the time we get some data, there's a new threat model for which we don't have enough data. So we can't create ALE models.

But there's another problem, and it's that the math quickly falls apart when it comes to rare and expensive events. Imagine you calculate the cost -- reputational costs, loss of customers, etc. -- of having your company's name in the newspaper after an embarrassing cybersecurity event to be $20 million. Also assume that the odds are 1 in 10,000 of that happening in any one year. ALE says you should spend no more than $2,000 mitigating that risk.

So far, so good. But maybe your CFO thinks an incident would cost only $10 million. You can't argue, since we're just estimating. But he just cut your security budget in half. A vendor trying to sell you a product finds a Web analysis claiming that the odds of this happening are actually 1 in 1,000. Accept this new number, and suddenly a product costing 10 times as much is still a good investment.

It gets worse when you deal with even more rare and expensive events. Imagine you're in charge of terrorism mitigation at a chlorine plant. What's the cost to your company, in money and reputation, of a large and very deadly explosion? $100 million? $1 billion? $10 billion? And the odds: 1 in a hundred thousand, 1 in a million, 1 in 10 million? Depending on how you answer those two questions -- and any answer is really just a guess -- you can justify spending anywhere from $10 to $100,000 annually to mitigate that risk.

Or take another example: airport security. Assume that all the new airport security measures increase the waiting time at airports by -- and I'm making this up -- 30 minutes per passenger. There were 760 million passenger boardings in the United States in 2007. This means that the extra waiting time at airports has cost us a collective 43,000 years of extra waiting time. Assume a 70-year life expectancy, and the increased waiting time has "killed" 620 people per year -- 930 if you calculate the numbers based on 16 hours of awake time per day. So the question is: If we did away with increased airport security, would the result be more people dead from terrorism or fewer?

Caveat Emptor

This kind of thing is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They've jiggered the numbers so that they do.

This doesn't mean that ALE is useless, but it does mean you should 1) mistrust any analyses that come from people with an agenda and 2) use any results as a general guideline only. So when you get an ROI model from your vendor, take its framework and plug in your own numbers. Don't even show the vendor your improvements; it won't consider any changes that make its product or service less cost-effective to be an "improvement." And use those results as a general guide, along with risk management and compliance analyses, when you're deciding what security products and services to buy.

This essay previously appeared in CSO Magazine.

So I put together a simple class that holds an XmlDocument and implements ISerializable and called it SerializableXmlDocument. I'm sharing the source code here in the hopes that

a) somebody will find it useful, and

b) somebody smarter than I am will point out how I screwed it up and help me make it better.

SerializableXmlDocument includes implicit conversion operators to make it easy to convert to/from an XmlDocument. It holds the actual document in a property called Value. This "isomorph" pattern is one that I picked up from Craig.

While writing this code, I also wrote a helpful extension method for getting a byte array out of a MemoryStream that is exactly the length of the data written to the stream so far (CopyUpToSeekPointer). So don't go looking in the docs for MemoryStream for this method :) This is obviously not the most efficient way to consume bytes written to a MemoryStream since it copies the data into a new byte array, but it's very convenient in many scenarios.

Here is SerializableXmlDocument.cs:

using System;
using System.Runtime.Serialization;
using System.Xml;
using System.IO;

namespace Pluralsight.Samples
{
    [Serializable]
    public class SerializableXmlDocument : ISerializable
    {
        public SerializableXmlDocument() { }
        public SerializableXmlDocument(XmlDocument value)
        {
            this.Value = value;
        }

        public XmlDocument Value { get; set; }

        #region ISerializable implementation
        public SerializableXmlDocument(SerializationInfo info,
                                       StreamingContext context)
        {
            byte[] serializedData = (byte[])info.GetValue("doc",
                typeof(byte[]));
            if (null != serializedData)
                this.Value = Deserialize(serializedData);
        }

        public void GetObjectData(SerializationInfo info,
                                  StreamingContext context)
        {
            byte[] serializedData = null;
            if (null != Value)
                serializedData = Serialize(Value);
            info.AddValue("doc", serializedData);
        }
        #endregion

        #region implicit conversion to/from XmlDocument
        public static implicit operator SerializableXmlDocument(
            XmlDocument doc)
        {
            return new SerializableXmlDocument(doc);
        }
        public static implicit operator XmlDocument(
            SerializableXmlDocument sdoc)
        {
            return sdoc.Value;
        }
        #endregion

        #region Xml serialization helper methods
        private static byte[] Serialize(XmlDocument doc)
        {
            MemoryStream stream = new MemoryStream();
            doc.Save(stream);
            return stream.CopyUpToSeekPointer();
        }
        private static XmlDocument Deserialize(byte[] serializedData)
        {
            XmlDocument doc = new XmlDocument();
            doc.Load(new MemoryStream(serializedData, false));
            return doc;
        }
        #endregion
    }
}

...and here's the CopyUpToSeekPointer extension method for MemoryStream:

using System;
using System.IO;

namespace Pluralsight.Samples
{
    public static class MemoryStreamExtensionMethods
    {
        public static byte[] CopyUpToSeekPointer(
            this MemoryStream stream)
        {
            // copy only the part of the buffer
            // that contains the serialized document
            long length = stream.Position;
            byte[] buffer = stream.GetBuffer();
            byte[] result = new byte[length];
            for (int i = 0; i < length; ++i)
                result[i] = buffer[i];
            return result;
        }
    }
}

...and here's a sample object that uses SerializableXmlDocument:

using System;

namespace Pluralsight.Samples
{
    [Serializable]
    public class Item
    {
        public string Name { get; set; }
        public SerializableXmlDocument Data { get; set; }

        public void Print()
        {
            Console.WriteLine("Name: {0}", Name);
            Console.WriteLine(Data.Value.OuterXml);
        }
    }
}

...and here's a sample program that creates an instance of Item, serializes it, then deserializes it, printing diagnostics along the way to show that it's working properly.

using System;
using System.Xml;
using System.Runtime.Serialization.Formatters.Binary;
using System.IO;
using Pluralsight.Samples;

class DemoProgram
{
    static void Main(string[] args)
    {
        XmlDocument doc = new XmlDocument();
        doc.LoadXml("<root><child>text</child></root>");

        Item item = new Item
        {
            Name = "Testing 123",
            Data = doc,
        };

        // print object before serialization
        item.Print();

        BinaryFormatter formatter = new BinaryFormatter();
        MemoryStream stream = new MemoryStream();
        formatter.Serialize(stream, item);

        byte[] serializedItem = stream.CopyUpToSeekPointer();

        Console.WriteLine("Serialized data (base64): {0}",
            Convert.ToBase64String(serializedItem));

        item = (Item)formatter.Deserialize(
            new MemoryStream(serializedItem, false));

        // print object after deserialization
        item.Print();
    }
}

Here's the output of the previous sample program:

Flame away!