So can open source weather out the economic storm? Emerging from the dot-com bust, open source has matured, its legal framework and values are established, and serious players are in the game. But as this post on ZDNet points out, consolidation is on the way. “IDC renamed its LinuxWorld Show in San Francisco next year Open Source World – a clear shot across the bow at O’Reilly’s OSCON.” Will open source (from free to lower-cost alternatives to commercial software) flourish in a time of tightening budgets or will projects quietly go away for lack of funding (VC and that pesky business model thing) and, let’s face it, the “extra time” of IT pros tasked yet again to do more with less?

It’s October 2008 and Charles Babcock writes, “CA Embraces Virtualization As Future of Data Center Management”. Beyond keeping up with what competitors are doing, I enjoy this article for the masterful way it depicts the nightmare that is working with traditional frameworks. Too slow, too expensive, too complex, too many modules – it’s all in here. And somehow, I don’t think that was the point of it. So, $154,000 for CA Data Center Automation Manager – which can “consult” the CA CMDB (pricing starting at what do you think, something like $500K to a million – don’t forget those services) plus CA Wily APM (Introscope 8 and Wily Customer Experience Manager 4.2; pricing anyone?) metrics that get fed back into Data Center Automation Manager to help determine the virtual machine resources that are needed. Plus can also integrate info from CA Endeavor’s software change management tracking and CA SysView and in future with CA Management Suite for Mainframe Linux, potentially. I am not kidding about this list. And, we’ve been hearing this for a while – “Unicenter” the brand goes away and is replaced by “CA NSM”. The brand goes away. Why retire a successful brand? Ah.

I love this post on EMC, “Eleven Things You Didn’t Know About the World’s Largest External Disk Storage Company.” Although I guess I really don’t know much about Joe Tucci, since #11 says:

“Contrary to conventional thought, it is not true that the EMC President/CEO is the older, gentler brother of the fictional patriarch of HBO’s hit television series.” Hunh. I just googled him, thinking maybe it was a resemblance thing. Nope."

And on a much lighter note. A funny from Dell. 2 years later, I just stumbled across this Proprietaryville , Jibjab-ish video, called Dell the Journey. Legacy systems being escorted onto the Retirement Home bus. Michael Dell as knight in shining armor, singing no less. Joe Tucci and Larry Ellison showing up as heroes leading the charge against Proprietaryville (yes, funny in and of itself). And my favorite, “Now let’s go kick some proprietary apps.”

The report was written by a committee whose members include William Perry, a professor at Stanford University; Charles Vest, the former president of MIT; W. Earl Boebert, a retired senior scientist at Sandia National Laboratories; Cynthia Dwork of Microsoft Research; R. Gil Kerlikowske, Seattle's police chief; and Daryl Pregibon, a research scientist at Google.

They admit that far more Americans live their lives online, using everything from VoIP phones to Facebook to RFID tags in automobiles, than a decade ago, and the databases created by those activities are tempting targets for federal agencies. And they draw a distinction between subject-based data mining (starting with one individual and looking for connections) compared with pattern-based data mining (looking for anomalous activities that could show illegal activities).

But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so."

A summary of the recommendations:

• U.S. government agencies should be required to follow a systematic process to evaluate the effectiveness, lawfulness, and consistency with U.S. values of every information-based program, whether classified or unclassified, for detecting and countering terrorists before it can be deployed, and periodically thereafter.

• Periodically after a program has been operationally deployed, and in particular before a program enters a new phase in its life cycle, policy makers should (carefully review) the program before allowing it to continue operations or to proceed to the next phase.

• To protect the privacy of innocent people, the research and development of any information-based counterterrorism program should be conducted with synthetic population data... At all stages of a phased deployment, data about individuals should be rigorously subjected to the full safeguards of the framework.

• Any information-based counterterrorism program of the U.S. government should be subjected to robust, independent oversight of the operations of that program, a part of which would entail a practice of using the same data mining technologies to "mine the miners and track the trackers."

• Counterterrorism programs should provide meaningful redress to any individuals inappropriately harmed by their operation.

• The U.S. government should periodically review the nation's laws, policies, and procedures that protect individuals' private information for relevance and effectiveness in light of changing technologies and circumstances. In particular, Congress should re-examine existing law to consider how privacy should be protected in the context of information-based programs (e.g., data mining) for counterterrorism.

Here are more news articles on the report. I explained why data mining wouldn't find terrorists back in 2005.

Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

Someone on the IE security team forwarded me a document that had additional details. So here, for your enjoyment, is a chart listing the default settings for each security level. To answer the newsgroup poster, "medium" and "medium-high" aren't the same.

About the formatting: to get it to fit within the width of the blog's text section, I've made some abbreviations.

In a few cases, the table shows a number rather than D or E or P; below the table is a description of each such entry.

At the very bottom of this post I've included the settings from the privacy tab, too.

Note: these settings reflect those for Internet Explorer 7 on Vista SP1. Please see the MDSN link above for differences between IE 6 and IE 7.

.NET Framework

.NET Framework-reliant components

     1 = High safety

ActiveX controls and plug-ins

Downloads

Enable .NET Framework setup

Miscellaneous

     1 = Prohibit downloads from software update channels
     2 = Cache content downloaded from software update channels
     3 = Automatically install software updates

Scripting

User authentication

     1 = Prompt the user for name and password
     2 = Automatic logon only in intranet zone
     3 = Automatic logon with current user name and password

Privacy settings (on the "Privacy" tab)

I say today is a good day for two reasons:  1.)  BT’s CSO Jill Knesek wrote an article called “Keys to establishing an end-to-end security strategy” which begs some discussion within context, and 2.)  Sara Peters on Twitter last night wanted to know why I thought “risk management” requires more than what most “best practices” around the subject suggest the effort requires.

WHAT SHOULD WE BE REFLECTING ABOUT?

Jill Knesek’s article gives us a rough outline of how to develop a security strategy.  It’s fairly high-level, Pragmatic CSO-ish type stuff.  It gives us a nice outline of

• Get a seat at the table
• Process
• People
• Technology

Nothing earth-shattering there.  But it is a very nice broad CISO-level taxonomy about what we have to reflect on.  The need to reflect is driven by something Jack told me long ago,

“The amount of risk we have is a function of the decisions we made and our ability to execute on them from some point in the past”.

As an Aside:  So Sarah if you’re reading, this quote does much to explain why I said I disagree with much of what our industry calls “risk management”.  We tend to define the process of risk management as essentially a tactical “issue whack-a-mole” exercise. Find the issue.  Analyze the “risk” around the issue.  Fix the issue.  Repeat. This hamster-wheel-of-pain, while sometimes an effective tool for the CISO, is incongruous with addressing root causes (the ability to match a tactical issue to the strategic shortcoming that created the issue is up to the expertise of the analyst or consultant).  It is only Kaizen without (good) Hansei, if you will.

Back to what Jill is writing - the sorts of things we should be reflecting about can be thought of in context of her outline.  Namely:

1. Once you have a seat at the table, what is the nature of that relationship?  Who are you reporting to and what are their concerns? What and how are you reporting and how might that be addressing their concerns?
2. What processes are in place?, How do you know that those are the processes that should be in place? If they are, what kind of job am I doing at those processes?
3. What is the quality of the skills and resources I have from a people perspective, and how do I know if they are adequate?  How do I know that the training they petition me for will effectively reduce organizational risk?
4. Are the Technology solutions I have in place effective, are we managing them effectively, and what sort of States of Knowledge could they provide me with (to make good decisions and execute upon them, from above)?

This, for the CISO, is Hansei.  The continuous management of it is Kaizen.  Not to particularly pick on Jill’s article, but creating a “risk register expressed in ALE” might be fine if you’re trying to explain to the board what your “first 100 days in office” will be like - but these sorts of lists are usually not very strategic in nature, and as such, depending on the outcome of that risk register (and the models used to create it) it might not actually be useful.

WHAT IS NEEDED FOR REFLECTION?

So what is needed for this sort of CISO-level Hansei?

The CISO must understand the

• Current State of Nature

turn that into a

• State of Knowledge

and use that to create a

• State of Wisdom.

CREATING A STATE OF NATURE FOR THE IRM PROGRAM

This Current State of Nature determination be done by applying analytical methods to a program audit.  We must understand questions like,  “What is in that program and how is it structured?”  before we can answer questions about “how (good/bad) are we at managing risk?”

There are many ways to structure an IRM program, but as an example - below is a graphic shared with me by Adrian Seccombe.  For those who know Adrian and the Trust Model - this is classified as “white” so it’s OK for public display and consumption.  But here’s what Adrian is trying to build at a high level:

So regarding Adrian’s program diagram:

1. Is a governance framework.  Think ITIL.
2. Is a risk framework.  Think ISO 27002 using FAIR as an analytical engine.  To be fair (pun) I believe this is really issue management, and it’s a process, but that’s OK.
3. Reg compliance should be self explanatory.  That’s essentially what GRC products do for you.
4. With architecture, I think Adrian is inclined towards TOGAF.
5. Security is the ISMS in place (27001, ISM^3, PCI, whatever…)
6. Are the processes that drive execution
7. Monitor (audit) is creating a State of Nature and Evaluate is creating a State of Knowledge from that State of Nature around items 1-6.

EVALUATE - CREATING A STATE OF KNOWLEDGE ABOUT THE IRM PROGRAM

That evaluate is Hansei/Kaizen.  Evaluation, done effectively, will drive actual organizational risk exposure.  Evaluate will even answer those four questions we raised in the “What Should We Be Reflecting About” section above:

1. Once you have a seat at the table, what is the nature of that relationship?  Who are you reporting to and what are their concerns? What and how are you reporting and how might that be addressing their concerns?
2. What processes are in place?, How do you know that those are the processes that should be in place? If they are, what kind of job am I doing at those processes?
3. What is the quality of the skills and resources I have from a people perspective, and how do I know if they are adequate?
4. Are the Technology solutions I have in place effective, are we managing them effectively, and what sort of States of Wisdom do they provide me with (to make good decisions and execute upon them, from above)?

If we could have a nice metric (or set of metrics) that answers these questions, we might call it something like “My Ability To Manage Risk” or MATMR for short.

GETTING TO A STATE OF WISDOM

What’s then missing is how you create a State of Wisdom around the State of Knowledge developed - your “MATMR” metric.  That is, given the current State of Knowledge - how can I be most effective?  This State of Wisdom requires proper models for what risk is, and what you can do to manage it applied in a probabilistic manner (because we can’t intrinsically *know* the future, we can only say with some degree of certainty what the desired course should be).

So the outcome of Hansei/Kaizen should be to create a State of Wisdom about Risk Management.  This is why reflection must be relentless - because your wisdom must be similarly abundant.

This is no small part of the reason RMI exists, why we build software and help organizations understand the things they do.

The session took place at the beginning of Partner Day. The “regular” conference sessions actually begin tomorrow. Today is spent focusing on partner issues and enablement.

The panel for this session included:

• Mark Chuang Group Manager, Product Marketing, VMware, Inc.
• Kenon Owens Staff Systems Engineer, VMware, Inc.

You have to remember that most of the Partners here are not vendors like ScienceLogic, but big and small shops that are selling IT, networking and now virtualization solutions into end-customer environments. For these guys, understanding what virtualization partner programs and tools are at NetApp, for example, is very useful. And many of these companies are already selling Microsoft software and surrounding services for Microsoft products. So if you’re VMware, what’s the message to these partners in the face of the Microsoft juggernaut?

Microsoft to partners: “You may not like to admit it, but you’re probably already in bed with us.”

VMware to partners: “Our hypervisor technology outperforms Hyper-V and Xen, especially at scale. And anyway, it’s not about the battle at the hypervisor. It’s about the V-services on top of the hypervisor – VMotion, Storage VMotion, DRS, etc.”

Interesting and what we all already know, or think we know. The scale issue is an interesting one – too soon for Hyper-V and who uses Xen? But also interestingly enough, no announcement or even talk about extending VMware management tools to other hypervisors. The point, as the VMware product marketing guy made a point of saying, is that the question they needed to answer used to be “Why Virtualization?” and now it’s “Why VMware?”.

One more tidbit – this survey run by VMware asking their customers:

What are the top 6 apps you are running on VMware today

• IIS
• Apache
• Active Directory
• SQL Server
• Sharepoint
• Exchange

That means, 5 of 6 are Microsoft applications. Certainly it makes it even more challenging for VMware to navigate a path here.

The change since 2004 – would have talked about why virtualize. And now why VMware. (Duh.)

Talking to partners – many of which already have a successful Microsoft business. How VMware enhances your existing Microsoft business.

Top 6 apps running on VMware today (5 of 6 are Microsoft applications)

• IIS
• Apache
• AD
• Sql server
• Sharepoint
• Exchange

Source: VMware survey

Esxi - VMware – true thin hypervisor; maximizes resources utilization (over 100% memory commitment – allows avg of 2:1 memory overcommit) – host system memory is usually the resource bottleneck – plus Advanced Scheduler runs VMs better under load and to a greater capacity (hard to show this part); performance acceleration – using binary translation (32bit), para-virtualization and Hardware Assist (for 64-bit)

(rvi – rapid virtualization indexing)

No parent partition that all hypervisors have to go through

Vs ms/xen

Parent partition – dom 0 => potentially problem at scale; i/o that could be a bottleneck

Hyper-v SPECjbb comparison

= 9 vms on VMware and hyper-v hypervisors

Outperform (CPU) by 50% - general purpose scheduler isn’t able to keep up? “got to be”

(cpu only test)

Also used VMmark – to demonstrate again that VMware is performance tuned and designed to run at scale vs Hyper-V

Size Does Matter:

Vmware ESXi: 32MB

Hyper-v – 2.6 GB

Xen – 1.2 GB

Hyper-V uses Microsoft Server Core – so the last two Patch Tuesdays had to make changes to Server Core (nothing to do with Hyper-V) but service interruption for Hyper-V.

VMware VMsafe – “Provides an unprecedented level of security” “virtual is more secure than Real” (uh oh – clearly didn’t read about the

*****************

VMware TEST:512 mb vms on server w/ 4gb ram –

7 vms - xensource (w/no memory overcommit)

6vms – hyper-v before error (w/no memory overcommit)

14vms - w/memory overcommit and management

Running sql io sim – heavy workloads

TCO – not just license; now ESXi is free – so hardware

809 - ESXi

871 – vi3 foundation ($995)

1168- vi3 enterprise ($5750)

1621 – hyper-v – 2x cost because of hw

Xen – 1618

Memory overcommit (89% in production vs. test/dev)

Survey – 37% of respondents at 2:1 RATIO OR HIGHER; real average is around 1.8: 1

*********************

This guy Mark sounds like a used car salesman:

“Always On, On Demand Data Center”

Hypervisor is very important but what is more important are the v-services on top of this. Manage shared, pooled resources. “Value Above the Hypervisor”

How does all this save “your customers” $$?

VMotion – saves cost on planned maintenance: no more overtime, no more time scheduling maintenance windows (see cost framework below)

10 (# of servers) x 6 (@ of updates) x [ (overtime cost 2hrs x $150/hr) + (scheduling downtime # of apps per server 15 x time spend scheduling per app 0.75 hr x $50/hr)] = $58,500

Same thing with using VMware Storage VMotion

Overtime cost + scheduling downtime + planning move + alternative tool cost - $68,750 (2.5 TeraBytes)

The Value of High Availability

- cost of lost business, lost work

- cost of lost productive time

4 hours of downtime x # of users per vm 10 x number of vms per host 15 x cost of user productive time $50/hr x failures per year in 10-host cluster 2 = $60K

(10 servers, 150 vms)

SAVINGS (using enterprise version)

Update management 149,760

HA 60K

DRS, VMotion Storage VMotion 187,250

808,259 – hw, power cooling, etc.

In a tale rich with incongruities, the Communist-run government of West Bengal State invited the Tata Group, a symbol of Indian capitalism, to set up its plant in an area called Singur. It acquired 1,000 acres from farmers on the company’s behalf.
As the project advanced, some farmers who had sold their land demanded it back. The main state-level opposition party, the Trinamool Congress, led protests demanding that the land be returned. Most people sympathetic to Tata accused the opposition of inducing the farmers to protest, while Tata’s critics said the farmers had legitimate grievances.

The issue simmered for months. But in recent days, protesters began surrounding the plant, blocking roads and preventing Tata workers from reaching the plant. “The existing environment of obstruction, intimidation and confrontation has begun to impact the ability of the company to convince several of its experienced managers to relocate and work in the plant,” Tata said in a statement on Tuesday.

The halt to the plant has caused many Indian business people to warn of a chilling effect on investment in the country. It is also unclear how Tata will be able to keep the Nano’s cost so low, since part of the affordable price reflects the company’s savings on the land in Singur.

There is a fundamental asymmetry between state and markets. It is easier to create markets than it is to create state capacity or to prevent its deterioration. Creating markets is a lot about letting go, establishing a reasonable policy framework, and allowing the natural hustling instinct to take over. In other words, hustling is the natural state. Building state capacity, on the other hand, is quite different. It involves overcoming collective action problems, mediating conflict, creating accountability mechanisms where outputs are multiple and fuzzy and links between inputs and outputs murky, and contending with the deep imprints of history. In Weber’s memorable words, building public institutions is like the “slow boring of hard boards”.

In that light, China’s task of improving its private sector seems easier to accomplish than India’s task of arresting institutional decline. So, while China and India can probably both count on more years of high growth, the odds still favour China pulling off that feat than India. That, and not just the meagre medal tally, should be what India mulls over after the Beijing Olympics.

It's easier to liberalise a functional state than it is to functionalise a dysfunctional one, of any ideological stripe.

It's become a big deal in IT security, too. Many corporate customers are demanding ROI models to demonstrate that a particular security investment pays off. And in response, vendors are providing ROI models that demonstrate how their particular security solution provides the best return on investment.

It's a good idea in theory, but it's mostly bunk in practice.

Before I get into the details, there's one point I have to make. "ROI" as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It's an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn't make sense in this context.

But as anyone who has lived through a company's vicious end-of-year budget-slashing exercises knows, when you're trying to make your numbers, cutting costs is the same as increasing revenues. So while security can't produce ROI, loss prevention most certainly affects a company's bottom line.

And a company should implement only security countermeasures that affect its bottom line positively. It shouldn't spend more on a security problem than the problem is worth. Conversely, it shouldn't ignore problems that are costing it money when there are cheaper mitigation alternatives. A smart company needs to approach security as it would any other business decision: costs versus benefits.

The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money.

Of course, that $1,000 has to reduce the chance of being robbed to zero in order to be cost-effective. If a security measure cuts the chance of robbery by 40 percent -- to 6 percent a year -- then you should spend no more than $400 on it. If another security measure reduces it by 80 percent, it's worth $800. And if two security measures both reduce the chance of being robbed by 50 percent and one costs $300 and the other $700, the first one is worth it and the second isn't.

The Data Imperative

The key to making this work is good data; the term of art is "actuarial tail." If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one.

Cybersecurity is considerably harder, because there just isn't enough good data. There aren't good crime rates for cyberspace, and we have a lot less data about how individual security countermeasures -- or specific configurations of countermeasures -- mitigate those risks. We don't even have data on incident costs.

One problem is that the threat moves too quickly. The characteristics of the things we're trying to prevent change so quickly that we can't accumulate data fast enough. By the time we get some data, there's a new threat model for which we don't have enough data. So we can't create ALE models.

But there's another problem, and it's that the math quickly falls apart when it comes to rare and expensive events. Imagine you calculate the cost -- reputational costs, loss of customers, etc. -- of having your company's name in the newspaper after an embarrassing cybersecurity event to be $20 million. Also assume that the odds are 1 in 10,000 of that happening in any one year. ALE says you should spend no more than $2,000 mitigating that risk.

So far, so good. But maybe your CFO thinks an incident would cost only $10 million. You can't argue, since we're just estimating. But he just cut your security budget in half. A vendor trying to sell you a product finds a Web analysis claiming that the odds of this happening are actually 1 in 1,000. Accept this new number, and suddenly a product costing 10 times as much is still a good investment.

It gets worse when you deal with even more rare and expensive events. Imagine you're in charge of terrorism mitigation at a chlorine plant. What's the cost to your company, in money and reputation, of a large and very deadly explosion? $100 million? $1 billion? $10 billion? And the odds: 1 in a hundred thousand, 1 in a million, 1 in 10 million? Depending on how you answer those two questions -- and any answer is really just a guess -- you can justify spending anywhere from $10 to $100,000 annually to mitigate that risk.

Or take another example: airport security. Assume that all the new airport security measures increase the waiting time at airports by -- and I'm making this up -- 30 minutes per passenger. There were 760 million passenger boardings in the United States in 2007. This means that the extra waiting time at airports has cost us a collective 43,000 years of extra waiting time. Assume a 70-year life expectancy, and the increased waiting time has "killed" 620 people per year -- 930 if you calculate the numbers based on 16 hours of awake time per day. So the question is: If we did away with increased airport security, would the result be more people dead from terrorism or fewer?

Caveat Emptor

This kind of thing is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They've jiggered the numbers so that they do.

This doesn't mean that ALE is useless, but it does mean you should 1) mistrust any analyses that come from people with an agenda and 2) use any results as a general guideline only. So when you get an ROI model from your vendor, take its framework and plug in your own numbers. Don't even show the vendor your improvements; it won't consider any changes that make its product or service less cost-effective to be an "improvement." And use those results as a general guide, along with risk management and compliance analyses, when you're deciding what security products and services to buy.

This essay previously appeared in CSO Magazine.