Blogger: Dan Blum

With the crisis in financial markets still unfolding, it is important to draw what lessons we can from the experience. Since the roots of the crisis lie in a monumental failure of risk management, it’s important to understand more about what happened, and then draw some parallels to our business risk management and  IT risk management situations.

The risk management failure in the housing market and on Wall Street had multiple interdependent dimensions:

• Mortgage lenders abandoned long standing prudent loan practices. They made too many loans that buyers might not be able to repay. Exotic instruments like ARMs, option ARMs, and interest only loans proliferated. In many cases, all pretense of lending standards were abandoned, so-called “liar loans” approved.
• Capital was grossly over-leveraged. Mortgage lenders and other financial services packaged loans into securities, which they sold to raise capital to support more lending. Real capital reserve requirements to back loans were reduced. Of course, if borrowers could not repay loans, all or parts of the derivative securities would become worthless.
• Risk was aggregated at Fannie Mae, Freddie Mac, and mortgage loan insurance companies. These companies bought or insured some mortgage loans, providing something of a backstop should loans fail. Government sponsored enterprises (GSEs) Fannie and Freddie in turn became over-leveraged and securities that they sold were in turn repackaged in the murky brew of mortgage-backed securities called collateralized debt obligations (CDOs) and other exotic instruments returning generous yields.
• Non-Caveat Emptor. Institutional wealth funds and financial services firms who should have known better bought securities that had been deliberately structured to obfuscate risk. They bought securities they didn’t understand with buried tranches of toxic subprime loans..

It was a great Ponzi scheme – one that kept working as long as housing prices were going up; the recipients of subprime loans could always flip that house to the next buyer. Everyone made money. As Chuck Prince of Citigroup famously put it during a July, 2007 interview: “So long as the music is playing, you’ve got to keep dancing. We’re still dancing.” But one month later, the music stopped. Since then, Citigroup and other financial institutions have taken massive writeoffs with more to come. Wall Street titans like Bear Sterns, Lehman Brothers, Merrill Lynch, and AIG have fallen or been bought out.

What can we learn from this risk management debacle?

As business risk managers and investors, we should ask questions like these:

• Does the executive incentive structure of the company encourage managers to dance around risk? Many Wall Street firms paid senior managers 5 times their salary in bonuses tied to annual growth alone.
• Is the company over-leveraged? Is it borrowing too much money and betting it on ventures with uncertain outcomes?
• Are financial models used for risk management realistic? Earlier, I described the mortgage market of the past few years as a Ponzi scheme, where risk management models must have assumed prices would keep rising. Unlike the dotcom boom whose demise many predicted, very few in the industry foresaw the sharp declines to come in housing prices and sales volumes. Historically, the U.S. housing market has been a steadily rising one, but on the other hand the 2000s saw unprecedented rates of price increases. In reality, what goes up must come down.
• Has your company’s risk council ever performed worst case scenario analysis and built adequate reserves? In the days before economics emerged as a would-be “hard” deterministic science, business leaders may have been more cautious, more aware of and more accepting of uncertainty. Events like the Great Tulip Bubble came once in decades or centuries – not every few years. Note that legendary investor George Soros has proposed a Theory of Reflexivity that, if true, helps explain the recent extremes of boom and bust cycles. This theory holds that market participants model market behaviors based on self-interest, and for a time, their manipulations change the reality of the market – until gravitational forces bring it back to earth. Has the music of ephemeral success played to the backbeat of deterministic-sounding economic models gone to your heads and infected your risk management models?
• Are cost cutting efforts pursued blindly? Outsourcing and other forays into treacherous global waters may be giving away the crown jewels. Smart companies cut costs, but they do it in smart ways. Smart companies think like intelligence agencies as they parcel out work to different partners with varying levels of dependability, and they check on those partners.

Risk management failures can also occur at the more technical level of IT security. As IT risk managers, we might ask questions like these:

• Are the accounting and financial systems your IT department supports under adequate control? As Fred Cohen wrote in one of our documents: “Many companies use computers to manage financial systems, and despite the Sarbanes-Oxley Act (SOX) claims about accounts being properly kept, there are many attacks on financial systems that remain. For example, most of the largest financial systems in the world running on common financial databases do not use double-entry bookkeeping and are thus susceptible to all manner of frauds by insiders.” We find it troubling that a prudent control dating back to the 12th century is going out of style in the name of convenience and cost cutting. Kind of like credit checking became anachronistic during the housing bubble, eh?
• Is the “separation” in your “separation of duty” (SoD) for real? Sure the SOX auditors are looking for SoD, and maybe you have different administrators with different accounts maintaining different systems or functions. But when they say Western civilization may be but one weak password from collapse they’re not lying. Look what happened to Sarah Palin’s email account! Weak and straggly SoD is a problem across all critical IT systems where deperimiterization and server consolidation may be bringing down protective barriers, identity management is weak, and strong process controls (e.g., where two people must sign on, one perform a critical operation such as backbone router reconfiguration, and the second observe) abandoned in the name of expediency.
• Are risks being aggregated to unacceptable levels in centralized control systems? There are many ways that risks aggregate within enterprise IT infrastructures as we pursue automation and cost cutting. Network risks aggregate when centralized domain name system control is implemented. Application risks aggregate when common infrastructure is shared among applications. And enterprises aggregate platform risks when they use low-assurance endpoints, authentication, and directory systems with single sign-on to access large numbers of resources and don’t separate high consequence systems.
• Non-caveat emptor: Has IT security really done the worst case consequence analysis, attack graphs, and vulnerability analysis to know when putting more eggs in a supposedly stronger basket aggregates risks to an unacceptable level? Or are you depending only on vendor claims about some black box appliance equivalent of a risk-obfuscated CDO security? Caveat emptor (buyer beware) again! (The good news is we’ll keep talking about promoting vendor and product rating systems so you don’t have to do all the detailed product analysis yourself, but that’s another post.)

There are many parallels between the monumental risk management failure in the financial markets, and the probable weaknesses in our day to day business risk management and IT risk management. Abandonment of prudent practices for profit; excessive leverage and centralization; ill-constructed risk analysis models; risk obfuscation; and a failure of caveat emptor seem to be common problems. Please take this as a wakeup call to sharpen up the risk management thinking, process, and execution.

Ten days before VMworld and VMware still can’t get good press. First their CEO, Diane Greene, gets ousted, then a high-profile licensing bug is found and now the Director of R&D, Richard Sarwal, leaves his $1.25 million salary after just 7 months. (Note to self: get into R&D) It will be interesting to take the pulse of the VMware community at the show and in person. And in the meantime, Microsoft Hyper-V comes out of the gate with customers already touting its benefits.

The hypervisor is the “new” operating system. If you didn’t think that before, take a look at Red Hat’s purchase of Qumranet for $107 million. With Qumranet, Red Hat gets KVM, described by CTO Brian Stevens as an extension to the Linux kernel that allows it to be used as a bare-metal hypervisor, running directly on the underlying hardware and hosting guest operating systems. But according to Brian Madden, the “press” around the purchase is all focusing on the not-so-interesting part. Along with KVM, the SolidICE product includes Spice, a remote display protocol for VDI.

I wonder if this will be like Symantec buying Altiris or Microsoft buying Softricity, where the portion that we care about sort of loses focus as The Borg concentrates on the parts of the acquired technology that are more relevant to them?

(I’m a sucker for quotes that reference The Borg)

Network World publishes “10 open source companies to watch”. On the list, Qumranet!

Also on the list: Kickfire, Marketcetera, Vyatta, Sonatype, Untangle, XAware, SnapLogic, Acquia and Openmoko. What’s best about the list: Matt Asay gives it a thumbs up.

In a tale rich with incongruities, the Communist-run government of West Bengal State invited the Tata Group, a symbol of Indian capitalism, to set up its plant in an area called Singur. It acquired 1,000 acres from farmers on the company’s behalf.
As the project advanced, some farmers who had sold their land demanded it back. The main state-level opposition party, the Trinamool Congress, led protests demanding that the land be returned. Most people sympathetic to Tata accused the opposition of inducing the farmers to protest, while Tata’s critics said the farmers had legitimate grievances.

The issue simmered for months. But in recent days, protesters began surrounding the plant, blocking roads and preventing Tata workers from reaching the plant. “The existing environment of obstruction, intimidation and confrontation has begun to impact the ability of the company to convince several of its experienced managers to relocate and work in the plant,” Tata said in a statement on Tuesday.

The halt to the plant has caused many Indian business people to warn of a chilling effect on investment in the country. It is also unclear how Tata will be able to keep the Nano’s cost so low, since part of the affordable price reflects the company’s savings on the land in Singur.

There is a fundamental asymmetry between state and markets. It is easier to create markets than it is to create state capacity or to prevent its deterioration. Creating markets is a lot about letting go, establishing a reasonable policy framework, and allowing the natural hustling instinct to take over. In other words, hustling is the natural state. Building state capacity, on the other hand, is quite different. It involves overcoming collective action problems, mediating conflict, creating accountability mechanisms where outputs are multiple and fuzzy and links between inputs and outputs murky, and contending with the deep imprints of history. In Weber’s memorable words, building public institutions is like the “slow boring of hard boards”.

In that light, China’s task of improving its private sector seems easier to accomplish than India’s task of arresting institutional decline. So, while China and India can probably both count on more years of high growth, the odds still favour China pulling off that feat than India. That, and not just the meagre medal tally, should be what India mulls over after the Beijing Olympics.

It's easier to liberalise a functional state than it is to functionalise a dysfunctional one, of any ideological stripe.

Companies think they own their reputation, but in reality they don’t. A reputation is the aggregate of the popular opinion about you. Opinions, or thoughts, belong to an individual, true or not, and a company doesn’t own a person’s thoughts, therefore a company doesn’t own its reputation. QED.

Yes.  Absolutely.  In fact, there are already changes in the works to the FAIR model that reflect this line of thinking that will allow us to approach reputation damage in a much more rational manner that anything else I’ve seen to date.

Second, RE:  Hansei & Kaizen, Richard left the following comment.

I don’t agree with your view on Gemba even if we live in a virtual world. Look into any company’s wiring closet and you’ll immediately see a reflection in its maturity from the state of the equipment, the labeling / documentation and overall neatness. “Man with messy wiring closet, will have messy virtual servers.”

However, the true benefit in Gemba is not in the actual visual inspection. It is in in the journey from your desk to the data center / wiring closet.

I agree that the benefit is in the journey.  I can’t see the wiring closet as the main destination (I just don’t see it as a useful prior).  Maybe I wasn’t clear, or was taking for granted that you guys have been reading the blog for the past 2 years, but the journey needs to be to the LOB that owns the application.  The example most given when describing Gemba is going to the production line to look at the issue that causes a problem in the ability to create and sell a car.  The “security” journey is not to the wiring closet, but to the system itself and the logs that we have for the system and whatever network-based controls might be applicable.  And we, as an industry, are just starting to understand that this “security” is only part of the picture.  The whole picture is represented by the factors that create risk.

And for our “risk journey” that security journey is only a one of serveral useful pieces of prior information for use in analysis.  For risk we have to also journey back to the “production line”, or, in our case, to the application/LOB owner.  It may also be to corporate counsel, to marketing, to all sorts of other places in the enterprise because probable losses (a necessary measurement we need in order to understand risk) may come from many different sources in the organization.  For those with FAIR knowledge, think of the six forms of loss to get an idea of what sorts of journeys we need to make.

This is why tomorrow’s post is designed to look at what should we be reflecting about, and what is needed for reflection.

Hint:  our models for risk & risk management can give us an idea of how to create structure around Hansei for the IRM program.

HANSEI - WHAT IS “RELENTLESS REFLECTION?” - And why we’re talking about it in the context of Risk Analysis.

Recall from yesterday’s post about how I got to thinking about the concept of Hansei-Kaizen, “relentless reflection” and “continuous improvement” and how we might apply that to risk management.  It’s a concept born of Toyota and is, in some way, the foundation for “Lean” production.

Call me biased, but I think that Hansei - the act of ‘relentless reflection’ made structured is the analytical function.  And I hate to debate (post-mortem) the father of Toyota quality success when he says that Hansei is the “check” in Plan/Do/Check/Act, but I think that Hansei also applies to the “Plan” of the P/D/C/A or Deming cycle.

You’ll recall the P/D/C/A cycle can be thought of even as an implementation of Scientific Method, in that it is Observation & Hypothesis Creation (P), Experiment (D), Analysis (Check), and Act (Revise/New Hypothesis, etc…).  Well then as such, the Hypothesis creation involves creating a model or creating an expected outcome for data using the currently accepted model.

So in our industry there is an opportunity for Relentless Reflection in both the Observation and Hypothesis (Plan) creation steps, and the Check step.  We create an estimate for control strength, or probable losses in the context of risk- then we go to Experiment step.  That hypothesis can be put it into production, have an audit, have a penetration test, whatever, in the context of the Do step.  BTW - using Hansei/Analytics in Plan is one way that strong analytical functions can really make penetration testing more useful - as a means to test the estimates and inputs into a model.  It’s Penetration Testing 2.0!  (<- tongue fully in cheek, yes)

Those who are versed in the reasons to merge Six Sigma and Lean together are probably already seeing where I’m going with this today.  But before you think that a simple DMAIC function is all that is needed to create proper “Hansei”, let me encourage you to keep reading.

Now if the analytical function can said to be “reflection”, why must it be relentless?

One word.  Change. There are essentially four separate “landscapes” or sources of change that we face (more on those tomorrow).  But anyone who has tried to manage system compliance, log management or policy exceptions knows that change is possibly the most difficult thing we security professionals must manage.  And when you think about it, there aren’t too many other business functions like information security where significant visibility and insight about the environment is needed for “complete” information (get bullish on Log Management is my recommendation).

HANSEI STEPS ADAPTED TO INFORMATION SECURITY

This is one of those quality control concepts that we can mangle adopt.  At Toyota, Hansei-Kaizen includes the following basic steps:

1. Initial problem perception
2. Clarify the problem
3. Locate area/point of cause
4. Investigate root cause (using an ask why 5 times approach)
5. Countermeasure
6. Evaluate
7. Standardize

Now it’s important to note that part of this includes the concept of Go See For Yourself, called “Gemba“.  Gemba can be translated as “the actual place” or “the place where virtue or truth is found.” At Toyota this might mean going to the shop floor to see the issue at hand in the production line.  But for us, that’s a problem because we live in the virtual world.  There’s usually not much use in hanging out in the wiring closets to try to see the problems.

But if you combine the concept of Gemba with the concept of “Nemawashi” –the process of discussing problems and potential solutions with all those affected- we can forge a similar concept using risk analysis.  That is discussing the issue and the risk associated with an issue (what some people would call “risk management”) with the business/LOB/data owner and let them accept authority and the risk decision.  We, the risk analyst, our goal is simply to perform items 1-5 (presenting countermeasure options that include transferring or accepting risk).  By going to the line of business and involving them, responsibility is shared.  Also, if you structure organizational behavior right, personal risk is transferred!

This sort of approach is also in harmony with concepts like “mutual ownership of problems,” or “genchi genbutsu,” (solving problems at the source instead of behind desks), and the “kaizen mind,” (an unending sense of crisis behind the company’s constant drive to improve).

One of the criticisms I have with the way most people try to implement DMAIC into “Lean”

REQUIREMENTS

Now to get this done, I really see three significant requirements.

1.)  A change in political structure.

2.)  Models that provide consistent, defensible analysis.

3.)  A Quantitative approach.  This means using actual units of measurement (not just amorphous percents, ordinal scales, etc.)  for risk and it’s subsequent factors.  Sure there are times when Q&D qualitative approaches are acceptable, but policy should be to have quantitative analysis whenever and wherever possible.

That last item - the quantitative approach - is really quite important.  And the reasons why will be discussed further in tomorrow’s post:

“What should we be reflecting about? & What is needed for reflection?”

P.S.  Your comments and suggestions, as always, are welcome.

P.P.S  Those who may be familiar with Lean/SixSigma/Kaizen sorts of mashups may be thinking - “hey, an Analytical step is built into SixSigma”.  Well, yes there is some prevision for analytical functions based on statistics, but I find SixSigma geared towards creating a State of Knowledge about operational processes, not towards creating a State of Wisdom for CISO’s around security & risks “big questions”.  In otherwords, the analytical function in DMAIC is in the context of Kaizen, and a different step than “reflective” analytics.

An interesting excerpt:

“CIOs have emphasized to us that they are buying on a need versus want basis, are often downsizing deals to fit with current budget constraints…In fact, contrary to general tightening in spending, purchases with an especially compelling ROI are being accelerated in the current environment.”

Hmm. Certainly we all understand prioritizing what to buy on need versus want– my friend who runs an art gallery that has only sold one piece in the past 2 months can certainly explain it. I “need” that Picasso? But does it take the entire economy slowing down before CIOs, even at Fortune 100 companies, to focus on ROI? So it’s not surprising what showed up at the top of the list for spending priorities for 2008-2009:

1. Server Virtualization
2. Server Consolidation
3. Cost Cutting

At the bottom of the list, grid computing and on-demand computing.

Compare this to last year’s spending survey where the top 10 priorities by rank were:

1. Applications integration
2. Security
3. Cost Cutting
4. BI
5. ERP
6. Web-based app development
7. Datacenter consolidation
8. Disaster Recovery
9. Compliance/risk management
10. Identity and access management

So in one year, the very hot “server virtualization” (and quite similar server consolidation) jumped to the top of the spending priority list. Can anyone have predicted just how much mindshare virtualization would capture in such a short time? Virtualization is not a new concept; it just seems that way. What will be # 1 next year?

ShareThis