Blogger: Dan Blum

With the crisis in financial markets still unfolding, it is important to draw what lessons we can from the experience. Since the roots of the crisis lie in a monumental failure of risk management, it’s important to understand more about what happened, and then draw some parallels to our business risk management and  IT risk management situations.

The risk management failure in the housing market and on Wall Street had multiple interdependent dimensions:

• Mortgage lenders abandoned long standing prudent loan practices. They made too many loans that buyers might not be able to repay. Exotic instruments like ARMs, option ARMs, and interest only loans proliferated. In many cases, all pretense of lending standards were abandoned, so-called “liar loans” approved.
• Capital was grossly over-leveraged. Mortgage lenders and other financial services packaged loans into securities, which they sold to raise capital to support more lending. Real capital reserve requirements to back loans were reduced. Of course, if borrowers could not repay loans, all or parts of the derivative securities would become worthless.
• Risk was aggregated at Fannie Mae, Freddie Mac, and mortgage loan insurance companies. These companies bought or insured some mortgage loans, providing something of a backstop should loans fail. Government sponsored enterprises (GSEs) Fannie and Freddie in turn became over-leveraged and securities that they sold were in turn repackaged in the murky brew of mortgage-backed securities called collateralized debt obligations (CDOs) and other exotic instruments returning generous yields.
• Non-Caveat Emptor. Institutional wealth funds and financial services firms who should have known better bought securities that had been deliberately structured to obfuscate risk. They bought securities they didn’t understand with buried tranches of toxic subprime loans..

It was a great Ponzi scheme – one that kept working as long as housing prices were going up; the recipients of subprime loans could always flip that house to the next buyer. Everyone made money. As Chuck Prince of Citigroup famously put it during a July, 2007 interview: “So long as the music is playing, you’ve got to keep dancing. We’re still dancing.” But one month later, the music stopped. Since then, Citigroup and other financial institutions have taken massive writeoffs with more to come. Wall Street titans like Bear Sterns, Lehman Brothers, Merrill Lynch, and AIG have fallen or been bought out.

What can we learn from this risk management debacle?

As business risk managers and investors, we should ask questions like these:

• Does the executive incentive structure of the company encourage managers to dance around risk? Many Wall Street firms paid senior managers 5 times their salary in bonuses tied to annual growth alone.
• Is the company over-leveraged? Is it borrowing too much money and betting it on ventures with uncertain outcomes?
• Are financial models used for risk management realistic? Earlier, I described the mortgage market of the past few years as a Ponzi scheme, where risk management models must have assumed prices would keep rising. Unlike the dotcom boom whose demise many predicted, very few in the industry foresaw the sharp declines to come in housing prices and sales volumes. Historically, the U.S. housing market has been a steadily rising one, but on the other hand the 2000s saw unprecedented rates of price increases. In reality, what goes up must come down.
• Has your company’s risk council ever performed worst case scenario analysis and built adequate reserves? In the days before economics emerged as a would-be “hard” deterministic science, business leaders may have been more cautious, more aware of and more accepting of uncertainty. Events like the Great Tulip Bubble came once in decades or centuries – not every few years. Note that legendary investor George Soros has proposed a Theory of Reflexivity that, if true, helps explain the recent extremes of boom and bust cycles. This theory holds that market participants model market behaviors based on self-interest, and for a time, their manipulations change the reality of the market – until gravitational forces bring it back to earth. Has the music of ephemeral success played to the backbeat of deterministic-sounding economic models gone to your heads and infected your risk management models?
• Are cost cutting efforts pursued blindly? Outsourcing and other forays into treacherous global waters may be giving away the crown jewels. Smart companies cut costs, but they do it in smart ways. Smart companies think like intelligence agencies as they parcel out work to different partners with varying levels of dependability, and they check on those partners.

Risk management failures can also occur at the more technical level of IT security. As IT risk managers, we might ask questions like these:

• Are the accounting and financial systems your IT department supports under adequate control? As Fred Cohen wrote in one of our documents: “Many companies use computers to manage financial systems, and despite the Sarbanes-Oxley Act (SOX) claims about accounts being properly kept, there are many attacks on financial systems that remain. For example, most of the largest financial systems in the world running on common financial databases do not use double-entry bookkeeping and are thus susceptible to all manner of frauds by insiders.” We find it troubling that a prudent control dating back to the 12th century is going out of style in the name of convenience and cost cutting. Kind of like credit checking became anachronistic during the housing bubble, eh?
• Is the “separation” in your “separation of duty” (SoD) for real? Sure the SOX auditors are looking for SoD, and maybe you have different administrators with different accounts maintaining different systems or functions. But when they say Western civilization may be but one weak password from collapse they’re not lying. Look what happened to Sarah Palin’s email account! Weak and straggly SoD is a problem across all critical IT systems where deperimiterization and server consolidation may be bringing down protective barriers, identity management is weak, and strong process controls (e.g., where two people must sign on, one perform a critical operation such as backbone router reconfiguration, and the second observe) abandoned in the name of expediency.
• Are risks being aggregated to unacceptable levels in centralized control systems? There are many ways that risks aggregate within enterprise IT infrastructures as we pursue automation and cost cutting. Network risks aggregate when centralized domain name system control is implemented. Application risks aggregate when common infrastructure is shared among applications. And enterprises aggregate platform risks when they use low-assurance endpoints, authentication, and directory systems with single sign-on to access large numbers of resources and don’t separate high consequence systems.
• Non-caveat emptor: Has IT security really done the worst case consequence analysis, attack graphs, and vulnerability analysis to know when putting more eggs in a supposedly stronger basket aggregates risks to an unacceptable level? Or are you depending only on vendor claims about some black box appliance equivalent of a risk-obfuscated CDO security? Caveat emptor (buyer beware) again! (The good news is we’ll keep talking about promoting vendor and product rating systems so you don’t have to do all the detailed product analysis yourself, but that’s another post.)

There are many parallels between the monumental risk management failure in the financial markets, and the probable weaknesses in our day to day business risk management and IT risk management. Abandonment of prudent practices for profit; excessive leverage and centralization; ill-constructed risk analysis models; risk obfuscation; and a failure of caveat emptor seem to be common problems. Please take this as a wakeup call to sharpen up the risk management thinking, process, and execution.

In any case, if you do pentests, think about all the RECENT cases where you break in to a major corporation through:

• a Solaris system with Internet-exposed telnet with a guessable password OR a telnet vulnerability (circa 1994!)
• an exposed VPN appliance with a manufacturer's administrator password
• a router with default "enable" password
• or, something else entirely - but something that rivals the above example in its unparalleled, unbelievable, abysmal, deep idiocy.

Indeed, many of my pentesting friends still report plenty of such cases (one was also featured in the presentation mentioned above). Whenever I hear about it from a pentester, I always ask:

Do you think "somebody bad" had already passed through the hole you just discovered?

Maybe an hour ago, a day ago - or a year ago?!

I cannot see how the answer can be "no."

Even though pentesters usually don't focus on forensics (no time for this), it is not uncommon to notice "your predecessor's" intrusion traces while you break through systems, "plant flags", change screen backgrounds [for the admins to notice that you've been there...], etc.

Let's think what this situation really means? Here are the choices I see:

1. Nobody discovered the hole - a law of large  numbers (aka "dumb luck") have "shielded" the company from an incident. Yes, Virginia, dumb luck IS a security strategy for some companies... AND it works for them.
2. It was discovered, but not used/abused by the attacker - maybe he was busy hacking other systems, or saved this for later and never came back due to his ADD. Congratulation, you win! The immense power of dumb luck wrapped you in a protective "security" blanket ... again :-)
3. It was discovered; the attacker went in, looked around and compromised a few others systems, but found nothing of interest (no low hanging fruits)  - and he was not a bot herder. Again, you win. Next time you are in Vegas, bet on "00."
4. It was discovered; the attacker went in and deployed a bot on "your" system - given how many botnets are there, this situation is clearly acceptable to many organizations. In this case, dumb luck strategy, apparently, still work: so they use your box to spam and phish somebody else ... big deal!
5. It was discovered; the attacker went in and stole all your credit card information (it is now for sale) - even in this case, the user of "the dumb luck strategy" still "wins" (in some perverse sense)! Unless and until the stolen information IS tracked back to you OR a friendly neighborhood PCI auditor come and jams a broomstick up your ..., you can still continue to be stupid at your leisure and ignore basic security practices.
6. It was discovered; the attacker went in and stole your CEO's Inbox, including the email related to his affair (it is now on CNN) - now, in this case, you lose AND it is time to stop being stupid! Welcome to the "0wned world." Time to launch (relaunch?) your security program and get serious.

What does this teach us about RISK? The lesson here is important:

• For a security professional, an Internet-exposed system with "root/root" is an obvious HUGE risk!
• For your boss's boss's boss, it is NOT!

This is exactly why I think that the most critical problem in security today is METRICS. Metrics that a) work AND mean something to decision makers and b) can be clearly communicated to said decision makers [BTW, a) and b) are two separate problems.] Metrics that cover not only threats and vulnerabilities we face, but also the effectiveness of security countermeasures we deploy. Metrics you can act on - and ones your boss (and his boss) will act on. Metrics that lead to correct decisions about which risks to accept, which to  mitigate (all while knowing with what efficiency such mitigation occurs) and which to transfer.

Until that time, the dreaded "C-word" (compliance) will trump "the other C-word" (common sense) as a driver for security ... and we will continue to live in the "0wned world."

Possibly related posts:

• Risk vs Risk