Blogger: Dan Blum

With the crisis in financial markets still unfolding, it is important to draw what lessons we can from the experience. Since the roots of the crisis lie in a monumental failure of risk management, it’s important to understand more about what happened, and then draw some parallels to our business risk management and  IT risk management situations.

The risk management failure in the housing market and on Wall Street had multiple interdependent dimensions:

• Mortgage lenders abandoned long standing prudent loan practices. They made too many loans that buyers might not be able to repay. Exotic instruments like ARMs, option ARMs, and interest only loans proliferated. In many cases, all pretense of lending standards were abandoned, so-called “liar loans” approved.
• Capital was grossly over-leveraged. Mortgage lenders and other financial services packaged loans into securities, which they sold to raise capital to support more lending. Real capital reserve requirements to back loans were reduced. Of course, if borrowers could not repay loans, all or parts of the derivative securities would become worthless.
• Risk was aggregated at Fannie Mae, Freddie Mac, and mortgage loan insurance companies. These companies bought or insured some mortgage loans, providing something of a backstop should loans fail. Government sponsored enterprises (GSEs) Fannie and Freddie in turn became over-leveraged and securities that they sold were in turn repackaged in the murky brew of mortgage-backed securities called collateralized debt obligations (CDOs) and other exotic instruments returning generous yields.
• Non-Caveat Emptor. Institutional wealth funds and financial services firms who should have known better bought securities that had been deliberately structured to obfuscate risk. They bought securities they didn’t understand with buried tranches of toxic subprime loans..

It was a great Ponzi scheme – one that kept working as long as housing prices were going up; the recipients of subprime loans could always flip that house to the next buyer. Everyone made money. As Chuck Prince of Citigroup famously put it during a July, 2007 interview: “So long as the music is playing, you’ve got to keep dancing. We’re still dancing.” But one month later, the music stopped. Since then, Citigroup and other financial institutions have taken massive writeoffs with more to come. Wall Street titans like Bear Sterns, Lehman Brothers, Merrill Lynch, and AIG have fallen or been bought out.

What can we learn from this risk management debacle?

As business risk managers and investors, we should ask questions like these:

• Does the executive incentive structure of the company encourage managers to dance around risk? Many Wall Street firms paid senior managers 5 times their salary in bonuses tied to annual growth alone.
• Is the company over-leveraged? Is it borrowing too much money and betting it on ventures with uncertain outcomes?
• Are financial models used for risk management realistic? Earlier, I described the mortgage market of the past few years as a Ponzi scheme, where risk management models must have assumed prices would keep rising. Unlike the dotcom boom whose demise many predicted, very few in the industry foresaw the sharp declines to come in housing prices and sales volumes. Historically, the U.S. housing market has been a steadily rising one, but on the other hand the 2000s saw unprecedented rates of price increases. In reality, what goes up must come down.
• Has your company’s risk council ever performed worst case scenario analysis and built adequate reserves? In the days before economics emerged as a would-be “hard” deterministic science, business leaders may have been more cautious, more aware of and more accepting of uncertainty. Events like the Great Tulip Bubble came once in decades or centuries – not every few years. Note that legendary investor George Soros has proposed a Theory of Reflexivity that, if true, helps explain the recent extremes of boom and bust cycles. This theory holds that market participants model market behaviors based on self-interest, and for a time, their manipulations change the reality of the market – until gravitational forces bring it back to earth. Has the music of ephemeral success played to the backbeat of deterministic-sounding economic models gone to your heads and infected your risk management models?
• Are cost cutting efforts pursued blindly? Outsourcing and other forays into treacherous global waters may be giving away the crown jewels. Smart companies cut costs, but they do it in smart ways. Smart companies think like intelligence agencies as they parcel out work to different partners with varying levels of dependability, and they check on those partners.

Risk management failures can also occur at the more technical level of IT security. As IT risk managers, we might ask questions like these:

• Are the accounting and financial systems your IT department supports under adequate control? As Fred Cohen wrote in one of our documents: “Many companies use computers to manage financial systems, and despite the Sarbanes-Oxley Act (SOX) claims about accounts being properly kept, there are many attacks on financial systems that remain. For example, most of the largest financial systems in the world running on common financial databases do not use double-entry bookkeeping and are thus susceptible to all manner of frauds by insiders.” We find it troubling that a prudent control dating back to the 12th century is going out of style in the name of convenience and cost cutting. Kind of like credit checking became anachronistic during the housing bubble, eh?
• Is the “separation” in your “separation of duty” (SoD) for real? Sure the SOX auditors are looking for SoD, and maybe you have different administrators with different accounts maintaining different systems or functions. But when they say Western civilization may be but one weak password from collapse they’re not lying. Look what happened to Sarah Palin’s email account! Weak and straggly SoD is a problem across all critical IT systems where deperimiterization and server consolidation may be bringing down protective barriers, identity management is weak, and strong process controls (e.g., where two people must sign on, one perform a critical operation such as backbone router reconfiguration, and the second observe) abandoned in the name of expediency.
• Are risks being aggregated to unacceptable levels in centralized control systems? There are many ways that risks aggregate within enterprise IT infrastructures as we pursue automation and cost cutting. Network risks aggregate when centralized domain name system control is implemented. Application risks aggregate when common infrastructure is shared among applications. And enterprises aggregate platform risks when they use low-assurance endpoints, authentication, and directory systems with single sign-on to access large numbers of resources and don’t separate high consequence systems.
• Non-caveat emptor: Has IT security really done the worst case consequence analysis, attack graphs, and vulnerability analysis to know when putting more eggs in a supposedly stronger basket aggregates risks to an unacceptable level? Or are you depending only on vendor claims about some black box appliance equivalent of a risk-obfuscated CDO security? Caveat emptor (buyer beware) again! (The good news is we’ll keep talking about promoting vendor and product rating systems so you don’t have to do all the detailed product analysis yourself, but that’s another post.)

There are many parallels between the monumental risk management failure in the financial markets, and the probable weaknesses in our day to day business risk management and IT risk management. Abandonment of prudent practices for profit; excessive leverage and centralization; ill-constructed risk analysis models; risk obfuscation; and a failure of caveat emptor seem to be common problems. Please take this as a wakeup call to sharpen up the risk management thinking, process, and execution.

You'll have to pardon me this morning if the round-up seems a bit off. I'm still a little stunned at the spectacle of an arena full of (seemingly sober and sane) adults chanting, "Drill, baby, drill".

So let's see, what's in the news? Well, last night Republicans trotted out a Massachusetts venture capitalist and governor, the former mayor of New York City, former executives of eBay and HP, and an Alaskan neophyte pol who as mayor of a small town delivered $4,000 in federal pork for every man, woman, and child, in railing against coastal elites and Washington politics, while supporting a candidate who's been in the Senate for 26 years.

MetroFi was one of the three most prominent pure play metro-scale Wi-Fi firms, if you count EarthLink's municipal wireless division as a separate operation, and Kite Networks, which was a subsidiary of a larger telecom firm. Each company had made a unique network hardware choice--MetroFi, SkyPilot; Kite, Strix; and EarthLink Tropos plus Motorola--and each had a sort of specialty. Interestingly, a fifth firm, BelAir powers Toronto (a small but super-fast Wi-Fi network) and Minneapolis (the only putatively completed large-city Wi-Fi network), and will be behind Cablevision's nearly $350m New York Wi-Fi plan.

MetroFi was the only major firm to back ad-supported no-fee access, coupled with paid, no-ads service, and higher tiered commercial offerings. They built mostly smaller cities, with Portland being their only real big city win. The firm began with the notion of building Wi-Fi out gradually as a way to provide broadband in communities that lacked service, with no municipal involvement. That plan required sparser networks and typically a home signal booster designed by SkyPilot. (Kite mostly focused on the Southwest; EarthLink on big cities.)

EarthLink was in many ways largely responsible for the mess that all Wi-Fi providers found themselves in last year by offering to build Philadelphia's network back in 2005 at no cost to the city--in fact, paying the city and the local utility fees. That set the stage for nearly all the RFPs that followed where, if EarthLink were a bidder or the city was aware of the alternatives, the notion was that no city dollars would be spent, even if taxpayer money wasn't "at risk"--that is, even if a city could save money by switching current line items in their telecom and data budget to a wireless network.

Haas noted via email that MetroFi has been working towards anchor commitments by cities for nearly two years, but the inertia of those early networks led municipalities to reject those options. In Toledo, where MetroFi had negotiated an anchor commitment, a change in administration led a new mayor to retreat from the plan.

Is there a future for metro-scale Wi-Fi? Yes. With thoughtfully constructed, outdoor-focused deployments centered on municipal purposes, with public access a secondary issue, it seems like these networks could still provide an inexpensive way for relatively high bandwidth compared to the alternative of cell data networks.

However, that advantage is likely short lived in larger markets. The near-future certainty now that there will be multiple provides offering wired broadband speed service starting later this year with Sprint/Clearwire's WiMax, and continuing through into 2012 with significant network buildout by Verizon and AT&T in several bands (including their new 700 MHz holdings).

While Sprint/Clearwire is talking about 120m to 140m homes passed by 2010 with their network, obviously focusing only on major markets, many of the 700 MHz licenses purchased by AT&T and Verizon carry buildout requirements with penalties. So cities outside the top 100 population markets and rural areas will still see some benefit. In those mid-tier markets, there's also the 3.65 GHz band for shared licensed use, which is a model that Azulstar is pursuing with new WiMax deployments, as I wrote about recently.

Competition will likely push the cost of mobile broadband far below its $60 per month 2-year contract rate of today, which then would beg the question why a city or county with good commercial coverage would need to build its own Wi-Fi network. There are still plenty of reasons to build dedicated, first-responder 4.9 GHz public safety networks, of course.

I've always described Wi-Fi on a metropolitan scale as the best, worst technology. The best, because everyone has Wi-Fi in their laptops and increasingly in handhelds and gadgets. The worst, because the technology is absolutely not designed for the purpose, unlike CDMA and GSM evolved cell standards and mobile WiMax.

It's possible that in the long term, looking five years out, that Wi-Fi on a metro-scale will only be needed in small towns, odd markets, and for highly particular purposes. Or, perhaps in a bit of irony, where companies like Cablevision feel Wi-Fi is necessary to retain the loyalty of their highly wired customer base.

Portland, Ore., considers its options with MetroFi's stalled network: The city of Portland alerted MetroFi in February that it considers the company "in default of contract," according to the (Portland) Oregonian. MetroFi told the paper that his firm won't be finishing the network without "financial support form the city and left open the possibility MetroFi will shut off the entire system." CEO Chuck Haas also seems to have sworn off ad-supported Wi-Fi, something the company switched to years ago, deciding there's truly not enough revenue there to turn a profit. Local group Personal Telco may move into a more leading role, given their steady work while MetroFi fiddled with their business model.

The Oregonian's blog cites some items from the 6 Feb. 2008 letter sent by Portland to MetroFi, noting a lack of ongoing communication and maintenance, as well as a failure to provide information about its advertising partner MSN's privacy practices.

Washington State Ferry Wi-Fi adjusts pricing: Ferry-Fi operator Parsons now offers 2-hour sessions for $3.95, and pre-paid packages of up to 20 sessions for $29.95 (about $1.50 per session). Monthly service remains $30 per month, but Parsons roams with Boingo and iPass at no extra charge.