Blogger: Dan Blum

With the crisis in financial markets still unfolding, it is important to draw what lessons we can from the experience. Since the roots of the crisis lie in a monumental failure of risk management, it’s important to understand more about what happened, and then draw some parallels to our business risk management and  IT risk management situations.

The risk management failure in the housing market and on Wall Street had multiple interdependent dimensions:

• Mortgage lenders abandoned long standing prudent loan practices. They made too many loans that buyers might not be able to repay. Exotic instruments like ARMs, option ARMs, and interest only loans proliferated. In many cases, all pretense of lending standards were abandoned, so-called “liar loans” approved.
• Capital was grossly over-leveraged. Mortgage lenders and other financial services packaged loans into securities, which they sold to raise capital to support more lending. Real capital reserve requirements to back loans were reduced. Of course, if borrowers could not repay loans, all or parts of the derivative securities would become worthless.
• Risk was aggregated at Fannie Mae, Freddie Mac, and mortgage loan insurance companies. These companies bought or insured some mortgage loans, providing something of a backstop should loans fail. Government sponsored enterprises (GSEs) Fannie and Freddie in turn became over-leveraged and securities that they sold were in turn repackaged in the murky brew of mortgage-backed securities called collateralized debt obligations (CDOs) and other exotic instruments returning generous yields.
• Non-Caveat Emptor. Institutional wealth funds and financial services firms who should have known better bought securities that had been deliberately structured to obfuscate risk. They bought securities they didn’t understand with buried tranches of toxic subprime loans..

It was a great Ponzi scheme – one that kept working as long as housing prices were going up; the recipients of subprime loans could always flip that house to the next buyer. Everyone made money. As Chuck Prince of Citigroup famously put it during a July, 2007 interview: “So long as the music is playing, you’ve got to keep dancing. We’re still dancing.” But one month later, the music stopped. Since then, Citigroup and other financial institutions have taken massive writeoffs with more to come. Wall Street titans like Bear Sterns, Lehman Brothers, Merrill Lynch, and AIG have fallen or been bought out.

What can we learn from this risk management debacle?

As business risk managers and investors, we should ask questions like these:

• Does the executive incentive structure of the company encourage managers to dance around risk? Many Wall Street firms paid senior managers 5 times their salary in bonuses tied to annual growth alone.
• Is the company over-leveraged? Is it borrowing too much money and betting it on ventures with uncertain outcomes?
• Are financial models used for risk management realistic? Earlier, I described the mortgage market of the past few years as a Ponzi scheme, where risk management models must have assumed prices would keep rising. Unlike the dotcom boom whose demise many predicted, very few in the industry foresaw the sharp declines to come in housing prices and sales volumes. Historically, the U.S. housing market has been a steadily rising one, but on the other hand the 2000s saw unprecedented rates of price increases. In reality, what goes up must come down.
• Has your company’s risk council ever performed worst case scenario analysis and built adequate reserves? In the days before economics emerged as a would-be “hard” deterministic science, business leaders may have been more cautious, more aware of and more accepting of uncertainty. Events like the Great Tulip Bubble came once in decades or centuries – not every few years. Note that legendary investor George Soros has proposed a Theory of Reflexivity that, if true, helps explain the recent extremes of boom and bust cycles. This theory holds that market participants model market behaviors based on self-interest, and for a time, their manipulations change the reality of the market – until gravitational forces bring it back to earth. Has the music of ephemeral success played to the backbeat of deterministic-sounding economic models gone to your heads and infected your risk management models?
• Are cost cutting efforts pursued blindly? Outsourcing and other forays into treacherous global waters may be giving away the crown jewels. Smart companies cut costs, but they do it in smart ways. Smart companies think like intelligence agencies as they parcel out work to different partners with varying levels of dependability, and they check on those partners.

Risk management failures can also occur at the more technical level of IT security. As IT risk managers, we might ask questions like these:

• Are the accounting and financial systems your IT department supports under adequate control? As Fred Cohen wrote in one of our documents: “Many companies use computers to manage financial systems, and despite the Sarbanes-Oxley Act (SOX) claims about accounts being properly kept, there are many attacks on financial systems that remain. For example, most of the largest financial systems in the world running on common financial databases do not use double-entry bookkeeping and are thus susceptible to all manner of frauds by insiders.” We find it troubling that a prudent control dating back to the 12th century is going out of style in the name of convenience and cost cutting. Kind of like credit checking became anachronistic during the housing bubble, eh?
• Is the “separation” in your “separation of duty” (SoD) for real? Sure the SOX auditors are looking for SoD, and maybe you have different administrators with different accounts maintaining different systems or functions. But when they say Western civilization may be but one weak password from collapse they’re not lying. Look what happened to Sarah Palin’s email account! Weak and straggly SoD is a problem across all critical IT systems where deperimiterization and server consolidation may be bringing down protective barriers, identity management is weak, and strong process controls (e.g., where two people must sign on, one perform a critical operation such as backbone router reconfiguration, and the second observe) abandoned in the name of expediency.
• Are risks being aggregated to unacceptable levels in centralized control systems? There are many ways that risks aggregate within enterprise IT infrastructures as we pursue automation and cost cutting. Network risks aggregate when centralized domain name system control is implemented. Application risks aggregate when common infrastructure is shared among applications. And enterprises aggregate platform risks when they use low-assurance endpoints, authentication, and directory systems with single sign-on to access large numbers of resources and don’t separate high consequence systems.
• Non-caveat emptor: Has IT security really done the worst case consequence analysis, attack graphs, and vulnerability analysis to know when putting more eggs in a supposedly stronger basket aggregates risks to an unacceptable level? Or are you depending only on vendor claims about some black box appliance equivalent of a risk-obfuscated CDO security? Caveat emptor (buyer beware) again! (The good news is we’ll keep talking about promoting vendor and product rating systems so you don’t have to do all the detailed product analysis yourself, but that’s another post.)

There are many parallels between the monumental risk management failure in the financial markets, and the probable weaknesses in our day to day business risk management and IT risk management. Abandonment of prudent practices for profit; excessive leverage and centralization; ill-constructed risk analysis models; risk obfuscation; and a failure of caveat emptor seem to be common problems. Please take this as a wakeup call to sharpen up the risk management thinking, process, and execution.

Defining Cloud Computing

Cloud computing is a very active community. The Google Group gets 600 posts per month and many bloggers are covering the space. However, “cloud computing” is impossible to define in a way that satisfies everyone (or even most). Cloud computing is not alone in this controversy, consider the definition and meaning of “Web 2.0″, “mashups” or “RESTful architecture”. All of these terms are relatively recent. According to Google Trends, these terms became popular to the general public sometime between 2005 and 2007:

• Web 2.0 - often confused with RIA, AKA Social Computing, Long-Tail Apps, Crowdware (2005 by O’Reilly Media)
• Mashup - made popular by Google Maps, AKA Composite/Situational Apps. (2005)
• REST - Has a strict definition, but many don’t understand it and abuse the term. (2006 by R. Fielding)
• Cloud computing - collides with many other terms, such as SaaS, Grid, Utility, PaaS, etc. (2007)

The definition of cloud computing is in progress:

There’s a Darwinian evolution of the exact definition of cloud computing running around. We’re about a country mile away from “knowing when I see it”, which is excellent progress. The cloud to everyone’s silver-lining has enough material to write a 3 volume desktop reference at this point. - Michael Cote, June 2008

Definition #1 - “Cloud computing is the realisation of Internet (”Cloud”) based development and use of computer technology (”Computing”) delivered by an ecosystem of providers. - Sam Johnston, July 2008

Definition #2 - “Cloud computing = network computing. I love the idea of cloud computing, the next evolution of the most network intensive architecture possible, but one that if it works well, is transparent. It’s all about the transparency.” - Douglas Gourlay, Cisco, May 2008

Definition #3 - “There seems to be a group myopia around so-called “cloud computing” and its definitions. What we’re really talking about are “cloud services” of which, “computing” is only a subset…Cloud services are not SaaS. They are far more akin to web services…” - Randy Bias, neoTactics, May 2008

(Anti-)Definition #4 - “Note that I refer to cloud services, not to the could. I am not interested in defining cloud as a term, because I don’t think it’s very useful. For those of us in the distributed computing’s pace

The Working Definition (Winner!):

“…the notion of providing easily accessible compute and storage resources on a pay-as-you-go, on-demand basis, from a virtually infinite infrastructure managed by someone else. As a customer, you don’t know where the resources are, and for the most part, you don’t care. What’s really important is the capability to access your application anywhere, move it freely and easily, and inexpensively add resources for instant scalability.” - Mitchell Crandell, Rightscale, June 2008

Taxonomies of the Cloud Space

Taxonomies are useful to provide insight into a market. It classifies a multitude of players into a smaller bucket.

Andreessen’s Platforms - September 2007

Provided an early taxonomy model for emerging cloud platforms

Platform being a system that can be programmed

• Access API - platform that provides web service endpoints
• Plug-In API - platform invokes your code, that you have deployed remotely
• Runtime Environment - your code runs inside the platform’s process space.

Mehta 11 Layer Stack, April 2008

1. Facilities (space, power, cooling)
2. Network
3. Hardware (e.g. servers Amazon EC2 runs)
4. Hardware virtualization (e.g. Xen for EC2) - optional
5. O/S (e.g. Linux)
6. Systems Management (e.g., tools to manage EC2 instances)
7. Application Middleware (e.g., MySQL on EC2)
8. Application Code
9. Application APIs / Web Services
10. GUI for Application
11. GUI for Application Development / Customization

Croll Cloud Stack, June 2008

7 layer stack within Turnkey app and Generic Platform.

Turnkey app

• SaaS
• Extensible app
• Generic IDE
• Constrained APIs
• App Cluster
• Virtual Data Center
• Virtual Servers

Generic Platform

The bottom of Alistair’s stack includes “root access “style compute clouds.

Robert Anderson, July 2008

3 layer stack

• Software (SaaS)
• Platform (PaaS)
• Infrastructure (IaaS)

This is the model taxonomy for this session.

Related Concepts and Terms

• Infrastructure as a Service (IaaS), Hardware as a Service (HaaS) are synonyms to cloud infrastructure.
• Virtualization
• Hosting
• Autonomic computing
• Distributed computing
• Grid computing

Cloud Applications

• SaaS
• S+S (Software+Services)
• Managed Service Provider (MSP)

Abstract -- We reverse engineer copyright enforcement in the popular BitTorrent file sharing network and find that a common approach for identifying infringing users is not conclusive. We describe simple techniques for implicating arbitrary network endpoints in illegal content sharing and demonstrate the effectiveness of these techniques experimentally, attracting real DMCA complaints for nonsense devices, e.g., IP printers and a wireless access point. We then step back and evaluate the challenges and possible future directions for pervasive monitoring in P2P file sharing networks.

Webpage on the research.

Some of the major obstacles we face with 802.1X center around creating a smooth end user experience.  We, as integrators, have the distinct ability to make ‘whatever’ work- we find a way. But, what I hear most from my customers is “it has to be easy for the end user.”  (Sometimes they go on a little further, but I’ll leave it at that.)

Why does it matter?

Wireless, wireless, wireless. Although wired 1X is popular with our customer-base, the world isn’t quite flocking to it yet. However, 802.1X is certainly the best way to increase security and ease management of wireless networks. It’s standard, it’s flexible, it’s widely-supported by devices and endpoints and it eliminates the need for pre-shared keys or secondary passwords. It’s what most enterprises, government and educational organizations are implementing now, so it’s important.

What are some of the problems?

The end user will have some adjustments to make, and network admins and support desks aren’t always thrilled with the propect of re-training users for these expectations.

For a long time I have been writing and speaking about the many ways that NAC can help with securing your endpoints and your network. Yesterday, Tim Greene lays out some good reasons for NAC and the many ways it can help.  However, he couches it in terms of NAC as a personal firewall.  I am not sure I agree with that one at all.  Personal firewalls are usually thought of as host based security on the endpoint.  While NAC certainly has an aspect of that, NAC is inherently about networks as well.

I am reminded by this article of Senforce.  They had one of the best personal firewalls in the market and were often called a NAC solution.  But when you spoke to Nolan Rosen and the folks at Senforce, they would tell you that they were not a NAC solution, but needed a network based NAC component to compliment their product.  That was the basis of a partnership we had with them.  In any event, I think we are seeing NAC used for a variety of uses and we will continue to see it evolve in the market.

For a long time I have been writing and speaking about the many ways that NAC can help with securing your endpoints and your network. Yesterday, Tim Greene lays out some good reasons for NAC and the many ways it can help.  However, he couches it in terms of NAC as a personal firewall.  I am not sure I agree with that one at all.  Personal firewalls are usually thought of as host based security on the endpoint.  While NAC certainly has an aspect of that, NAC is inherently about networks as well.

I am reminded by this article of Senforce.  They had one of the best personal firewalls in the market and were often called a NAC solution.  But when you spoke to Nolan Rosen and the folks at Senforce, they would tell you that they were not a NAC solution, but needed a network based NAC component to compliment their product.  That was the basis of a partnership we had with them.  In any event, I think we are seeing NAC used for a variety of uses and we will continue to see it evolve in the market.

Stiennon talks about Rohati and their ‘new’ approach to NAC in the form of their NBEC, Network-based Entitlement Control. I, unlike some bloggers in our network, decided to check it out before formulating an opinion.

So, I checked it out and I’m a little disappointed… on several fronts. First, all the information I have with which to draw a conclusion is limited to the online ‘product demo’ available on their website. It’s not really a product demo, hence disappointment number 1.

Let down number 2 comes in the realization that the features they’re touting in the ‘product demo’ are actually things we can do today, with traditional hardware-based NAC solutions from those daily house-hold names… Symantec, StillSecure, Juniper, ProCurve, Enterasys and even Cisco. Rohati does (potentially) have a unique statement of being able to enforce policies without touching the client. But, again, we ‘can’ do that with several of the products I just mentioned. And I’m wondering how we could create the tunnel-like enforcement and security Rohati claims to offer without some type of agent on the client… after all, any encryption tunnel has to have endpoints, right?

I attempted what I usually do when I’m checking out security solutions, I went to the support section of the website to download product manuals or configuration and implementation guides. Even some white papers. I wanted to see how they’re really going about it all. But, disappointment number 3 jumped up and got me when I saw that the only resource on their support page was an email address. Hmm….

The company seems to be comprised mostly of long-term ex-Cisco employees. Out of the 8 members of the management team, there’s 1 President, 6 VPs and a director- 5 of which are co-founders. With just 2 years under their belt, I’m wondering what all they can have up their sleeve past a slight variation of current NAC solutions.

I may be completely wrong about the company and product(s). If I am, I’m sure someone will offer to send over some product manuals for me to read through…

The bottom line is… a rose by any other name would smell as sweet… or stink as bad.

# # #

One of the objections I've had about REST for a while is that it appears to ignore Deutsch's fallacies of network computing

1. The network is reliable.

2. Latency is zero.

3. Bandwidth is infinite.

4. The network is secure.

5. Topology doesn't change.

6. There is one administrator.

7. Transport cost is zero.

8. The network is homogeneous.

Now REST specifies 8, assumes 1, 2 and 3 and takes 4 to mean HTTP/S with Basic Authentication. Now to be clear I've seen people doing Web Services who believe in pretty much all 8 of these fallacies and they create crap systems. But with things like WS-RM and WS-Security at least there are answers to a few elements.

That basic auth is bypassable has been known for some time, thanks to Amit Klein. It would be nice to Restafarians move the conversation towards better security models like SAML and WS-Security. The current state for Rest is both disappointing and weak. The response side is pretty solveable using XML Signature and XML Encryption to sign and encrypt the responses (of course someone will need to tell the "you just leverage the existing infrastructure types" that we'll need to be deploying keys and certs to all the endpoints but at least the primitives are there on the response side), the request side remains problematic.

More on the Fallacies by Arnon Rotem-Gal-Oz, who incidentally if you are interested in building a secure service has an interesting Service Firewall pattern, which I refer to as a TIDE firewall - dealing with Tampering, Information Disclosure, Denial of Service, and Elevation of Privilege threats at the edge. I understand why Arnon left Spoofing off his list, but would like to see him add audit logging to deal with Dispute.