I find that legacy code poses a unique challenge for organizations rolling out a new security process.  Often, the resources dedicated to maintaining older code are a small fraction of those devoted to new features or products.  Furthermore, the original developers for such features have often moved on, leaving no subject matter experts to drive reviews.  The astute reader will ask “How do I apply the principles of the Microsoft SDL to legacy code when I have no development resources and nobody knows how it works?”

The answer is “Start small, and build expertise over time.”

A Rising Tide Lifts All Boats

The best thing a security engineering team can do to improve security in the short term is to drive code quality, and the first step in this process is to define and enforce a secure coding standard.  This helps on two fronts: 

1.       It will improve code quality and reduce implementation flaws across the entire code base.  Unlike other security processes, driving a secure coding standard is relatively easy to accomplish across an entire code base, regardless of the code’s age, by a focused security team.  That is not to say that it is easy without qualification – a large batch of spaghetti code will require a lot of work to untangle!  Such an effort can only be called “easy” when compared to, say, comprehensive identification and remediation of design flaws across legacy features.  Even so, improving code quality through the use of secure coding standards offers a unique combination of high impact, applicability to features, and ability to be carried out by a core team that makes it a sensible first step.

2.       The security team might notice that some sections of code have more standards violations or outright flaws than others.  This is an instance of vulnerability clustering, a concept that has been used to predict vulnerability rates and improve quality in the functional realm.  The evidence is anecdotal, but it stands to reason that portions of code that consistently violate secure coding standards are good places to start looking for other classes of security flaw.  These are security hotspots, and should be high on the prioritized list for further review.

Security testing may also be applied to legacy code, but initial activities should be considered on a case-by-case basis based on the expected return on investment.  Such testing ranges from using inexpensive off-the-shelf tools to exercise common interfaces to rather expensive custom testing and formal analysis.  It is worthwhile to begin with off-the-shelf tools, such as those that target file parsers or web applications, and tools created as part of your greater secure development efforts.  These can help identify easily-found flaws and suggest improvements to the coding standards.  Comprehensive security testing, on the other hand, is best tackled after the Legacy Security Push.

The Legacy Security Push

Coding standards and basic testing provide bang for the buck, but formal security processes seek to provide security assurance.  The challenge for legacy code is that it needs to play catch-up.  Security processes that occur early in the development cycle, such as requirements analysis, design review, and threat modeling, are particularly difficult to achieve years after the fact.  The main goal of the Legacy Security Push is to create the deliverables from these efforts, the most important of which are security requirements and a full risk analysis.

It may sound trivial, but security requirements are essential.  Not only do they define proper operation for the system in question, they also define assumptions that are suitable for relying systems.   It is very common to find security flaws in legacy systems that arise from well-intentioned but incorrect assumptions such as “I assume that the Foo authenticates server Bar when initiating a bank transfer.”  It stands to reason that Foo would do so for such an important activity, but this assumption must be validated.  It is very common for older features to have been deployed in and written for different environments where the security assumptions that are "obvious" today just didn't apply at the time.

When reviewing legacy systems, the first step is to identify such requirements.  If the original architects, developers or managers are available, they can provide valuable insight at this stage.  More often than not this is not the case, and analysis must instead rely on what documentation is present and interaction between the software and its consumers.  The goal is the same as in requirements analysis during project inception, except that in this case one must turn the process on its head and reverse engineer requirements from system behavior.  At the conclusion of this effort, requirements can be theorized – “Foo must authenticate its server Bar before initiating a bank transfer.” 

Risk analysis can be performed once a plausible set of requirements have been identified.  Threat modeling is a more structured means of performing such an analysis, with the eventual goal of identifying means by which requirements can be violated by an attacker. 

As with requirements analysis, original developers would be a valuable resource to consult.  With or without such help, the first step is to identify how the software works.  In many cases, help is not available and performing this task requires a great deal of effort.  For features of moderate size, this author has spent upwards of a month reading code, using process profiling tools, and walking through the software with a debugger to identify program flow and security-sensitive functionality.

Once completed, actual system behavior should be documented and compared against the requirements theorized.   It might be that the requirements should be re-evaluated (New requirement:  Do not assume that Foo requires server authentication) or the system may need to be changed (New bug:   Foo does not verify the CN for Bar).  At the end, this information should be sufficient to support a comprehensive threat modeling exercise where security requirements, risks, and their mitigations can be documented.

Next Steps

Bringing a legacy feature up to par with its newer kin requires a relatively small number of items:  improved code quality, clear security requirements, and a thorough threat model.  As we have seen, performing even these tasks is quite the effort!  I am sure that it is little comfort to be reminded that accomplishing these tasks has simply laid the foundation, and that the true benefit is that the newly-reviewed legacy feature is able to participate fully in the security processes that remain: reviewing cross-component security requirements and assumptions, comprehensive testing, and incident planning, to name a few.

Unfortunately, there is no silver bullet in security assurance.  The soundness of the design and implementation of legacy software is just as important as in newer software, which is why any complete secure software development process will look backwards as well as forwards.  Feature by feature, from higher priority to lower, the overall security of the software improves as legacy code receives the full security treatment it deserves.

One of the most interesting operations was the laundry mat [sic]. Having lost many troops and civilians to bombings, the Brits decided they needed to determine who was making the bombs and where they were being manufactured. One bright fellow recommended they operate a laundry and when asked "what the hell he was talking about," he explained the plan and it was incorporated -- to much success.

The plan was simple: Build a laundry and staff it with locals and a few of their own. The laundry would then send out "color coded" special discount tickets, to the effect of "get two loads for the price of one," etc. The color coding was matched to specific streets and thus when someone brought in their laundry, it was easy to determine the general location from which a city map was coded.

While the laundry was indeed being washed, pressed and dry cleaned, it had one additional cycle -- every garment, sheet, glove, pair of pants, was first sent through an analyzer, located in the basement, that checked for bomb-making residue. The analyzer was disguised as just another piece of the laundry equipment; good OPSEC [operational security]. Within a few weeks, multiple positives had shown up, indicating the ingredients of bomb residue, and intelligence had determined which areas of the city were involved. To narrow their target list, [the laundry] simply sent out more specific coupons [numbered] to all houses in the area, and before long they had good addresses. After confirming addresses, authorities with the SAS teams swooped down on the multiple homes and arrested multiple personnel and confiscated numerous assembled bombs, weapons and ingredients. During the entire operation, no one was injured or killed.

The report was written by a committee whose members include William Perry, a professor at Stanford University; Charles Vest, the former president of MIT; W. Earl Boebert, a retired senior scientist at Sandia National Laboratories; Cynthia Dwork of Microsoft Research; R. Gil Kerlikowske, Seattle's police chief; and Daryl Pregibon, a research scientist at Google.

They admit that far more Americans live their lives online, using everything from VoIP phones to Facebook to RFID tags in automobiles, than a decade ago, and the databases created by those activities are tempting targets for federal agencies. And they draw a distinction between subject-based data mining (starting with one individual and looking for connections) compared with pattern-based data mining (looking for anomalous activities that could show illegal activities).

But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so."

A summary of the recommendations:

• U.S. government agencies should be required to follow a systematic process to evaluate the effectiveness, lawfulness, and consistency with U.S. values of every information-based program, whether classified or unclassified, for detecting and countering terrorists before it can be deployed, and periodically thereafter.

• Periodically after a program has been operationally deployed, and in particular before a program enters a new phase in its life cycle, policy makers should (carefully review) the program before allowing it to continue operations or to proceed to the next phase.

• To protect the privacy of innocent people, the research and development of any information-based counterterrorism program should be conducted with synthetic population data... At all stages of a phased deployment, data about individuals should be rigorously subjected to the full safeguards of the framework.

• Any information-based counterterrorism program of the U.S. government should be subjected to robust, independent oversight of the operations of that program, a part of which would entail a practice of using the same data mining technologies to "mine the miners and track the trackers."

• Counterterrorism programs should provide meaningful redress to any individuals inappropriately harmed by their operation.

• The U.S. government should periodically review the nation's laws, policies, and procedures that protect individuals' private information for relevance and effectiveness in light of changing technologies and circumstances. In particular, Congress should re-examine existing law to consider how privacy should be protected in the context of information-based programs (e.g., data mining) for counterterrorism.

Here are more news articles on the report. I explained why data mining wouldn't find terrorists back in 2005.

I’d love to have another blog even more specific - “Ok, that Assurance is Evidence Of What, Exactly?

Today he has a great article called:

What’s the matter with Risk Management?

And “in short, it’s everything.” It pretty much sums up why I had to grow to re-evaluate how our industry does risk, risk management, approaches controls & vulnerability and find a new way.   A couple of things jump out at me in reading Chris’ article:

1.)  Just because that Deming cycle sucks and is full of unknowns doesn’t mean “risk” doesn’t exist, nor that it isn’t of primary importance. Nor does it mean that in the absence of model & methodology, we won’t be “doing” risk analysis anyway - just in an ad hoc method and completely from “the gut”.

Our industry calls these unstructured risk analysis “Best Practices”, as it’s an easy and convenient way of sweeping the unknowns under the rug of bureaucracy and enforcing it via peer pressure.

2.)  What this “suckiness” does mean is that your model and methodology aren’t helping you. As Chris intimates, there is too much uncertainty in the inputs for his model (they are, in the language of Bayesians - too subjective to be useful priors).

Take for example how we might be approaching the “controls” part of our analysis.  Chris writes:

“2. What are the controls that we have to employ?
800-53, ISO 27001, PCI, etc.

Still kinda good, but we basically know that ISO is relatively voluntary and NIST supplies a control catalog and not policies. So here we have to take the control catalog, and mash our policies into it.”

I wouldn’t call this “kinda good” at all :)  These control catalogs only provide a hierarchy within which to look for evidence of  our ability to resist an attacker.  They are incapable of making any claim about the effectiveness of the controls when they are operated at 100% efficiency, or more importantly, what % efficiency our specific organization operates at.

Let’s use Chris Hayes’ Initech as our fictional example.

Initech has a control (a back door on a loading dock).  Now the locks on the door are 100% capable of locking the door.  This is different than saying that they are capable of frustrating all but the top 5% of lockpicking burgalars.  It is also diffferent than saying that in a sample of several “walk around audits” the doors are left open 20% of the time (they are not in compliance with policy 100% of the time).  Even worse, that 80% of the time the door is not propped open?  Yeah, tailgating is a known issue.

So we have several different variables here that we need to account for (and it’s just a door).  But the analogy stands that most “risk management” methodologies are “We have a door, yes/no?” And most GRC platforms, when asked for their “opinion” will simply say “door is needed” or, even worse, “a door policy is needed”.

3.)  Criticality and the Source of Value is all messed up in these Risk Management models.

Chris writes:

Someone wants me to tell them which boxes are more critical than others. This is mainly because of budgetary or operational reasons. To which I usually say “All of them, it is a system after all”.

This literally made me laugh out loud.  And this sort of “rate the firewall as Risk = 500 but rate the actual business application as Risk = 157″ thing is also endemic.  Now Chris is very smart here.  He correctly identifies that the value is tied to the business process the systems support, and not to a specific box.  Oh, we scan at the specific box level - but because of the nature of systemic failures - all the boxes in the process are inexorably interrelated.

One of the reasons I really like FAIR is that the losses are quantified (or qualified) based not on some amorphous value of the box or the process itself, but losses are linked to the actions that the threat will take. Take systems in a highly regulated industries as an example.  Usually the most probable losses aren’t due to system compromise per se, but in the disclosure the compromise causes (regulators are a threat source, after all).  But many “risk management” methodologies will say “online banking is worth $2 billion, the value of the systems is therefore $2 billion”.  And suddenly we’re telling executive management that there’s a 60% probability that they’ll lose $2 billion.

4.)  If the primary source of prior information for your “risk management” methodology is a vulnerability scanner - you’re doing it wrong.  Chris writes:

So we ran a scan and now we have a report. A snapshot in time to make all decisions. Where did these vulnerability ratings come from? Do I even know if my system is at risk? What if I spend my time on vulnerabilities that have no threat?

So first, my thoughts are that actual “vulnerability” must be a comparison of the force a threat can apply, and our ability to resist that force (this is a probability statement, btw).

Changing your thinking about vulnerability now helps us understand the problem in several new ways.  First, you can start to divorce yourself from the scanner.  After all, the scanner is simply providing you with current state information that is usually just relevant variance from policy. It doesn’t really tell you about real “weakness in a system” because the system is an interrelated mess of people, processes and IT assets.

5.)  Finally, most “risk management” approaches just *don’t* do a good job of helping us understand the how’s and why’s of managing risk. In the past, I’ve referred to these standards as really being “issue management” because they are at their heart, an act of discovery - a formal process around gathering prior information.  They are not, in and of themselves, capable of linking the issues discovered to the root cause.  And these root causes?  Yeah, they’re the things that create “risk”.  Not a threat, not a vulnerability, not the existence of an asset - the amount of risk that we have stems from our capability to manage it.

So Chris, I completely agree - but I wouldn’t give up yet.  There actually are a few of us who are focused on what you suggest:

Where to go from here: A fundamental revamp of how to deal with Risk. Where risk professionals focus on the treating the sickness and not the symptoms, and come up with some new success/actionable metrics.

Chris, there’s nothing I want to do more than that.

Dan described the Managed Hosting and colocation sector as “on fire” The sector is humming – incredible growth, outstanding execution, blowing away expectations. I must say, looking back 5 years ago after the tech bubble collapse, I can’t believe how strong the sector bounced back from those very difficult times.

His presentation was focused on a future, and a longer view for the industry. The HTS conference is packed this year with the largest attendance of Datacenter owners, Managed hosting and colocation companies ever to attend this conference.

• Demand steady or increasing in all markets, driven largely by capex constraints and greater awareness and choices.
• Supply is growing more slowly in the past 18 months as the credit crunch has hurt the ability of providers to expand ( it is very hard to get mortgages, loans only on new datacenter projects). Expansion build-out of existing shells is occurring, but very little on spec.
• Demand Growth of 15% in 2008. (Steady and increasing in the out years) However after supply growth peaked at 7.5% in 2007 supply growth now has slowed to 5%
• Dan believes that supply growth will pick back up again in 2011

Conclusions – supply is tight, demand is high and growing…this very good news for the industry.

• Some other trends:
  • The green initiatives are more than just a trend as datacenter owners who don’t figure out how to maximize power efficiency will be painted as villains.
  • Internet traffic and services consumption are linked as Internet traffic growth has been doubling every year (2005-2007)
  • Prediction: 2011 -2012 - internet traffic will get an exaflood – it is coming with a new breed of applications (set to boxes HD Video, games, etc.) that will drive new traffic patterns. Growth driven by consumer broadband + applications (HD video) applications, which in turn will drive demand for Managed Hosting / Colocation Services…

Managed Hosting Services Highlights

• Incredibly fast growth 30%+
• $10 Billion worldwide revenue by end of 2008
• We’ll keep growth pace until at least 2011
• Good news, Dan believes that fears about slowdown in growth are wildly overblown.

Why is managed hosting growing so fast?

• Demographic shifts – new breed of IT employees that embrace outsourcing
• Growth in internet applications (SaaS) The acceptance and growth of browser based applications has been enormous!
• Ambiguity between web hosting and managed hosting has turned positive

Dan’s Key success factors managed hosting and services

• High margin services – and not too many – it is so tempting in our day to day business when a customer comes along and wants to come and give us money for a unique on-off service… at this point the answer has to be no – or do it through a partner.
• High level of support delivery is critical – don’t cut pay in support people or outsource support to save a nickel… what you are selling is support. Keep doing this well or you will head into a bad place… just as examples in retail like Home Depot and others who have struggled with customer service challenges – the whole business starts to slide into the toilet… High levels of support delivers a strong word of mouth buying cycle

Final thoughts, the industry is healthy and will continue to thrive. Customers are looking for the one stop shop, one company that is a trusted advisor to the customer. As customers place more eggs in the Managed Service bucket, the industry will need to tighten-up those SLA’s. Today some parts of the industry have been getting away with loose SLA’s… as customers get more sophisticated and have more on the line, they will become more demanding and require robust multi-component SLAs and back-it –up.