ScienceLogic: Can you share any of the strategies/advice that you give to companies embarking on their BSM journeys?

Doug McClure: Well, first they’ve got to have a BSM strategy. Nearly all the clients I talk to or hear about wanting to do BSM do not have a BSM strategy. I talk a lot about this on my blog and with clients and it is relevant whether you’re going to think about “BSM Lite” or “BSM Heavy” approaches.

Once we have a BSM strategy, we need to establish a BSM roadmap that guides us in how we’ll implement the BSM strategy in a more tactical manner, focusing on short term iterative quick wins and 30-60-90 day projects. For more of my thoughts on BSM strategy and roadmapping, see the following blog posts.

• Elements of Business Service Management Part 3: Getting Business Service Management on the Radar Screen

• Elements of Business Service Management Part 4: What’s your Business Service Management Strategy?

As I’ve alluded to previously, a client first must define and understand what “BSM Lite” may mean to them. Don’t take what the analysts or the vendors pitch for what you should do to achieve BSM or what value you should get from it.

For any type of BSM to be successful, each client must define what BSM means to them and state what they expect to get from BSM. They must make it personal, make it a part of their company culture and elevate it to be as an important initiative as compliance, risk management, SOA, ITIL, or other initiatives may be within the company.

Please don’t get scared off from this strategy thing. Please don’t blow this off as something that the secret enterprise architecture council should be doing. If you’re unable to get an audience in these areas within your company, start within your own sphere of influence.

Your strategy could be as simple as enabling the local operations center to more efficiently classify, triage and resolve problems based on a simple business service or application contextual understanding. Focus on how this changes the game within your environment. Come up with your own metrics and measures to assess the value this has to this organizational use. Trust me, you’ll need to justify your investment some time in the future.

Another trait of successful BSM implementations is that of the formal monitoring and management tools group has established some sort of database or knowledge repository that enables them to “manage the business of IT management and monitoring” if you will. In my opinion, the vendor community has let their clients down significantly in this area. The CMDB may be the correct answer, but most companies just don’t value monitoring enough to demand that this be included in their formal CMDB initiatives.

In my last job, we developed an application that I referred to as the “Service Management Database” or “SMDB”. Others may call it something else, but in essence, it was the database that captured what was monitored, how it was monitored, who owned it, what business services and applications it supported, the impact an outage or event from it had on the business services or applications, etc.

One key component of this “SMDB” was establishing the relationships of real and synthetic user and transaction monitoring steps to associated servers and applications. This is a significant gap area in many tools and vendor CMDBs.

Clients who have instituted something formal such as this generally have a very good handle on management and monitoring within their environment. Far too many clients do not have adequate monitoring (read visibility) in place to begin their BSM journey.

I’d strongly recommend a good hard look at how well the client’s monitoring and management practices are implemented and managed. Simply put, if they don’t have adequate visibility into how well those business services and applications are performing, you can’t expect to manage what you can’t “see” that may be impacting the business, clients, revenue, etc.

Just ask yourself this – can you explicitly state what monitoring is in place for a given business service or application? Can you quantify the impact of a simple event to a business service or application? Can you explain why something is red, yellow, purple or green and what causes it to change from one color to another? If you can’t, your BSM journey will be challenging.

Those with formal CMDB initiatives have their hands full with high risk, long time to value projects to just get a handle with traditional configuration management models. Taking these low level configuration items (CI’s) and establishing application and service dependencies comes after a lot of work getting through the organizational challenges of getting systems access to populate the CMDB.

I strongly recommend that the formal monitoring and management tools group create an authoritative database that enables them to establish end-to-end visibility into the service and application delivery chain and the impacts it has on the business, customer, etc. This ultimately becomes part of a more realistic federated CMDB within the business.

ScienceLogic: Can you provide an example of a successful implementation of BSM? Were there specific factors that especially contributed to its success?

Doug McClure: I’ve touched on the highlights of the most successful BSM implementations throughout my previous answers. Clients that have rallied around an organizational change or transformation focusing every team member’s efforts and energy towards ensuring that the business goals and objectives are being met through the delivery of highly available business services and applications.

Far too often the “change” never happens and it’s the “talking heads” that are preaching to the choir about what should be done. Every person on the front line, in the support teams, at the help desk, etc. must understand how they support or impact the business in business terms. Try putting this simple phrase after job titles “Hi, my name is Doug. I’m a Systems Administrator, Supporting the Business”.

That was a mouthful, but simply put, these clients have an impressively instrumented business and IT environment with the right amount of visibility into each area, joined together with an organization that thinks, operates and responds based on their understanding of the business goals and objectives and how these business services and applications enable business success.

The operational model for an organization fully adopting BSM identifies ways to establish a service management mentality across the entire business service and application delivery and support chain. The delivery, operations and support organizations must be incented to manage the services and applications being delivered with this end-to-end context.

A leading, outside the box “service management organization” may include the traditional IT silos but within a matrixed fashion focused on one or more key business services and applications. The “service management organization” is then incented to work together, as a team, for the end-to-end delivery and support of these services or applications.

It’s no longer one’s job to just be the systems administrator, database administrator or network engineer, their job is now to support specific business services and applications. They provide the subject matter expertise needed to support the services and applications together, as a team, eliminating the finger pointing or “not my problem” attitudes that exist in the majority of IT organizations today.

Overall, the KISS approach is what will enable BSM of any type (lite, heavy) to be the most successful. If it just feels natural, doesn’t take any additional effort, clicks or tasks to do then it’s going to work. BSM should be transparent and not just another buzz word. It’s not a form that gets filled out or a special process to follow in the run book. It’s doing the right thing for the business, no matter what the situation, crisis, buzz word or technology initiative of the day is.

ScienceLogic: How did you get involved in BSM?

Doug McClure: I think the foundations of my service management background and passion were initially established during my service in the US Navy. Today, I relate that experience to what I call BSM for the Military or Mission Services Management (MSM).

We had been taught over and over that extreme attention to the details of the mission at hand (aka “the business”) was the number one priority and that all of our technology, services, and applications existed for those Sailors and Marines on the other end (the “customer”). I can recall countless instances where mission critical communications services (telephony, orderwires, teletypes, command and control systems, etc.) were impacted in one way or another. It was extremely critical that we understood who was impacted and to what degree so that contingency plans could be activated. We weren’t just talking about lost revenue, poor sales or customer experience; we were talking about human lives and the security of the United States.

It is that military bearing, attention to detail and real world experience that drives me with many of my modern day BSM endeavors. That migration from “Mission Services Management” to BSM was honed working for over 10 years working in the Internet Service Provider (ISP) and datacenter, hosting and colocation business.

In those rapid growth businesses during the Internet boom, service differentiation was what “made you millions” or paved your way to bankruptcy. The companies I worked for had an extreme passion and focus on ensuring that their services, applications and Internet access products were of the highest quality, highly reliable and just plain better than the competition.

Again, the IT infrastructure, service quality and customer experience relationship was ingrained in all of our heads. It was all hands on deck when Webmail, Internet access, DNS, or the network experienced problems. We were measured in terms of how many customers experienced a busy signal or dropped connection or if you couldn’t log in fast enough to read your email. Companies like Keynote Systems and LionBridge/Veritest/Inverse tested the quality of our networks, services and applications and publicly ranked us against our competition. We thought in terms of customer experience and impact every minute of the day, 24×7.

It was in my last job managing a traditional enterprise management and monitoring development group for a nationwide ISP where I was able to work with emerging technology to help get a handle on the complexities of these rapidly growing IT environments filled with emerging technologies and products. Applying this early technology to complex service problems in our environment proved to me that the technology, coupled with the right emphasis on how the technology was implemented and an emphasis on the people and processes within the organization could bring BSM to life.

Where I felt left out in the cold was with my vendor relationship. While their technology gave me the potential, they didn’t teach me how to work through the organizational and technological problems to successfully implement the BSM strategy. My very first end-to-end BSM pilot was extremely successful and provided visibility into the IT environment and business service impact that have never been available before.

And here I am today, working at a software vendor for the first time. Welcome to the “dark side” as they say. The approach and methodology we followed for BSM has become the basis of the core BSM Methodology that I teach IBMers and our clients around the world today.

My personal mission and drive here at IBM Tivoli is to ensure that BSM is something that the typical monitoring tools administrator can actually implement and that our BSM story is something that any of our clients can be successful with. The sales and marketing slicks must be backed up by something like this whomever you are these days. Clients shouldn’t put up for “marketecture”, me too and gee whiz buzz words.

BSM takes a partnership and commitment to every client’s success, and I want to be involved in those BSM efforts in every industry or market worldwide. We need more thought leaders collaborating together in an open and public forum to change legacy attitudes about BSM and do what we can to enable client’s to be as successful as they can be.

ShareThis

When the deluge began in 2006, officials scurried to come up with software "patches," "wraps," and other bits of triage. The effort got serious last summer when top military brass discreetly summoned the chief executives or their representatives from the 20 largest U.S. defense contractors to the Pentagon for a "threat briefing." BusinessWeek has learned the U.S. government has launched a classified operation called Byzantine Foothold to detect, track, and disarm intrusions on the government's most critical networks. And President George W. Bush on Jan. 8 quietly signed an order known as the Cyber Initiative to overhaul U.S. cyber defenses, at an eventual cost in the tens of billions of dollars, and establishing 12 distinct goals, according to people briefed on its contents. One goal in particular illustrates the urgency and scope of the problem: By June all government agencies must cut the number of communication channels, or ports, through which their networks connect to the Internet from more than 4,000 to fewer than 100. On Apr. 8, Homeland Security Dept. Secretary Michael Chertoff called the President's order a cyber security "Manhattan Project."

It can only help for the U.S. government to get its own cybersecurity house in order.

One of the phrases I often hear during vision and strategy planning meetings at Microsoft is "What is the crawl, walk, run?" We use this phrase to differentiate the initial activities that will get us quickly moving toward our larger goals and then supplement them with other activities that may require longer preparation or planning. As I help non-Microsoft companies implement SDL into their development lifecycles, this "crawl" phase toward full adoption of SDL is very important. Usually some person in an organization picks up on the principles of SDL and is ready to roll them out immediately. However, that person usually is faced with competing interests that complicate full adoption: the team is mid-stream in development, short on budget, or management wants to see clear evidence before investing in the changes to support full SDL adoption.

Since we usually focus on how to roll out the full Lifecycle, I want to take a shot at defining what it means to start “crawling” toward SDL. One very important note before I start. What I describe below is not Microsoft’s SDL process. It matches some of the tools and principles, but does not encompass the holistic application security solution provided by SDL.

In my mind, to start crawling toward SDL, you need to execute on some of the core principles. They obviously need to be low-cost and effective. So, I want to summarize these into three components.

1.       Detailed awareness of your architecture and its attack surface.

2.       Tools that will perform security analysis on your application.

3.       Results that show how the analysis resulted in improved security.

The good news is that you can attain these components with tools that are already available. The one consistent minimum requirement is that your code compiles/builds within Visual Studio 2005 SP1. The SP1 piece of this is important because some of the important defenses I discuss below were first made available in that version. Let’s look at some of the tools you can use to get “crawling” toward SDL today:

Detailed awareness of your architecture and its attack surface

Threat Modeling

Even if you are past the design phase, assign someone to do a retrospective model (perhaps as part of a pre-release review). This will likely give you a better understanding of your overall architecture and uncover holes in places you may have inadvertently overlooked.

Tools that will perform security analysis on your application

This is probably one of the most often discussed topics around SDL, so I’ll spend some time providing more detail. Let’s break this down into how it impacts differing parts of your team or organization: developers, testers, and operation.

Developers

You should start by strengthening your compiler defenses. Depending on whether you are writing native or managed code, these will differ.

For C and C++ code:

Strengthen your compiler defenses

·         Use the latest compiler and linker because important defenses are added by the tools

·         If using Visual C++,  use Visual Studio 2005 SP1 or later

·         Compile with appropriate compiler flags

·         Compile clean at the highest possible warning level

·         Compile with –GS to detect stack-based buffer overruns

·         Link with appropriate linker flags: /NXCompat to get NX defenses, /DynamicBase to get ASLR,  and /SafeSEH to get exception handler protections

Do not use banned APIs in new code

·         Use #include “banned.h” header file to find banned C/C++ functions in your code quickly. This header file is included in the companion disk in the Security Development Lifecycle book.

·         Compile regularly with /W4 and fix all C4996 (banned C Runtime function) warnings

For all Languages:

Strengthen your compiler defenses

·         Use the latest compiler, linker and libraries because defenses are added by the tools and code

o   If using C#, use  C# v2.0 or later and if using VB.Net use 8.0 or later

·         Use .NET Framework 2.0 or later

·         Do not use weak crypto in new code

o   Use only AES, RSA and SHA-256 (or better)

·         Prevent XSS vulnerabilities by using filtering and escaping libraries around all Web output

·         Secure your SQL script by only using prepared SQL statements - no string concatenation or string replacement

Run these tools habitually

·         PREfast (in Visual Studio 2005, use the /analyze compiler option) – a static analysis tool that identifies defects in C/C++ programs and enables you to perform quick desktop error detection on small code bases

·         FxCop – an application that analyzes managed code assemblies (code that targets the .NET Framework common language runtime) and reports information about the assemblies

·         Application Verifier (AppVerif) – detect and help debug memory corruptions, critical security vulnerabilities, and limited user account privilege issues.

Testers

James Whittaker has covered testing in the SDL on this blog in the past. In a “crawl” scenario, you need to keep it simple while maximizing the value of output. I would recommend focusing on fuzz testing. This is likely something you will need to invest some time creating.  Scott Lambert’s article on Fuzz Testing at Microsoft and the Triage Process provides some good guidance on how to think through what type(s) of fuzzing to exercise against your application.

If you choose to expand beyond fuzz testing, I would point you back to James’ article on the broader topic of Testing in SDL. You may come to the conclusion that expanded security testing  may come later in your “walk” or “run” phases, but I would take some time to think through testing even while “crawling” to ensure you are getting broad enough coverage for your application. James’ article highlights the three-pronged approach to security testing we use at Microsoft. You should use these three approaches to ensure your own fuzz testing is comprehensive.

1.       Attacks against the application’s environment.

2.       Direct attacks against the application itself.

3.       Indirect attacks against the application’s functionality.

Results that show how the analysis resulted in improved security

Response planning

Protecting your customers is the entire reason for focusing energy on application security.  If there are holes in your code that you don’t uncover, someone else will. It is absolutely critical that you are prepared to respond rapidly and protect your customers. It is equally important that you construct your  response plan to serve as a front-line barometer for detecting the resilience of your security design  and what pieces of your applications security should be proactively bolstered to  address externally reported vulnerabilities.  The knowledge you harvest from these security incidents (typically through root cause analysis) is the primary way to improve your code and security tooling for the future.  Do everything you can to learn lessons from the vulnerabilities others find. If you don’t have a response plan in place, you need to get one in place as soon as possible. If you don’t know where to start, take a look at how our own Microsoft Security Response Center does it and fit to your scale or pick up the Security Development Lifecycle book and dig into the four-step process outlined.

The four steps of the emergency response process:

1.       Watch

2.       Alert and Mobilize

3.       Assess and Stabilize

4.       Resolve

Bugs, Bugs, Bugs

Gathering evidence that clearly shows your work has improved the security of your application is always a challenge. Trying to keep it lightweight adds to that challenge. The most effective way to create traceable and practical evidence without a lot of overhead is detailed management of security issues in your bug database.  The key here is that your bug database is configurable and able to be queried in a variety of ways to pull out this data. From the time you set out to implement this plan, be strict in tracking every discovery from threat modeling, the mitigations to those threats, and every bug you expose in tool analysis. This library of security bugs will give you an easy way to go back and gather evidence that shows the quantity of issues you discovered, the mitigations you used, and the impact the changes had on your application.

I have provided a fairly detailed view of these components. As I indicated, many of these defenses are available for you in Visual Studio 2005 SP1 or various linked resources above. If you are unsure whether you are taking advantage of all available defenses in your development tools, take the time to check.

It is my hope that some of you can use this scaled back entry into the principles of SDL to get moving toward improved security assurance. In the non-Microsoft SDL engagements I have been involved in, we have seen these steps effectively establish a baseline architectural understanding of your application security and identify critical weaknesses while providing solid evidence to support the decision to “run” forward into full SDL adoption.

[I want to thank Michael Howard for providing some of the key data for the Developer pieces in this article.]

Previously, James Whittaker posted a blog entry on Testing in the SDL in which he mentioned that many folks equate fuzz testing with security testing.  While fuzz testing doesn't come close to describing how security testing is done at Microsoft it does happen to be one of our most scalable testing approaches to detecting program failures that may have security implications. 

As Michael Howard has pointed out before, we do our best to ensure that the SDL incorporates lessons learned from vulnerabilities that required us to release security updates.  It turns out that the animated cursor bug patched in MS07-017 had a positive impact on the automatic triaging our fuzz testing tools perform.  In this post, I'd like to shed some light on how we monitor for program failures when fuzzing parsers and how the recent animated cursor bug, MS07-017 caused us to revisit and ultimately improve our fuzzing tools.

Background

For our purposes, fuzz testing is a method for finding program failures (code errors) by supplying malformed input data to program interfaces (entry points) that parse and consume this data (e.g. file, network, registry, shared memory parsers).  At Microsoft, we view fuzz testing as six distinct stages in which the output of each stage can impact or influence both the current and next iteration through the stages (e.g. after completing analysis work in stage 5 you could decide to change how you malform and deliver fuzzed data [stage 2 and 3], which exceptions get logged [stage 4], which tests you re-run [stage 6] and even which parsers you might decide to go after next [stage 1], etc).  Below is a brief listing of each stage and its associated tasks.

Stage 1: Prerequisites

• Identifying the targets (program interfaces to fuzz)
• Prioritizing your efforts (test planning)
• Setting Bug Bar

Stage 2: Creation of fuzzed data (malformed data)

• Will we be format-aware (e.g. most files follow a format)? Context-aware (e.g. order and/or timing of data may be important)?
• Will we use existing data (mutation) or generate it from scratch (generation)?
• Will the malformations we apply be based on type? Use interesting patterns? Over how many bits/bytes?
• Will we apply malformations with or without restriction? Are we going to be deterministic or random or both? How many times in a single iteration do we apply any given malformation?

Stage 3: Delivery of fuzzed data to the application under test

• Determining the best method to get the application under test to consume the fuzzed data (e.g. load path from cmd-line or GUI; API hooking; MITM proxies; DLL redirection; in-memory start-stop-rewind, etc)
• Implementing the appropriate delivery mechanism and conducting the test

Stage 4: Monitoring of application under test for signs of failure

• What should we look for?
• What do we do when we see it?

Stage 5: Triaging Results

• How can we classify and analyze issues found?

Stage 6: Identify root cause, fix bugs, rerun failures, analyze coverage data (rinse and repeat)

How we do file fuzzing

There are a number of approaches taken by product teams to meet the SDL file fuzzing requirements.  They often include the use of generation and mutation-based fuzzers as well as a combination of multiple internal and externally available fuzzing tools and/or frameworks. 

When fuzzing file parsers, we monitor for both handled and unhandled exceptions in the application under test.  Exceptions are events that typically represent error conditions encountered during the execution of an application.  They can be generated both by the hardware (initiated by the CPU) and/or software (initiated by the executing program or the OS).  To monitor for these exceptions, we created a mini-debugger using the Win32 Debugging APIs (For an example of how to integrate a debugger into your fuzz testing tool, check out Michael Howard and Steve Lipner's SDL Book at http://www.microsoft.com/MSPress/books/8753.asp).  The mini-debugger launches the application under test and monitors the parent and all subsequent child processes and associated threads.  When an exception occurred, the first version of this tool simply logged the file that caused the exception along with associated details such as the timestamp, exception code, exception address, stack trace and dump file.  More recent versions have included the ability to monitor for CPU and memory spikes as well as enabling full page heap settings on all processes launched from the mini-debugger.

As a general rule, all exceptions must be triaged (reviewed) by the tester to determine if a bug needs to be filed.  When fuzzing over a period of time however, we might generate hundreds of exceptions and it becomes a very labor-intensive process to sift through all of them.  What we needed was a way to ease the burden placed on the tester.

To that extent, the mini-debugger was extended to enable the automatic "bucketization" of logged exceptions to reduce the chance of having to look at duplicates during the triaging process.  This was accomplished by creating unique bucket ids calculated from the stack trace using both symbols and offset when the information is available.  The bucket id was used to name a folder that was created in the file system to refer to a unique application exception.  When an exception occurred, we calculated a hash (bucket id) of the stack trace and determined if we had already seen this exception.  If so, we logged the associated details in a sub-directory under the bucket id folder to which the exception belonged.  The sub-directory name was created from the name of the fuzzed file that caused the exception.  Thus, we were able to reduce the number of potential exceptions that a tester would have to look at during the triage process.  It is often the case that certain exceptions are noisy and/or expected so we also added the ability for the tester to dampen exceptions by exception code.  Dampening ensured that those exceptions were not logged (recorded) for triage during a fuzz run.  Nonetheless, despite our best efforts it is still possible for two different stack traces to have the same underlying root cause.

Even with all of this automated assistance, the tester might still have several hundred cases to triage.  In an effort to prioritize which cases should be triaged first, we introduced the notion of classifying exceptions.  Again, we extended the mini-debugger to perform classification on the exception code and relevant details.  In particular, we added an extra hierarchy over the automatically generated directory structure described above.  To do this we introduced the following categories of exceptions:

• Must Fix
• Further Investigation necessary
• Usually not exploitable

I know what you're thinking, but remember that this classification doesn't exclude a tester from the requirement of having to triage all exceptions.  The "Must Fix" category was composed of write access violations, read access violations on EIP, /GS and NX related access violations and read access violations where any one of the following was true*: 

• The access violation happens on a rep assembly instruction (on an Intel processor) where the count register (ecx) is large. 
• The access violation happens on a mov instruction where the result is used as the destination of a call in the instructions immediately after the mov.
• The access violation happens on a mov instruction where the result is later used in a rep instruction as the source (esi), destination (edi) or count (ecx).

*Fully automating the classification of these cases is complex and almost always requires an entire execution trace.   As such, teams are also provided with guidance to assist them during their analysis when our tool is unable to classify beyond "read and write access violations".

The "Further Investigation necessary" category was composed of read access violations that didn't meet the criteria above as well as other specific cases.  Finally, the "Usually not exploitable" category was composed of other exceptions such as divide-by-zero, C++ exceptions and the like.  Another thing to keep in mind is that the interpretation of "Usually not exploitable" is different for server-based components.  In other words, a divide-by-zero exception in a server product is probably more than just a robustness issue...it might be a denial of service!

Remember that regardless of this classification the tester is still required to triage all exceptions and file bugs accordingly.  I'll defer more details on the subject of exploitability of program failures to the upcoming annual security issue of MSDN Magazine in November.

To recap, we had a debugging plug-in (mini-debugger) that not only monitored for exceptions but also reduced the number of exceptions to triage after a fuzzing session was completed.  This also included monitoring for CPU and memory spikes as well as the use of page heap to capture heap corruptions that might not manifest themselves as an application crash (exception) during the fuzz session.  What could go wrong?  Enter MS07-017.  The software responsible for invoking the vulnerable code [to parse animated cursors] made use of an exception handler to recover from pretty much any exception that could be generated and continue operating as if nothing had occurred (Read more about it at http://blogs.msdn.com/sdl/archive/2007/04/26/lessons-learned-from-the-animated-cursor-security-bug.aspx).

The Animated Cursor bug caused us to revisit our mini-debugger.  Why?  Put simply, we hadn't introduced the "bucketization" and classification mechanisms for first-chance exceptions.  Naturally, this meant the tester was back to square one in terms of having no assistance on the labor-intensive triaging process. To deal with the "recover from anything" exception handling code we introduced the concept of classifying and bucketing "dangerous" first chance exceptions to help reduce the number of first chance cases the tester would need to triage.  This means we look for both write access violations and read access violations on EIP.  Additionally, we added support to continue after a first chance exception, allowing exception handlers to be called and continue and possibly proceed on to other more interesting crashes.

As you can see fuzz testing scales pretty well, but simplifying and scaling the triage process is not an easy task.  Even more challenging is the integration of technology into an effective lifecycle. We're constantly working with teams within Microsoft to further advance our tools, you can learn more by viewing http://research.microsoft.com/research/pubs/view.aspx?id=1333&type=Technical+Report and http://research.microsoft.com/Pex/. 

-Scott Lambert