Specification versus Analysis

When Larry Osterman was writing about threat modeling, he casually tossed out:

A threat model is a specification, just like your functional specification (a Program Management spec that defines the functional requirements of your component), your design specification (a development spec that defines the architecture that is required to implement the functional specification), and your test plan (a test spec that defines how you plan on ensuring that the design as implemented meets the requirements of the functional specification).

Just like the functional, design and test specs, a threat model is a living document - as you change the design, you need to go back and update your threat model to see if any new threats have arisen since you started.

I found this pretty surprising. I think of threat modeling as an analysis technique, but hey, if test plans are specs, threat models aren’t that different. (It took some private back and forth with Larry to convince me.) This brings me to the topic of what words we use to describe things.

On language

[The English language] becomes ugly and inaccurate because our thoughts are foolish, but the slovenliness of our language makes it easier for us to have foolish thoughts. The point is that the process is reversible. Modern English, especially written English, is full of bad habits which spread by imitation and which can be avoided if one is willing to take the necessary trouble. If one gets rid of these habits one can think more clearly, and to think clearly is a necessary first step toward political regeneration: so that the fight against bad English is not frivolous and is not the exclusive concern of professional writers. (George Orwell, Politics and the English Language.)

We ask people to threat model (as part of the SDL) by threat modeling their application. To do that, they model the design, and then have a threat modeling meeting in which someone runs the threat modeling tool to produce a threat modeling document. In our particular case, I’m not sure it’s easily avoided.

“We ask people to analyze their designs (as part of the SDL). To do that, they diagram the design, then have a threat modeling meeting in which someone runs the codename tool to produce a threat modeling document.”

In fact, I’ve worked hard to reduce jargon in the process, but at one point, went too far. I tried to convince people to say “diagram” instead of DFD, and they revolted. Actually, they didn’t really revolt, but they did refuse to go along. Experienced threat modelers actually felt that the term DFD was more clear than saying ‘diagram,’ because DFD includes a specification of what kind of diagram, and they had already integrated the acronym into their thinking. Having integrated it, they could use it without thinking about it. Well, as Adam Barr points out, “you can pick your friends, and you can pick your nose, but you can't pick your online community's lexicon.”

It didn’t add to their cognitive load to say “DFD.”

Cognitive Load

People can only handle so much new information at one time. Learning to drive a stick shift is harder than learning on an automatic, because there’s a whole set of tasks you can ignore while the automatic transmission handles them for you. In my approach, this relates pretty closely to the concept of flow. If you’re so focused on rules and jargon, you can’t focus on what you should be building. Cool, well-designed features to help your customers.

In Conclusion

We started this series by looking at the trouble with threat modeling. How people had trouble getting started, relating the process to their other work, or validating what they’d done. We’ve looked at the new threat modeling process, and the steps involved. We talked about how to get into the flow with threat modeling. How clear goals, direct and immediate feedback, and balancing ability and challenges set people up to focus their attention and succeed. We’ve talked a bit about making threat modeling work better by adding structure to chaos, and how to use self-checks and rules of thumb to give people confidence they’re on the right trail. We’ve talked a very little bit about how to customize the process for your own needs, and where that customization can be dangerous.

All of this has come out of looking at our threat modeling activity as a human activity, and asking how we can best shape it to help people get to the results that we want. I hope that our readers have enjoyed the series, which we’ve converted into a downloadable document (here.)

In my last post, I talked about how almost everyone in software draws on whiteboards regularly, and this makes it an ideal first step. It’s an ideal first step because everyone can do it, see that they’ve done it, and feel like they’re making progress.

That wasn’t quite complete. Not only do we want people to see that they’ve done it, we want to give them a way to validate their work or other people’s work. So we ask them to tell a story. We’re not asking for Shakespeare here, we’re asking them to explain how their software will be used, and to make sure that their diagram supports that story, and that it relates to their actual software.

We also give them rules of thumb (lots of rules of thumb) about things we often see wrong in diagrams:

• Don't have data sinks: you write the data for a reason. Show who uses it.
• Data can’t move itself from one data store to another: show the process that moves it.

(Larry Osterman has some in his blog post, "Threat Modeling Rules of Thumb" I helped edit those, but want to suggest additional changes. In particular, “you need to be concerned” is not actionable. “Review this carefully,” or “Focus your attention here” are more actionable. People threat modeling are already concerned.)

Good “rules of thumb” encourage flow by empowering people to make a snap decision and move along.

One of the largest changes that we’ve made is to a simplified process (and diagram). I like to say that this looks pretty much like every other software process diagram you see today. That’s intentional. There’s only so much we can expect people to take away from a class, and making this simple and familiar helps ensure there’s room for the other important parts.

First, the “process hamster wheel,” (with apologies to Yankee Group analyst Andy Jaquith):

Now that you’ve seen the wheel, I’ll briefly describe the steps:

1. Vision: Consider your security requirements, scenarios and use cases to help frame your threat modeling. What are the security aspects of your scenarios? What do your personas expect or hope doesn’t happen? What are the security goals of the system you’re building, and how do those interact with the system as it stands?
2. Model: The basic idea is to create a diagram of your software, showing all trust boundaries.

a. Draw a diagram of your software. We encourage use of the DFD formalisms, which Larry Osterman describes in this post.

 

Essentially, the elements are

1. External entities (anything outside your control)
2. Processes (running code)
3. Data stores (files, registry entries, shared memory, databases)
4. Data flows (which connect all the other elements)

b. Draw trust boundaries between components. You can do this on a whiteboard, in Visio, or in one of the specialized threat modeling tools we’ve built. (A trust boundary is anywhere that more than one principal can access an object, such as a file or process.)

c. If your trust boundary crosses something which isn’t a data flow, you need to break it into two logical elements, or draw a sub-diagram with more details. (This is different advice: we used to tell people trust boundaries could only cross data flows. People drew them anywhere that felt right. We decided to go with what people were doing—there was important information in what they were expressing.)

d. If you need more details to express where trust boundaries are, add an additional diagram.

e. When you don’t have any more trust boundaries to draw, you’re done.

f. If a diagram doesn’t have any trust boundaries, you may have drawn too many details.

3. Identify Threats using STRIDE/element

For each element in your diagram, consider threats of the types indicated in this chart. (We’ll come back to the chart’s origins in a later post.)

There’s an important mis-conception we often see, which is that STRIDE is appropriate for use as a classification system. It’s really hard to use STRIDE to describe attacks—the impacts blend together really quickly. The most valuable use of STRIDE is to help people think about how threats have impacted elements of a design in the past. That is, it’s a framework for finding threats, not for describing them. “What if someone spoofs this host?”

 

4. Mitigate

Here on the SDL strategy team, we love threat modeling. We know that not everyone feels that way, and we ask teams to threat model so that they can find and mitigate threats. A threat model document with great diagrams and lots of threats isn’t very useful if you don’t take the key step of addressing the issues you find. There are four ways to do that:

a. Redesign to eliminate threats.

b. Use standard mitigations, such as those provided by OS features, to mitigate threats.

c. Invent new mitigations, understanding that this is a subtle art.

d. Accept risk, when allowed by the SDL

5.  Validate

There are two levels of validation. The first is within each stage, the second is a validation pass at the end of the process. That end of process validation entails:

a. Make sure that the diagrams are up-to-date and accurate

b. Ensure that you have STRIDE threats per data flow that crosses a trust boundary, and for each element that such a trust boundary connects to

c. Make sure you’re mitigating each threat

i. You have a bug filed per threat that you want to mitigate. The bug should be of the form “attacker can do X. Proposed fix: Y.” You might include tradeoffs you’re making, and possibly have test plans in the bug, if you include those.

ii. You have a valid reason for each non-mitigated threat not being mitigated.

iii. All threats are in class i or ii.

5.a. On change, re-validate

 

This hamster wheel has a very intentional brake on it: the word change, above validate. What that means is you want to go through the process again when you make changes that need to be on the diagram. Checking to see if your diagrams change is a relatively simple check that allows people to track changes against their threat model as their design iterates.

In the next post, I’ll talk about the reasoning behind the design, and offer up some process tools that anyone can use to make a process more user-friendly.

I said recently that I wanted to talk more about what I do. The core of what I do is help Microsoft’s product teams analyze the security of their designs by threat modeling.   So I’m very concerned about how well we threat model, and how to help folks I work with do it better.   I’d like to start that by talking about some of the things that make the design analysis process difficult, then what we’ve done to address those things.  As each team starts a new product cycle, they have to decide how much time to spend on the tasks that are involved in security.  There’s competition for the time and attention of various people within a product team.  Human nature is that if a  process is easy or rewarding, people will spend time on it.  If it’s not, they’ll do as little of it as they can get away with.  So the process evolves, because, unlike Dr No, we want to be aligned with what our product groups and customers want

There have been a lot of variants of things called “threat modeling processes” at Microsoft, and a lot more in the wide world.   People sometimes want to argue because they think Microsoft uses the term “threat modeling” differently than the rest of the world.  This is only a little accurate.  There is a community which uses questions like “what’s your threat model” to mean “which attackers are you trying to stop?”  Microsoft uses threat model to mean “which attacks are you trying to stop?”  There are other communities whose use is more like ours.  In this paragraph, I’m attempting to mitigate a denial of service threat, where prescriptivists try to drag us into a long discussion of how we’re using words.)   The processes I’m critiquing here are the versions of threat modeling that are presented in Writing Secure Code, Threat Modeling, and The Security Development Lifecycle books.

In this first post of a series on threat modeling, I’m going to talk a lot about problems we had in the past.  In the next posts, I’ll talk about what the process looks like today, and why we’ve made the changes we’ve made.   I want to be really clear that I’m not critiquing the people who have been threat modeling, or their work.  A lot of people have put a tremendous amount of work in, and gotten some good results.  There are all sorts of issues that our customers will never experience because of that work.  I am critiquing the processes,  saying we can do better, in places we are doing better, and I intend to ensure we continue to do better.

We ask feature teams to participate in threat modeling, rather than having a central team of security experts develop threat models.  There’s a large trade-off associated with this choice.  The benefit is that everyone thinks about security early.  The cost is that we have to be very prescriptive in how we advise people to approach the problem.  Some people are great at “think like an attacker,” but others have trouble.   Even for the people who are good at it, putting a  process in place is great for coverage, assurance and reproducibility.  But the experts don’t expose the cracks in a process in the same way as asking everyone to participate.

Getting Started 

The first problem with ‘the threat modeling process’ is that there are a lot of processes.   People, eager to threat model, had a number of TM processes to choose from, which led to confusion.  If you’re a security expert, you might be able to select the right process.  If you’re not, judging and analyzing the processes might be a lot like analyzing cancer treatments.  Drugs?  Radiation?  Surgery?  It’s scary, complex, and the wrong choice might lead to a lot of unnecessary pain.   You want expert advice, and you want the experts to agree.

Most of the threat modeling processes previously taught at Microsoft were long and complex, having as many as 11 steps.  That’s a lot of steps to remember.  There are steps which are much easier if you’re an expert who understands the process.  For example, ‘asset enumeration.’  Let’s say you’re threat modeling the GDI graphics library.  What are the assets that GDI owns?  A security expert might be able to answer the question, but anyone else will come to a screeching halt, and be unable to judge if they can skip this step and come back to it.  (I’ll come back to the effects of this in a later post.)

I wasn’t around when the processes were created, and I don’t think there’s a lot of value in digging deeply into precisely how it got where it is.  I believe the core issue is that people tried to bring proven techniques to a large audience, and didn’t catch some of the problems as the audience changed from experts to novices.

The final problem people ran into as they tried to get started was an overload of jargon, and terms imported from security.  We toss around terms like repudiation as if everyone should know what it means, and sometimes implied they’re stupid if they don’t.  (Repudiation is claiming that you didn’t do something.  For example, “I didn’t write that email!,” “I don’t know what got into me last night!”  You can repudiate something you really did, and you can repudiate something you didn’t do.)  Using jargon sent several unfortunate messages:

1. This is a process for experts only
2. You’re not an expert
3. You can tune out now
4. We don't really expect you to do this well

Of course, that wasn’t the intent, but it often was the effect.

The Disconnected Process

Another set of problems is that threat modeling can feel disconnected from the development process.  The extreme programming folks are fond of only doing what they need to do to ship, and Microsoft shipped code without threat models for a long time.  The further something is from the process of building code, the less likely it is to be complete and up to date.  That problem was made worse because there weren’t a lot of people who would say “let me see the threat model for that.”   So there wasn’t a lot of pressure to keep threat models up to date, even if teams had done a good job up front with them.  There may be more pressure with other specs which are used by a broader set of people during development.

Validation

Once a team had started threat modeling, they had trouble knowing if they were doing a good job.  Had they done enough?  Was their threat model a good representation of the work they had done, or were planning to do?  When we asked people to draw diagrams, we didn’t tell them when they could stop, or what details didn’t matter.  When we asked them to brainstorm about threats, we didn’t guide them as to how many they should find.  When they found threats, what were they supposed to do about them?  This was easier when there was an expert in the room to provide advice on how to mitigate the threat effectively.   How should they track them?   Threats aren’t quite bugs—you can never remove a threat, only mitigate it.  So perhaps it didn’t make sense to track them like that, but that left threats in a limbo.

"Return on Investment"

  The time invested often didn’t seem like it was paying off.  Sometimes it really didn’t pay off.    (David LeBlanc makes this point forcefully in “Threat Modeling the Bold Button is Boring”) Sometimes it just felt that way—Larry Osterman made that point, unintentionally in “Threat Modeling Again, Presenting the PlaySound Threat Model,” where he said “Let's look at a slightly more interesting case where threat modeling exposes an issue.”  Youch!  But as I wrote in a comment on that post, “What you've been doing here is walking through a lot of possibilities.  Some of those turn out to be uninteresting, and we learn something.  Others (as we've discussed in email) were pretty clearly uninteresting”  It can be important to walk through those possibilities so we know they’re uninteresting.  Of course, we’d like to reduce the time it takes to look at each uninteresting issue.

Other Problems

Larry Osterman lays out some other reasons threat modeling is hard in a blog post: http://blogs.msdn.com/larryosterman/archive/2007/08/30/threat-modeling-once-again.aspx
 

One thing that was realized very early on is that our early efforts at threat modeling were quite ad-hoc.  We sat in a room and said "Hmm, what might the bad guys do to attack our product?" It turns out that this isn't actually a BAD way of going about threat modeling, and if that's all you do, you're way better off than you were if you'd done nothing. 

Why doesn't it work?  There are a couple of reasons:

It takes a special mindset to think like a bad guy.  Not everyone can switch into that mindset.  For instance, I can't think of the number of times I had to tell developers on my team "It doesn't matter that you've checked the value on the client, you still need to check it on the server because the client that's talking to your server might not be your code.".

Developers tend to think in terms of what a customer needs.  But many times, the things that make things really cool for a customer provide a superhighway for the bad guy to attack your code. 

It's ad-hoc.  Microsoft asks every single developer and program manager to threat model (because they're the ones who know what the code is doing).  Unfortunately that means that they're not experts on threat modeling. Providing structure helps avoid mistakes.

With all these problems, we still threat model, because it pays dividends.  In the next posts, I’ll talk about what we’ve done to improve things, what the process looks like now, and perhaps a bit about what it might look like either in the future, or adopted by other organizations.