My friend, I’m going to give you a hint.  If your firewall is “high risk” and your actual business applications are “low risk” - you might be doing it wrong.

The big difference between the two cities is the amount of time that the InteropNet team gets to produce a live, fully operational and redundant network. In Las Vegas, this was nearly a full week of time - a tight timeframe across 17 different vendors, but now we’re looking back at that timeframe as a luxury. In NY, we’ll be getting started Saturday morning, and the network needs to be delivered on Sunday morning for the registration desk and exhibitor move-in to begin. If you’re keeping score, that’s about 24 hours to deliver a working network. Sounds hard, but it’s even harder when you consider that this means four DS-3s from two different locations, 17 full and 7 half racks of network gear, all the fiber and copper that the network is delivered over, etc all have to get done. Good thing that with 2 and 3/4 kids, I’m not planning on much sleep, and I don’t think the rest of the team is either.

In order to try and get the network delivered in that short timeframe, we worked hard at Hot Stage to assure that everything is ready to go. With some luck, the work that we’ve done here will allow us simply to roll the network gear into place, run the cables, fire up and go.

Now, things never really work out that way but that’s what EM7 is going to be there for. We’ll watch in real time as the network elements come live and be able to let the other InteropNet vendors know if their gear isn’t behaving as expected or is not visible for all the areas of the network that it should be. We’ll keep track of all of this in the EM7 ticketing system so that after the show we’ll be able to analyze the behavior of the network and systems as we did after Vegas.

I’m looking forward to the show and once again working with some of the top engineers in the country on a complex and rapidly deployed network.  Speaking of which, we’re still looking for volunteers to help in the NOC.  Volunteers get to work with some really smart people, get an education that would be hard to get anywhere else, and get a trip to NY where your expenses (for things like hotel accommodations and food provided by the show) are taken care of.  Sound interesting?  Be sure and check out the application.

ShareThis

Ted Morgan, the head of Skyhook, explained in an interview that while GPS is certainly the gold standard, and while it works well in stand-alone devices designed for continuous use and navigation, it's not the right choice by itself for mobile devices. It can take 5 or 10 minutes for a GPS-only device to get an accurate fix on the satellites it needs to give you accurate information. (Various shortcuts can provide less accurate information more quickly.)

"This notion of 'tell a user or consumer to stand outside for 30 seconds before they can search for the nearest pharmacy' is pretty silly," Morgan said. He noted that with all the radios now found in newer mobile devices, using several of them produces a fast and much more accurate result. The iPhone 3G, for instance, sports quad-band 2G, tri-band 3G, Bluetooth, Wi-Fi, and GPS chips.

Morgan said that A-GPS (assisted GPS) already combines cell tower information with GPS. A cell phone can be told approximately where it is, and thus instead of cycling through 24 satellites, start with the two that are most directly overhead. This can reduce the time to gain a location to as little as 20 seconds, Morgan said, although any kind of movement usually lengthens the time to 30 to 60 seconds.

Skyhook's system takes advantage of this aspect of A-GPS. They let a GPS system grab onto two satellites quickly to correct data from their Wi-Fi Position System (WPS). Morgan said that this reduces the WPS error by 35 to 40 percent through "weak fixes."

Within cities' concrete canyons, "you can only get a true GPS fix about 70 percent of the time outdoor, but you get two satellites all the time," Morgan said. "In the entire footprint, we're able to use this hybrid technology, even though GPS is only available 70 percent of the time." Outside of metro areas, cell towers can still be used to improve GPS startup times.

Skyhook has continued to expand its European coverage for WPS; they cover about 8,000 cities in the US and Canada, which is roughly 70 percent of the population; "it looks exactly like a cellular coverage map," Morgan said, and includes "any town with five streets in it."

In Europe, their current big push, partly because of their inclusion in the iPhone, they cover 70 percent of population in the current countries--the UK, France, and Germany--but they're now at 50 percent of the population of the rest of Western Europe. They're working assiduously in Japan, Korea, Hong Kong, and Australia as well, and looking into China and India. India has very little Wi-Fi, so they may rely more on cell towers there.

The company also announced a partnership with wireless chip maker CSR today, which is a major providers of Wi-Fi and Bluetooth chips to computer and handset makers. Nearly a year and a half ago, Skyhook partnered with SiRF, the dominant worldwide chip supplier for stand-alone GPS gear, that's also making a push into mobile devices. Skyhook obviously needs a win with a cell chip maker, like Infineon, Broadcom, or Qualcomm, given the XPS technology, to score a place in tens of millions of cell phones beyond the iPhone.

Skyhook's technology most recently appeared in a soon-to-ship model of the Eye-Fi--the Explore. The $130 Secure Digital card with Wi-Fi built in allows you to take pictures with any camera, and have the Wi-Fi signal space recorded for later lookup when you upload photos. The pictures are geotagged with that information. The card can optionally be used with Wayport's 10,000 strong Wi-Fi network in the U.S for $15 extra per month. David Pogue of The New York Times recently wrote up the Eye-Fi Explore.

“Uh hi honey, sorry to wake you. Could you login to my email for me?”

Ah, the fun begins I thought.

“OK, my user name is [REDACTED] and my password is [REDACTED]”

Sigh, some folks just don’t get it. But, it gets better.

“No sweetie, that’s for my Gmail account”

Score.

At this point I glance around the train to see that I wasn’t the only person that found this fella’s call amusing. There were smirks to be seen. Then this takes a darker turn, for him at least.

“Now, sweetie I’ll need you to order a wireless router from [REDACTED] and my credit card number is 5, 5…

He proceeded to read out the entire number with expiry date and CVN. I was a little worried for the guy at this point. But, I guess Darwin was right. Then I heard a woman’s voice utter, “jackass”. I glanced up to see a little old lady shaking her head as she looked at the loud talker in disgust. A smile crept across my face.

1. We don’t believe that the goal of Quantitative Risk Analysis is to be precise.  We believe the goal is to be accurate. Subtle but important difference.
2. FAIR can be used both Quantitatively and Qualitatively.   The decision on which method to be used depends on various factors that Steve lays out nicely in the article there.
3. We believe that Risk Management is more than looking at specific vulnerabilities, their likelihood and impact.  It must encompass all aspects of the organizations ability to effect the probable frequency and magnitude of loss on an aggregate level, not just within the context of a discreet technical or policy issue.

That last point is important.  And it’s related to my post today.

WHAT DO YOU MANAGE TOWARDS?
This blog is blessed to have some very smart people be part of it.  There are security managers from all sorts of industries that read and comment and contribute.   And so today’s blog is more of an open-ended question for you all.  It’s a question that, if I have a comfortable relationship with the organization I like to first ask the senior manager, and then subsequently ask the direct reports.

When you think about it, Sales & Marketing managers have goals they manage towards.  CFO’s have goals that they manage towards.  COO’s have goals and measurement that they manage towards (cost management, production, etc…).  So what does the CSO manage towards?  I’m guessing if we took a national poll, we’d get all sorts of very nice sounding answers to that question.  I thought I’d list some of the answers I’ve heard and talk about them with you today.

1.)  Being Secure or “Managing to Security”

Generally, this concept of being secure is the most common answer.  And when I’m given that answer, it generally means that management focuses on Vulnerability Management, Patch Management, and to some degree, log analysis from various sources.  These are very basic core security functions, and the  belief is that if we do these well, we will be “secure”.  Ok, well… what does this “secure” mean, and how can we talk to management about whether we are meeting this goal?   If you examine that question, you actually find out what a “Being Secure” organization is really managing towards, another answer I hear often:

2.)  Being Incident-Free or “Managing to Perfection”

Security Person:  “Alex, our goal is not to have any incidents.”  Alex:  “Good luck with that.”

OK, that’s not what I really say, but here’s the problem I see with this common answer and the one above both of these common answers:  How do you know if you’re good or just lucky?

Well, are you, punk? (youtube link)

In my six years of working with a Penetration Testing team, nobody ever really “passed” with a perfect score*.  Some did better than others, some folks looked really, really good - but the degree  of good/bad was really more dependent on scope than the actual state of controls or the ability of the team to overcome them.  That is to say, when pressed, the mature security professional must admit that, given a strong, capable threat community -  there is no secure.   And therefore any state of “incidentlessness” deals with some combination of amount of control strength, and some lack of attacks (frequency!) by someone with enough skills and resources to overcome those controls.  If that last sentence sounds very FAIR-Like, that’s because it is.  If FAIR really accounts for those things that create Risk, then Managing to security or lack of incident means that you’re primarily concerned with FAIR Vulnerability, and ignoring other critical aspects of risk (like frequency of attacks, controls that reduce the probable impact of an event due to an ability to respond well to external stakeholders, etc…).

3.) Being Compliant or “Managing to Compliance” (External Compliance Pressures)

Because that’s what business buy, right?  They buy compliance!   Or so I’m told.  So let’s say that you go out and actually twist senior managements arm to get them to cough up enough dough so that you can be as compliant as Large Accounting Firm says you need to be.  Good on you!

But what I always wonder is, what happens when you want to manage something beyond compliance?  What happens when the checklist you’re managing towards is run by a bureaucracy that can’t keep up with a changing threat landscape?   For many people, the answer is “GOTO 1″ and try to sell upper management using FUD (hey, it used to work, maybe it’ll work again).  An alternative is to get to the next step:

4.)  Being Measured or “Managing to Metrics”

Say what you will, but “quants” have one thing right.  What gets measured gets done.  And a few mature organizations have spent a ton of time and effort on being able to create dashboards of KPI’s that attempt to measure security.  Problem is, that we don’t know if a 98% on patch levels is good or bad or just right.  We don’t know what value, if any, does creating metrics around the number and severity of vulnerabilities found in a monthly scan actually have.  So we’ve come up with this thing called “GRC” that’s supposed to make sense of those things we can measure empirically and help you find out if/when you’ve fixed them. And while GRC tools can tell you some good information about systems out of compliance, they tend to give you fantastic information like how your “risk = 57“.

Wha….?

Risk = 57 means very little to someone who doesn’t spend their life in the machinations of the GRC indicies.  So again, measurement without a (good) model still falls down when faced with that ultimate business decision.  Or, as Shurdlu so eloquently puts it in her post on GRC:

“By contract, risk is personal.  It’s variable as hell.  It “governs” what you spend your money on, and therefore, with or without a dashboard, your CEO is already doing risk assessment every time she decides what your security budget is going to be.  Will you really be able to change her mind by showing her the dashboard and saying, “But—but—the needle is pointing to RED!” when you’re sitting there with your line items in your fiscal shopping cart? “

5.)  Using Risk or “Risk Management”

Which brings us to my favorite, using risk (as defined as the probable frequency & probable magnitude of loss event(s)) as a means to manage.  Now many industry veterans will tell you how jaded we all are on the term “Risk Management”.  And we have every right to be, as Risk Management has been horribly abused by vendors, committees and standards bodies alike.

These days, the term has been narrowly defined to mean an extension of vulnerability management.   This is small, small thinking, IMHO.  To me, Risk Management isn’t the management of individual issues deemed as “risky” as much as it is measuring (see 4) our ability to make decisions through the lens of risk.  Maybe I should start saying “Risk-Based Management” instead of “Risk Management”.

This Risk-Based Management approach provides meaning to metrics. We can know what we’re measuring and why we care about it.  And why we care about it needs to match what management cares about.  If your approach to Risk Management results in some metric or KPI that non-IT (or non-security) management doesn’t understand or speak to them in an evident language, it’s time to find a new model.  This is why “Quants will win” and where risk = 57 is wrong.  Risk, expressed as “expect a once in 5 year chance to lose $875,000 if we don’t spend $90,000 now” actually gives executives something beyond an arbitrary ordinal number or color to work with.  And what’s interesting is, if your model does the right things in getting you to that expression - then metrics and KPIs - those “why/when/where” questions we have a tough time answering about metrics - they become easier to discover.

DISPROVING RISK MANAGEMENT

As a side note, originally I was going to write today a completely different post on how we can disprove whether or not OCTAVE or 800-30 or ISO 27001 risk management efforts are really “Risk Management” - and one significant point was “Does your non-IT management really care about the deliverable?”   This thought came to me after seeing a few too many emails into the ISO27001 mailing list asking “How can I get management to fund ISO 27001 certification?”  Of course, the value of implementing the ISMS and the value of certification are two separate business propositions, but if you can’t sell the first, then are those efforts really good risk management?  You know, the kind of effort that we can use to make meaningful reporting?

=============================

* I can tell you that at times we were asked to test products out for clients before they made a significant investment.  One biometric device stands out in memory as not being “hacked” in the time alloted for the engagement by a defense contractor.  After it passed the “Gummi Finger” test - we were going to try using a recently severed finger, but oddly enough nobody on the team volunteered their digit for the sake of security.  Bunch of slackers.

What can we conclude?

First, good documentation never hurts :-) - indeed, the most popular information to look for when facing a new log record is documentation on what it means. While some software vendors are great in this regard, many other don't bother documenting their logs or document them only when customers complain.

Second, I was not sure that the second popular choice would be "Other logs from about the same time (this and other systems)."  This strongly points at huge value of cross-device log analysis (see this recent log entry on that),  where all the logs are consolidated and analyzed together (it goes without saying that time is synchronized OR at least corrected across those logs). Indeed, if you are confused about a log and documentation is not available, reviewing "what else was/is going on?" is smart. Trusting log time stamps across many systems is also key for that.

Third, having IP addresses in logs is great, but human-readable names are better: IPs in logs needs to be mapped to DNS or Netbios names. Indeed, given that often such names reveal where the system is, who might own it, what its function is, etc this information is not just a mapping, but true log information enrichment.

Fourth, so, what's next? The above 3 top responses are indeed universally useful, but the next choice digs deeper: flows, packets, connections and other network information does complement logs and is often studied in combination with logs (e.g. see a strange log entry then go see who connected to the system at that time or where the system itself connected to).

Fifth, next comes a group of pretty much everything else: other logs from the same system, logs about the same system as well as loosely defined 'similar' log entries. These come handy, but are not top choices. In fact,  from this I conclude that a lot of additional context information is needed to make sense of a confusing log entry.

Sixth, what was surprising? I thought that identity lookups (e.g. IP to real name or other user identity information) would score higher.  I also suspect that people were confused by "logs ABOUT the same systems" (what I meant is, for example, use firewall logs that mention the system which log we are now analyzing) and this should score higher.

Seventh, anything fun in the "Other" category? Yes, there were a few insightful ones: first, results of a Google search (supposedly for the info from the log entry in question)! Very true indeed. Also named were logs from the same daemon/program (how can I miss it?),  logs from previous incidents and information on the logging system owner.  All very useful indeed. Thanks for good ideas!

Finally, a brief message to people that work for a certain log-related vendor of ill repute who keep polluting my polls: if I catch you, I will kick you in the butt :-) Or, better, I will hammer you with a big and heavy log (you know, the wooden kind) over your miniscule heads ...

Past logging polls and their analysis:

There was always that one kid in class. You know, the one that didn’t always get it. Or spent most of the day staring out the window. Daydreaming knuckle heads that were nowhere near inclined to excel. Well, it appears that they US gov’t is one of those kids. Well, on average anyway.

From the Washington Post:

The federal government earned an overall grade of “C” for securing its computer systems and networks from cyber attack last year, a slight improvement from the “C-minus” mark the government was given in 2006.

The report cards were issued today by Rep. Tom Davis of Virginia, the ranking Republican on the House Committee on Oversight and Government Reform.

Nine agencies earned failing grades for 2007, including the departments of Agriculture, Commerce, Defense, Interior, Labor, Transportation, Treasury, Veterans Affairs, as well as the Nuclear Regulatory Commission. The grades are based on data submitted by the agencies and agency inspector generals to the White House for fiscal year 2007.

There are a couple bright spots. The DOJ, SSA, EPA and the GSA were among eight agencies that managed to score an “A” on their report card. They get to go to McDonald’s.

But, the NRC gets no hot apple pie with their happy meal.

Article Link

As I touched on in my December posting on Common Criteria, and as Michael Howard discussed in his post on security metrics, trying to objectively quantify and measure “How secure is secure” is far more difficult than one might think. I’d like to share my perspective that there are two “dimensions” useful to consider when characterizing software security metrics: security functional requirements and security engineering quality requirements. While the SDL is focused primarily (but not exclusively) on the latter, both are ultimately important when assessing the security of a given bit of software. However, for reasons I’ll elaborate on below, the SDL does focus on trying to prevent the most common causes of vulnerabilities today and hence looking at the ways in which Microsoft tracks and measures individual products teams’ compliance with SDL requirements offers some interesting fodder for the security metrics debate. I’m not offering a complete solution, but am sharing our experience at Microsoft with measuring how development teams actually follow the SDL. It’s helped us deliver more secure software, and sharing this will hopefully help others as well as putting more data on the table for consideration when discussing security metrics.

Putting aside computer security for just a moment, it’s interesting to look at other ways in which we attempt to measure security in our society. Padlocks offer security protections, and organizations such as the American Standard for Testing and Materials (ASTM) provide standards like F883-04 Standard Performance Specification for Padlocks that characterize padlock security ratings. Prisons provide security protections as well. Prisoners reside in different facilities that vary by security level. The US Bureau of Prisons uses a numbered scale from one to six to represent the security level. Both of these examples are similar in that the threats and risks each of them must protect against are reasonably well understood and relatively static (meaning the threats don’t change much over time). Computer security is still evolving with new classes of attacks still being discovered, and while hackers understand how to exploit known types of vulnerabilities – software developers are still catching up in learning how to modify engineering practices to be resilient against both new and old types of attacks. Hence, metrics are more challenging for computer security.

Several attempts have been made by governments to come up with a security rating system similar to the examples listed above. In the 1980’s, the US Department of Defense created the “Trusted Computer System Evaluation Criteria (TCSEC)” that tried to establish a standard for measure operating system security. The “Orange Book” offered a relatively simple system for assigning “score” summarized below:

D (Minimal Protection)

C (Discretionary Protection)

C1: Discretionary Security Protection

C2: Controlled Access Protection

B (Mandatory Protection)

B1: Labeled Security Protection

B2: Structured Protection

B3: Security Domains

A (Verified Protection)

A1: Verified Design

In the 1990’s, the US and other nations combined their efforts to create an international security standard for software known as the Common Criteria (ISO 15408). Common Criteria also has a rating system that scores products with “evaluation assurance levels” (EALs):

EAL 1: Functionally Tested

EAL 2: Structurally Tested 

EAL 3: Methodically Tested and Checked 

EAL 4: Methodically Designed, Tested, and Reviewed 

EAL 5: Semi-formally Designed and Tested 

EAL 6: Semi-formally Verified Design and Tested 

EAL 7: Formally Verified Design and Tested 

Both TCSEC and Common Criteria (CC) are primarily focused on “security functional requirements” (as called out earlier, distinct from “security engineering quality requirements”). The EALs reflect the amount of rigor and attention to claimed security functional requirements a developer applied while creating a product. Furthermore, the EALs also reflect increasing levels of effort and resources necessary by anyone reviewing a product in order to evaluate the product’s claimed security functional requirements. However, EAL ratings for commercial products have historically not correlated with the number of vulnerabilities found in commercial products after release. As I discussed in my December posting on Common Criteria, this is because CC is primarily focused on “security functional requirements” and fails to adequately address “security engineering quality requirements”.  This leads a question on how to measure those aspects of software security that earlier efforts have been unable to successfully address. 

Microsoft has been releasing security bulletins since 1999. Based on some informal analysis that members of our organization have done, we believe well over 50% of *all* security bulletins have resulted from implementation vulnerabilities and by some estimates as high as 70-80%. (Some cases are questionable and we debate if they are truly “implementation issues” vs. “design issues” – hence this metric isn’t precise, but still useful). I have also heard similar ratios described in casual discussions with other software developers. In other words, most vulnerabilities can be addressed by the “security engineering quality requirements” described via SDL. This is not to say that “security functional requirements” are unimportant or that SDL ignores secure design (as Adam has described in his threat modeling series), but rather that it is not where vulnerabilities are being most frequently encountered. With SDL, we adopt a pragmatic approach in looking at identifying the root causes of security vulnerabilities, and trying to prevent those root causes from reoccurring. The challenge lies in how we actually validate that development teams are indeed adopting and executing whatever changes SDL requires in engineering (either in terms of process or tools). Process changes are often difficult to quantify, as we must rely upon development teams truthfully attesting they have followed the process. As long as development teams believe the process results in better code, they generally will adopt and follow such practices. Tool usage becomes more interesting and valuable in that using tools becomes a vehicle for objectively and independently verifying if code satisfies requirements or not. But that is just the tip of the iceberg…

As I said above in my comments on EALs, the amount of time required by anyone reviewing a product to assess “security” is relevant since security review can be a very time and resource intensive activity. However, running static code analysis tools, verifying build tools and switches, searching for banned APIs, and recording the output of other tools that inspect code and/or binaries for potential implementation vulnerabilities is a key element in how we approach the challenge of trying to measure compliance with SDL requirements from product groups at Microsoft today. While not every technique required by SDL has a corresponding tool, we try to provide both tools and automation if and wherever possible. There is still much work to be done in terms of standardizing tool output formats and creating automation to assess tool output. However, these “grass roots” metrics derive from practical experience of changing engineering requirements based on actual vulnerabilities. We look objectively at what is causing vulnerabilities, and target solutions to address the root causes of those issues. As the saying goes, “If it hurts when you do that, stop doing that”. If what we have done in the past has hurt our customers by creating vulnerabilities requiring security bulletins, we want to stop doing that. J

The challenge in using a plethora of individual detailed metrics such as I describe above (that we do internally at Microsoft for measuring SDL compliance), is that they don’t roll up into a nice aggregate score that customers can easily understand.  However, they have translated into reduced numbers of vulnerabilities as Michael Howard wrote a few weeks ago. Coupling these types of scores with assessment of compliance with “security functional requirements” might be the basis for coming up with a metric that is useful to customers, both in the government and private sector.

What do you think?

Yesterday, Microsoft published the new Security Intelligence Report for the 2nd half of 2007. (home page is http://www.microsoft.com/sir, and the download page is here).

As one of the contributors for the report, I'd like to highlight the findings summary for the Industry vuln trends:

• Vulnerability disclosures decreased by about 5 percent in 2007, reversing a multiyear trend of increasing disclosures. Almost all of this decrease was observed in the second half of the year, which had the fewest disclosures since 2H05.
  
• Despite the decrease, the number of new disclosures across the industry remains in the thousands, with the number of disclosures in 2007 surpassing that of every other year in the study except 2006.
  
• The Common Vulnerability Scoring System (CVSS) used to score vulnerabilities in the NVD was revised in 2007 to increase its accuracy, consistency, and applicability. Retroactively applying the new formula to vulnerabilities disclosed in previous years classifies a much higher percentage of vulnerabilities as High-severity than was previously
  the case. The vulnerabilities disclosed in 2007 continue this trend, with High-severity vulnerabilities accounting for about half of the total number of vulnerabilities.
  
• Vulnerabilities requiring a Low-level of complexity in order to exploit accounted for
  about half of all vulnerabilities disclosed in 2H07. Although this number is relatively
  large, the number has declined significantly from earlier periods.

Here is the high level trend chart from the report:

Regards ~ Jeff