Scrawlr quickly came under fire on the Web Security mailing list for having some pretty major limitations. Billy Hoffman et al have been quick to point out that the tool was designed to address a very specific subset of SQL Injection vulnerability — the type affected by the mass attacks — and is not designed to be a general purpose replacement for existing SQL Injection scanners. Let’s look at the limitations, as outlined on the HP page, one by one.

Limitation: Will only crawl up to 1500 pages

Depends on what they mean by 1500 pages. For example, if I have these links on my front page, is that one URL or three?

• http://www.veracode.com/blog/?p=111&foo=1
• http://www.veracode.com/blog/?p=111&foo=2
• http://www.veracode.com/blog/?p=111&foo=3

Or, does it mean that it will really only crawl 1500 pages total, so if I have the same link 1500 times on the front page, it won’t go any further? Either way, for most smaller websites this is probably fine. If you need more than 1500 you could give it different starting URLs in an attempt to improve coverage. It would be nice to have a clearer definition of what it means to “crawl up to 1500 pages” though.

Limitation: Does not support sites requiring authentication

Well, this will render it useless for the majority of enterprise apps. But there are still a lot of sites out there that don’t require authentication, including some of the ones that got hit during the mass attacks, such as the United Nations, UK government, etc.

Limitation: Does not perform Blind SQL injection

They have taken a lot of flack for this but Billy describes it as a conscious choice:

An early version of the tool checked for blind SQL injection, but the final verison of Scrawlr did not. … The biggest feedback we got from early testing was developers wanted to “see” the vulnerability. Differential analysis is kind of difficult to visualize in a way that is helpful for the average dev, and pulling the table names through blind was too much of a performance issue.

I can sort of understand this rationale. Blind SQL Injection testing is much more susceptible to false positives. As users of any commercial web scanner or source code analyzer will attest, the more time you spend chasing down FPs, the less likely you are to put any faith in future results. It’d be nice if there was a way to toggle Blind SQL Injection testing on and off, though (could be off by default so nobody gets confused).

Limitation: Cannot retrieve database contents

Who cares? Find and fix the vulnerability. Pulling down the entire database “because you can” is a total ego move.

Limitation: Does not support JavaScript or flash parsing

Nobody does this very well anyway, particularly the JavaScript part. Writing a great crawler is probably the hardest part of writing an automated web scanner and it’s one of the biggest differentiators from one product to the next. You’re not going to get that for free.

Limitation: Will not test forms for SQL Injection (POST Parameters)

This is probably the toughest one to swallow. It’s not that difficult to parse out forms from HTML, and form POSTs can represent a major chunk of the attack surface. Granted, the Chinese tool associated with the mass attacks did operate solely on GET requests (i.e. parameters in the query string) so HP can defend this again by saying the tool is really aimed at the sites being targeted by the mass attacks. I think it’s a little short-sighted though; chances are that the mass attacks will evolve and it’s better to be proactive about it than reactive.

Conclusion

It’s tough to bash someone for releasing a free tool. I personally think HP should add an option for enabling Blind SQL Injection testing, and that they should consider supporting POSTs as well as GETs. You’re basically getting a (massively) stripped-down WebInspect for free, so take it for what it is. No single tool is a panacea.

The jury is still out on how effective Scrawlr is against the things it does claim support for. Keep watching the Web Security list; the reviews are filtering in.

In reply to Mark Palmer’s rebuttal, What Does it Mean to be Mature?, the figure below illustrates the popular Gartner Hype Cycle.  You can click on the illustration to get a clearer image.

In context to the Gartner Hype Cycle, CEP is closer to the “Technology Trigger” phase than anywhere else in the hype cycle.  CEP has not yet reached the “Peak of Inflated Expectations”, but is inching closer and closer.

In addition, as a correlating reference point, if you look at a recent Gartner Hype Cycle that covers EDA, for example, you will find that EDA  (Event Driven Architucture) is at a similar phase, the “Technology Trigger” phase. 

Nowhere is that shift clearer than at the FBI's Regional Forensic Computer Lab here, which once lifted traces of incriminating Google searches from a suspect's hard drive to help convict him of murder. This week the lab became the sixth computer forensic lab in the nation to be accredited by the American Society of Crime Laboratory Directors, in another sign that computer forensics is no longer just about investigating hacker attacks.

"We've found video of gangsters rapping a song about a murder they committed," RCFL examiner John Leamons says.

The growth of law enforcement computer labs is an indication of how technology is increasingly involved in, or on the periphery of, criminal activity. San Diego-area law enforcement agencies founded the first regional forensic lab in 1998; there are now 14 such labs in the United States, with two more coming online this year. Last year the labs collectively performed more than 13,000 forensics examinations. The San Diego lab alone handled more than 1,000 requests from 40 law enforcement agencies in 2007, including 171 child pornography cases and 160 murder investigations.

In its early days, the RFCL examiners not only recovered the data, they analyzed it for evidentiary value based on the particulars of the case. But with exponentially growing data and caseloads, the 22 examiners here now focus on collecting and preserving data in a manner that will hold up in court, then hand that data back to the police agency for analysis.

Not surprisingly, the most valuable information comes from the files that suspects thought they had deleted, but which remained hidden in the nooks and crannies of their hard drives. "The key to computer forensics is unallocated space," says Leamons, who is on loan to the lab from the San Diego Police Department.

No one can remember a case being kicked because the lab made an error, but they can remember cases where they found evidence that exonerated people charged with crimes, Leamons says.

Cellphones pose a particular challenge, says Rebecca Adimari, one of the five examiners who work on them.

"Each has its own operating system and frequency -- there's probably over 500 makes and models and not many of them are the same," she explains. "There can be so much evidence on there."

From the unique ringtone caught on camera during a holdup -- to the accidentally recorded conversations on voice notes, to the Israeli thug keeping notes of extortion visits on his PDA -- the way people use their phones can be pretty incriminating.

"When they arrested the Arellano Felix people (a gang of Mexican drug lords later convicted of murder and drug crimes in 2007), they recovered 14 phones including one with a photo of a machine gun," Adimari says.

She has hundreds of power and data cables, since they're all peculiar to individual phones. And she has a special box that blocks signals on the phones in the lab, so no information is lost or compromised.

Examiner Patrick Lim, from the Naval Criminal Investigative Services, says he recently recovered data from a hard drive that had been burnt to a crisp. Asked if it was from an arson or a murder, Lim says he can't reveal the details.

"It was burned. That's all I can say."

When I talked to Bruzzo, he was clearly focused on how to improve the culture of the stores, with technology being one tool. He talked about connectivity being "a core part of the Starbucks experience" (that's Experience with a [tm]), and that he wanted Starbucks customers to be able to "tell stories" about coffee, music, and other things. That implies a kind of online medium for discussion and interaction that doesn't yet exist, but that is more likely to happen with Bruzzo's expanded role.

Bruzzo had already tipped me to the fact that Starbucks has caching media servers in its stores; that's how the Starbucks iTunes Wi-Fi Music Store combination technology works with iTunes, the iPhone, and the iPod touch in the several markets in which that's offered. (Those plans never advanced much after the initial launch, by the way: Seattle, Chicago, and the San Francisco Bay Area got service, but Chicago and Los Angeles are still listed as "coming soon," and other metropolitan areas are now "by the end of 2008," which would tie in neatly with Starbucks' other plans.)

With caching servers, content is pushed to the edge. Retrieving a 2 GB movie from iTunes thus becomes a matter of a few minutes to a laptop (or even faster if 802.11n networks are being deployed by AT&T), rather than 30 to 120 minutes over a typical home broadband connection. Stop in to Starbucks and fill up--with media. Neat, huh?

Back in February, Bruzzo described how the company has a unique relationship with its customers, who are already bringing their digital lifestyle into the stores, allowing hyper-local conversations to take place. "Starbucks is uniquely positioned to provide that kind of very local opportunity. It's what we do. The beginning of that is what we do today when we curate music, and books." The new AT&T relationship, he said, "gives us a landscape to continue to experiemnt with those kinds of things even at a local level."

As for the kinds of devices used, "We shouldn't be limited in our thoughts about connected devices to just communications devices; they should be PSPs [PlayStation Portables] and cameras." I expect that we will see a lot of change, much of workshopped in Seattle-area stores, in the digital side of Starbucks this year.

I will also repeat my expectation that the launch of a 3G iPhone will involve a Starbucks tie-in, and that the date for the first Starbucks AT&T markets to go live with AT&T in charge will coincide with the release of the 3G iPhone. The timing is too close to be coincidental. (Rumors today are that the 3G iPhone will be announced at the June 9 developers conference that Apple runs. I'll be at that event's keynote.)

Bruzzo has been with the company for not much over a year, coming off a few years as head of communications (talking, not technology) at Amazon. In January 2008, he was boosted to chief technology and chief information officer, as well as being appointed a vice president. That's a pretty fast rise; he must have, you know, a few good ideas. He's behind My Starbucks Idea, the site the company is using to let its customers give it free, valuable advice. One of the fascinating, Cluetrained elements of that site is the transparency: ideas that are submitted can be viewed by other visitors to the site, and voted upon. Suggestion boxes are usually locked tight, whether in the real world or on the Net. Some posts have thousands of votes and hundreds of comments.

Today's announcement also included a note that Starbucks is selling its Hear Music division to its partner in the venture, Concord Music Group. Hear signed Paul McCartney among other musicians; Starbucks will keep working with Concord, so this might not be quite as big a change in direction as a change in its internal focus. This is yet another move of many by company head Howard Schultz, who took charge of the firm again, and started getting rid of top executives, reorganizing divisions, and making announcements about massive changes in the stores, notably replacing its barista-hiding super-automated coffeemakers with shorter, more controllable systems, and tearing out the stinking breakfast sandwich ovens.

In the Microsoft Auto group we spent a lot of time trying to figure out what the SDL meant to our product. We knew we needed to do threat modeling, primarily because threat modeling is probably the most commonly known requirement of the SDL. Beyond threat modeling though, members of the various disciplines in our product team didn’t know what parts of the SDL applied to our product and what parts applied to technologies, platforms, or programming languages we didn’t use and thus could safely ignore. One of our program managers set out to sift through the SDL requirements and associated tools to try and determine what was applicable to our environment. While we eventually made the right decisions on what SDL requirements we needed to focus upon, we spent more time than we would have liked trying to figure it all out.

With our most recent update to the SDL at Microsoft we’ve made one significant change to try and help in this scenario. That change is to take all of the SDL requirements and plug them into a filterable framework that allows a person or a team to match requirements with specific technologies. Now, instead of being presented with a large document that covers all SDL requirements, a team is presented with a dynamic Web site that allows them to selectively filter requirements based on their product type (Client, Server, Hardware, Online Service etc), code type (Native, Managed, JavaScript etc), platform type (Win32, Win64, WinCE, Mac etc), or applies to their specific role (Program Manager, Developer, Test Engineer, Operations, etc).

This means if I’m a program manager for a Win64 Client product, I can view just the SDL requirements that apply to that criteria and the result is a clearer starting point for what you need to do to begin adopting the SDL for your project. This applicability filtering also allows product groups to more easily divide up the responsibility for ramping up on the SDL instead of overloading a single person in their group with figuring out what needs to be done. For instance, a product group could assign a person from each discipline in their team to identify which SDL requirements need to be met and at what point in the product cycle. A program manager can now more easily identify the SDL requirements that need to be thought about and met during the Requirements phase of a product, and likewise a test engineer can identify and begin working on the test collateral for SDL requirements that will be needed later in the schedule during the verification phase.

As the SDL continues to grow to address evolving security concerns and new technologies, it’s necessary for the SDL to be able to scale and have this type of filtering in place. Enhancing the functionality and depth of our tools that we use in the SDL is an ongoing process. These tools don’t always apply to every code type or product type. We have test tools that only run on native code while other tools run only against managed code, and that’s just one example. It’s important that we leverage a filterable framework like we have to address these differences and help teams understand where they need to focus their resources and what just doesn’t apply to their product or technology.

This might not be everybody's idea of utopia -- and it certainly doesn't address the inherent value of privacy -- but this theory has a glossy appeal, and could easily be mistaken for a way out of the problem of technology's continuing erosion of privacy. Except it doesn't work, because it ignores the crucial dissimilarity of power.

You cannot evaluate the value of privacy and disclosure unless you account for the relative power levels of the discloser and the disclosee.

If I disclose information to you, your power with respect to me increases. One way to address this power imbalance is for you to similarly disclose information to me. We both have less privacy, but the balance of power is maintained. But this mechanism fails utterly if you and I have different power levels to begin with.

An example will make this clearer. You're stopped by a police officer, who demands to see identification. Divulging your identity will give the officer enormous power over you: He or she can search police databases using the information on your ID; he or she can create a police record attached to your name; he or she can put you on this or that secret terrorist watch list. Asking to see the officer's ID in return gives you no comparable power over him or her. The power imbalance is too great, and mutual disclosure does not make it OK.

You can think of your existing power as the exponent in an equation that determines the value, to you, of more information. The more power you have, the more additional power you derive from the new data.

Another example: When your doctor says "take off your clothes," it makes no sense for you to say, "You first, doc." The two of you are not engaging in an interaction of equals.

This is the principle that should guide decision-makers when they consider installing surveillance cameras or launching data-mining programs. It's not enough to open the efforts to public scrutiny. All aspects of government work best when the relative power between the governors and the governed remains as small as possible -- when liberty is high and control is low. Forced openness in government reduces the relative power differential between the two, and is generally good. Forced openness in laypeople increases the relative power, and is generally bad.

Seventeen-year-old Erik Crespo was arrested in 2005 in connection with a shooting in a New York City elevator. There's no question that he committed the shooting; it was captured on surveillance-camera videotape. But he claimed that while being interrogated, Detective Christopher Perino tried to talk him out of getting a lawyer, and told him that he had to sign a confession before he could see a judge.

Perino denied, under oath, that he ever questioned Crespo. But Crespo had received an MP3 player as a Christmas gift, and surreptitiously recorded the questioning. The defense brought a transcript and CD into evidence. Shortly thereafter, the prosecution offered Crespo a better deal than originally proffered (seven years rather than 15). Crespo took the deal, and Perino was separately indicted on charges of perjury.

Without that recording, it was the detective's word against Crespo's. And who would believe a murder suspect over a New York City detective? That power imbalance was reduced only because Crespo was smart enough to press the "record" button on his MP3 player. Why aren't all interrogations recorded? Why don't defendants have the right to those recordings, just as they have the right to an attorney? Police routinely record traffic stops from their squad cars for their own protection; that video record shouldn't stop once the suspect is no longer a threat.

Cameras make sense when trained on police, and in offices where lawmakers meet with lobbyists, and wherever government officials wield power over the people. Open-government laws, giving the public access to government records and meetings of governmental bodies, also make sense. These all foster liberty.

Ubiquitous surveillance programs that affect everyone without probable cause or warrant, like the National Security Agency's warrantless eavesdropping programs or various proposals to monitor everything on the internet, foster control. And no one is safer in a political system of control.

This essay originally appeared on Wired.com.

I think we will finally get a true cross site XSS work in 2008. Combining XSRF and XSS to propagate a worm across multiple sites and multiple domains. The first one will be benign but the others will be much more malicious in nature. Leading victim candidate are social network sites that are becoming increasingly open.

2. More consolidation in the security industry.

There is still a great difference between the small security players and the giant ones in terms of cash flow. As the old guard (McaFee, Symantec, etc) see dwindling revenue on various fronts they will begin to convert some of that pesky cash into acquisitions. Could this be the year Qualys gets gobbled up?

3. PCI will clarify section 6.6

This is more of a hope really. Since it goes into full effect mid-2008 I hope to see some clearer definitions around what companies are expected to do.

4. 2008 will set another record for breaches

Yeah big shocker! The trend will continue with more smaller breaches this year as opposed to a few massive ones.

5. RBN will disappear again. Someone related to them will get busted.

With the light too bright they will morph again and change tactics. Money will still flow in to them by the millions though. However with increasing public knowledge of the group someone will get busted and connected to them. No one high up in the group, but some poor sucker at the wrong place at the wrong time. Law Enforcement will trump it as a “significant” blow to the group. RBN won’t notice.

Post from: Grumpy Security Guy

5 Security Predictions for 2008