In this interview, cloudsecurity.org talks to Guido van Rossum about Python, Google App Engine and security.

Guido is the creator of the Python programming language and more recently, Google App Engine team member.  His involvement with the App Engine project was pretty late - the code “was almost ready for release” when he get involved.  The security architect of App Engine was primarily project lead, Kevin Gibbs, supported by the rest of the App Engine crew and the Google Security Team.

The Interview

cloudsecurity.org: What security principles did you follow for App Engine?

GvR: While I can’t share any specifics on what we’re doing to secure App Engine, I can say that the main principle we’ve followed could be called “defense in depth”. We’re not relying exclusively on a secure interpreter, or any other single security layer, to protect our users.

cloudsecurity.org: Please provide some examples of how those principles played out in terms of the current implementation?

GvR: Sorry, we don’t divulge such information.

cloudsecurity.org: What criteria did you apply to Python module selection?

GvR: We first looked for modules that were useful and straightforward to audit. If a module was large or complex, we’d only audit it (fixing things we found) if it was deemed essential or at least useful for a large number of users; otherwise we’d exclude it.

cloudsecurity.org: What do you see as the security risks inherent in exposing an interpreter runtime in a shared environment?

GvR: I presume you’re asking about risks to users, like providing accidental access to data belonging to another app. We’ve taken extensive measures to isolate different apps from each other. For example, each app runs in a separate process, and the datastore prevents an app from accessing data belonging to other apps.

cloudsecurity.org: I recently attended a fascinating talk by Justin Ferguson (a Seattle based security consultant) at eusecwest in London.  He gave a great talk exploring security vulnerabilities in language interpreters and specifically highlighted some security weaknesses in Python App Engine.  What are your thoughts on his research and specifically the Python issues he highlighted?  When do you anticipate they will get fixed?

GvR: We’ve anticipated all of the possibilities raised in Justin’s talk, and took measures to protect our users. Justin highlighted weaknesses in Python, but not in App Engine. Furthermore, our security model does not rely solely upon protections within the Python interpreter; there are additional protections that these external analyses have missed.

cloudsecurity.org: How do you contain an attacker that exploits bugs in App Engine from exploiting the underlying OS and potentially interfering with other users processes or attacking backend systems?

GvR: You are correct that there are strong measures in place, but I’m not at liberty to discuss details.

cloudsecurity.org: Python was the first language to get the App Engine treatment, what language is next and what are some of the language specific security challenges the team has had to deal with?

GvR: Although I can’t comment on what language is next, we are working on this, and have gotten a lot of great feedback from our developers. As far as language-specific security challenges, they stemmed mostly from the complexity of the Python interpreter. We spent a lot of time auditing this, and did a great deal more than just identifying buffer overflows.  I can also add that Google is actively researching the security of interpreted languages.  Google engineers routinely contribute security fixes to open source projects, including but not limited to Python.

cloudsecurity.org: How does the team decide when ‘enough is enough’ in terms of hardening the interpreter?

GvR: That’s not really how we approach it. We realize that security is an ongoing effort, and try to stay ahead of threats through continuous monitoring and testing.

cloudsecurity.org: Some commentators have suggested that perhaps the difficulty of auditing the implementation led to some modules being more heavily restricted than perhaps necessary.  What are your thoughts on that and what plans, if any, are there to bring back code objects/functions that were eliminated in the initial release?  (with the benefit of hindsight).

GvR: The only thing we are likely to put back is the _ast module, which was not audited based upon an underestimation of its usefulness (see my answer to question #3 above).  We will also put back some dummy functions and other objects whose absence currently prevents some popular frameworks from being loaded without modifications. For example, some harmless functionality in the imp module will come back. We’re also looking into making urllib2 work (to some extent), though that’s not really a security issue but merely a matter of API adjustment.

cloudsecurity.org: It is reported that Google encourages small groups to go off and create.  How involved were the Google security team with App Engine in terms of design and implementation review/testing?  Given the dynamics, is it possible to have a meaningful security process that shadows the development process?

GvR: The Google Security team is involved in everything we do. They have been extremely helpful.

cloudsecurity.org: How can people report security weaknesses they discover in App Engine?  What commitment does Google give in terms of dealing vulnerability reports?

GvR: There is a standard process for submitting security issues. See http://www.google.com/corporate/security.html. Google moves very fast to protect its users when a verifiable security vulnerability is reported.

cloudsecurity.org: One concern is the potential misuse of App Engine to exploit security vulnerabilities in visitors browsers.  This is not a new problem per se, shared hosting providers know all about this.  But with Google and other Cloud providers, the scalability potential is much higher.  What are your thoughts on this and what pro-active steps is Google taking to detect and terminate evil apps?

GvR: This is high on our list of concerns. We deal with this through a combination of restrictions on what you can do (e.g. certain HTTP headers and ports are off-limits) and, again, monitoring.

cloudsecurity.org: Beyond App Engine, what role do you think Python will play in the Cloud both now and in the future?

GvR: Sorry, I’m not prone to philosophizing about the future.

cloudsecurity.org: Trust is often cited as a barrier to enterprise adoption of Cloud Computing.  What role do you personally think Google can play in building that trust?

GvR: I think trust is built up over a long period of experience. Our actions in terms of being open to our users will be the most important factor in establishing trust. Of course, Google’s reputation also helps: everybody understands that Google doesn’t want its name associated with a bad product.

cloudsecurity.org: Looking at the Cloud Computing landscape beyond Google, what are your thoughts on the current state of Cloud Computing and Security?

GvR: It’s obvious that Cloud Computing is only just taking off. The next few years will be very exciting.

cloudsecurity.org: Lastly, what are some of your favourite App Engine apps?

GvR: There are too many to enumerate. If you insist on a highlight, well, I like Rietveld (http://codereview.appspot.com), a tool for collaborative code review which I (largely) wrote myself. It is open source and includes some essential components from Mondrian, a similar internal tool which I created before I joined the App Engine team.

Thanks

My thanks to Guido for his time and sharing his views.

From The Register:

An Aussie risk advisory services manager has issued a chilling security alert concerning the Jura F90 net-connected coffee machine, warning caffeine-heads that the hi-tech brewing device could open their Windows PC to exploitation by internet paedophiles and al-Qaeda*, CNET reports.

Craig Wright, who works for “professional services” outfit BDO, posted his alert yesterday on the BugTraq security email list, highlighting possible lines of attack, including buffer overflow menaces. The F90 (see pic) is apparently the “world’s first household espresso machine with internet capability”, which allows users to select “coffee specialities” via their PCs without having to walk the five feet between their desks and the machine.

Someone took the Trojan Room Coffee Machine idea and made it open to TERRORISM. Run screaming if you feel so compelled.

LOL! Yes there is a fair degree of tongue in cheek in this story. Thanks to Lester Haines for making my day. Be sure to read the full piece on the Reg.

Article Link

This type of thing is only going to increase as people slap more stuff onto the network with little to no care about security. These things generally all have web UIs which makes the vulns that much more interesting. It is somewhat easy to detect the spread of a mass SQLi attack on public facing web sites but what happens when we get this attack on internally facing systems? They are much harder to track and even detect. What if my coffee maker now does drive by malware attacks? What if my wireless router does? Our jobs are only geting harder people.

Link

Post from: Grumpy Security Guy

Dude Don’t Hack My Coffee

In a previous post I explained how to use HeapSetInformation correctly. In short there's an option when calling this function that will terminate your application if the heap manager detects some form of heap corruption, or the potential to cause heap corruption.

I would recommend you read the previous post before continuing.

You guessed it, the number one email I got after this post was, "So, what sort of corruption will terminate my app?"

So for all those who emailed me, here's a list:

• Corruption of an uncommitted range (region inside heap segments which are reserved but not committed)
• Heap header corruption, for example the heap header checksum is invalid. This can be a single header, or multiple headers.
• Walk of the large virtual blocks shows corruption (all blocks above about 512Kb on x86 and 1Mb on 64 bit are not allocated from segments; they are direct virtual allocations, the heap just holds a list of them along with some metadata to assure consistency with the rest of the heap. They are chained in a double linked list so corruption can be detected by walking the list.)
• Buffer overrun: the next block header size does not match the expected current block size.
• Buffer underrun: same as above, but the previous block header size does not match the expected current block size.
• Attempting to free a free'd block (double-free bug)
• Attempting to free a non 8-byte aligned block.
• Passing a bogus heap handle, it could simply be an invalid heap or a handle to a different heap.
• Corruption of free block list. A bit of a catch-all, including: writing after free, overrunning a previous and managing to step over the list entry.

But there is one huge and critically important caveat to using the defense: it only works if you use the Windows heap manager. You might be surprised to learn that many applications actually implement their own heap functionality for various reasons, often legacy reasons based on historically poor performance of operating system heap managers. A great deal of performance work was performed on the Windows Vista and Windows Server 2008 heap managers, but the work performed is way beyond the scope of this document. Another common scenario is to allocate a huge block of memory from the operating system and then perform custom allocation within that heap block. Again, if you do this, you will not get benefit from using the heap corruption termination capability and you will still be subject to repeatable heap based attacks.

Another down side of not using the native Windows heap manager (or if you use your own sub-allocation mechanism) is you cannot take advantage of Windows leak-detection tools because you are not using the Windows heap in the way it's meant to be used, or you're not using the Windows heap at all.

With all this said, I realize that moving off a custom heap to another heap is never an easy task, but if you want to take advantage of this defense, you should add "Move off our custom heap" to the list of development tasks.

You may have read recently about a large number of Web servers that were compromised through a SQL injection attack. The malicious SQL payload is very well designed, somewhat database schema agnostic and generic  so it could compromise as many database servers as possible. While the attack was a SQL injection attack that attacked and compromised back-end databases courtesy of vulnerable Web pages, from a user's perspective the real attack was compromised Web pages that serve up malware to attack user's through their browsers. In essence, there were two sets of victims: the Web site operators and the users who visited the affected Web sites. In this post, I want to focus on what the first set of users, the Web site operators, can do to protect themselves.

The fact that the malicious payload was so generic shows that the science of SQL injection has not taken a back seat to research in other vulnerability types, such as buffer overflows or cross-site scripting issues.

I think the first lesson from this attack is this:

If you have a Web server (doesn't matter what type), and it's hooked up to a database (doesn't matter what type) you need to go in and review your code that performs the database work.

So now that you've determined the database access code, now what? The SDL is very specific about what do here, there are three requirements - they are requirements not recommendations, which means you must do the following coding requirements and defenses

• Use SQL Parameterized Queries
• Use Stored Procedures
• Use SQL Execute-only Permission

Use SQL Parameterized Queries

From the SDL documentation:

"Applications accessing a database must do so only using parameterized queries.

Creating dynamic queries using string concatenation potentially allows an attacker to execute an arbitrary query through the application. This vulnerability allows for unauthorized, interactive, logon to a SQL server which may result in the execution of malicious commands leading to the possible modification (or deletion) of Operating System or user data.

Combining the use of parameterized queries and stored procedures helps to mitigate the risk of successful exploitation of user input which is not correctly verified."

This defense has been known about forever; heck, David and I discussed this in detail in the first edition of Writing Secure Code in 2002:

From page 320, "Another way to perform this kind of processing is to use placeholders which are often referred to as parameterized commands."

Just about every database access technology supports parameterized queries; work out what they are for your DB technology and use them: the defense for a PHP/MySQL combo will not be the same as a C#/SQL Server combo.

The most likely cause of these recent compromises is using string concatenation to build SQL statements. Just don't do it, even if you think you're safe, just don't use string concatenation to build SQL statements! There are some very specialized cases where string concatenation is valid, but they are rare, especially for Web apps. In my opinion, any use of string concatenation in a Web application is a high-priority bug.

Use Stored Procedures

From the SDL documentation:

"Applications accessing databases should do so only using stored procedures. "

-and-

"Do not use "exec @sql" construct in your stored procedures.

Using stored procedures helps to mitigate the SQL injection threat to a great extent since type checking is available for parameters. If the attacker supplies input that does not match the type constraints the stored procedures will throw an exception. In the vast majority of the cases, this should be properly handled within the application.

However, if the stored procedures perform string manipulation in their code and then execute that query using the "exec @sql" construct incorrect handling of user input can produce the same SQL injection vulnerability as would be seen at the application layer."

Note the words "help mitigate," by themselves stored procedures do not remove SQL injection vulnerabilities; they just raise the bar on the attacker by hiding much of the underlying database schema from the attacker.

Use SQL Execute-only Permission

This next defense is interesting in that it is a defense in depth method; in this case it assumes the attacker has successfully found a SQL injection bug in your code. Now what? Thankfully, this defense will stop most every attack dead in its tracks.

From the SDL documentation:

 "Only grant ‘execute' permission on all stored procedures, and grant that permission only for the application domain group.

Ensure that this group is granted execute permissions only on your stored procedures. Do not grant any other permission on your database to any other user or group."

This is a great defense, because if the attacker attempts to access any other database object other than through a stored procedure (you can use views also), the underlying database permissions model prevents the attack by denying access to the attacker.

It's interesting that the SDL offers three SQL injection requirements; only one actually remedies the problem (secure by design) and the other two offer mores defenses assuming failure (secure by default.)

Of course, a simple set of rules is not a substitute for careful design, implementation, and test. The SDL is a holistic process that covers the software lifecycle end-to-end, so don't mistake these simple rules as a guarantee that you will avoid SQL injection problems. You need to understand the situations in which the rules apply. You may find, for example, that string concatenation is the best - or perhaps only - solution to a particular problem and these rules may not guard against SQL injection in those situations. Follow secure development practice throughout the lifecycle of your project - including things we left out of this blog, like testing and security response, for best results.

Vulnerabilities are software mistakes--mistakes in specification and design, but mostly mistakes in programming. Any large software package will have thousands of mistakes. These vulnerabilities lie dormant in our software systems, waiting to be discovered. Once discovered, they can be used to attack systems. This is the point of security patching: eliminating known vulnerabilities. But many systems don't get patched, so the Internet is filled with known, exploitable vulnerabilities.

New vulnerabilities are hot commodities. A hacker who discovers one can sell it on the black market, blackmail the vendor with disclosure, or simply publish it without regard to the consequences. Even if he does none of these, the mere fact the vulnerability is known by someone increases the risk to every user of that software. Given that, is it ethical to research new vulnerabilities?

Unequivocally, yes. Despite the risks, vulnerability research is enormously valuable. Security is a mindset, and looking for vulnerabilities nurtures that mindset. Deny practitioners this vital learning tool, and security suffers accordingly.

Security engineers see the world differently than other engineers. Instead of focusing on how systems work, they focus on how systems fail, how they can be made to fail, and how to prevent--or protect against--those failures. Most software vulnerabilities don't ever appear in normal operations, only when an attacker deliberately exploits them. So security engineers need to think like attackers.

People without the mindset sometimes think they can design security products, but they can't. And you see the results all over society--in snake-oil cryptography, software, Internet protocols, voting machines, and fare card and other payment systems. Many of these systems had someone in charge of "security" on their teams, but it wasn't someone who thought like an attacker.

This mindset is difficult to teach, and may be something you're born with or not. But in order to train people possessing the mindset, they need to search for and find security vulnerabilities--again and again and again. And this is true regardless of the domain. Good cryptographers discover vulnerabilities in others' algorithms and protocols. Good software security experts find vulnerabilities in others' code. Good airport security designers figure out new ways to subvert airport security. And so on.

This is so important that when someone shows me a security design by someone I don't know, my first question is, "What has the designer broken?" Anyone can design a security system that he cannot break. So when someone announces, "Here's my security system, and I can't break it," your first reaction should be, "Who are you?" If he's someone who has broken dozens of similar systems, his system is worth looking at. If he's never broken anything, the chance is zero that it will be any good.

Vulnerability research is vital because it trains our next generation of computer security experts. Yes, newly discovered vulnerabilities in software and airports put us at risk, but they also give us more realistic information about how good the security actually is. And yes, there are more and less responsible--and more and less legal--ways to handle a new vulnerability. But the bad guys are constantly searching for new vulnerabilities, and if we have any hope of securing our systems, we need the good guys to be at least as competent. To me, the question isn't whether it's ethical to do vulnerability research. If someone has the skill to analyze and provide better insights into the problem, the question is whether it is ethical for him not to do vulnerability research.

This was originally published in InfoSecurity Magazine, as part of a point-counterpoint with Marcus Ranum. You can read Marcus's half here.