In this interview, cloudsecurity.org talks to Guido van Rossum about Python, Google App Engine and security.

Guido is the creator of the Python programming language and more recently, Google App Engine team member.  His involvement with the App Engine project was pretty late - the code “was almost ready for release” when he get involved.  The security architect of App Engine was primarily project lead, Kevin Gibbs, supported by the rest of the App Engine crew and the Google Security Team.

The Interview

cloudsecurity.org: What security principles did you follow for App Engine?

GvR: While I can’t share any specifics on what we’re doing to secure App Engine, I can say that the main principle we’ve followed could be called “defense in depth”. We’re not relying exclusively on a secure interpreter, or any other single security layer, to protect our users.

cloudsecurity.org: Please provide some examples of how those principles played out in terms of the current implementation?

GvR: Sorry, we don’t divulge such information.

cloudsecurity.org: What criteria did you apply to Python module selection?

GvR: We first looked for modules that were useful and straightforward to audit. If a module was large or complex, we’d only audit it (fixing things we found) if it was deemed essential or at least useful for a large number of users; otherwise we’d exclude it.

cloudsecurity.org: What do you see as the security risks inherent in exposing an interpreter runtime in a shared environment?

GvR: I presume you’re asking about risks to users, like providing accidental access to data belonging to another app. We’ve taken extensive measures to isolate different apps from each other. For example, each app runs in a separate process, and the datastore prevents an app from accessing data belonging to other apps.

cloudsecurity.org: I recently attended a fascinating talk by Justin Ferguson (a Seattle based security consultant) at eusecwest in London.  He gave a great talk exploring security vulnerabilities in language interpreters and specifically highlighted some security weaknesses in Python App Engine.  What are your thoughts on his research and specifically the Python issues he highlighted?  When do you anticipate they will get fixed?

GvR: We’ve anticipated all of the possibilities raised in Justin’s talk, and took measures to protect our users. Justin highlighted weaknesses in Python, but not in App Engine. Furthermore, our security model does not rely solely upon protections within the Python interpreter; there are additional protections that these external analyses have missed.

cloudsecurity.org: How do you contain an attacker that exploits bugs in App Engine from exploiting the underlying OS and potentially interfering with other users processes or attacking backend systems?

GvR: You are correct that there are strong measures in place, but I’m not at liberty to discuss details.

cloudsecurity.org: Python was the first language to get the App Engine treatment, what language is next and what are some of the language specific security challenges the team has had to deal with?

GvR: Although I can’t comment on what language is next, we are working on this, and have gotten a lot of great feedback from our developers. As far as language-specific security challenges, they stemmed mostly from the complexity of the Python interpreter. We spent a lot of time auditing this, and did a great deal more than just identifying buffer overflows.  I can also add that Google is actively researching the security of interpreted languages.  Google engineers routinely contribute security fixes to open source projects, including but not limited to Python.

cloudsecurity.org: How does the team decide when ‘enough is enough’ in terms of hardening the interpreter?

GvR: That’s not really how we approach it. We realize that security is an ongoing effort, and try to stay ahead of threats through continuous monitoring and testing.

cloudsecurity.org: Some commentators have suggested that perhaps the difficulty of auditing the implementation led to some modules being more heavily restricted than perhaps necessary.  What are your thoughts on that and what plans, if any, are there to bring back code objects/functions that were eliminated in the initial release?  (with the benefit of hindsight).

GvR: The only thing we are likely to put back is the _ast module, which was not audited based upon an underestimation of its usefulness (see my answer to question #3 above).  We will also put back some dummy functions and other objects whose absence currently prevents some popular frameworks from being loaded without modifications. For example, some harmless functionality in the imp module will come back. We’re also looking into making urllib2 work (to some extent), though that’s not really a security issue but merely a matter of API adjustment.

cloudsecurity.org: It is reported that Google encourages small groups to go off and create.  How involved were the Google security team with App Engine in terms of design and implementation review/testing?  Given the dynamics, is it possible to have a meaningful security process that shadows the development process?

GvR: The Google Security team is involved in everything we do. They have been extremely helpful.

cloudsecurity.org: How can people report security weaknesses they discover in App Engine?  What commitment does Google give in terms of dealing vulnerability reports?

GvR: There is a standard process for submitting security issues. See http://www.google.com/corporate/security.html. Google moves very fast to protect its users when a verifiable security vulnerability is reported.

cloudsecurity.org: One concern is the potential misuse of App Engine to exploit security vulnerabilities in visitors browsers.  This is not a new problem per se, shared hosting providers know all about this.  But with Google and other Cloud providers, the scalability potential is much higher.  What are your thoughts on this and what pro-active steps is Google taking to detect and terminate evil apps?

GvR: This is high on our list of concerns. We deal with this through a combination of restrictions on what you can do (e.g. certain HTTP headers and ports are off-limits) and, again, monitoring.

cloudsecurity.org: Beyond App Engine, what role do you think Python will play in the Cloud both now and in the future?

GvR: Sorry, I’m not prone to philosophizing about the future.

cloudsecurity.org: Trust is often cited as a barrier to enterprise adoption of Cloud Computing.  What role do you personally think Google can play in building that trust?

GvR: I think trust is built up over a long period of experience. Our actions in terms of being open to our users will be the most important factor in establishing trust. Of course, Google’s reputation also helps: everybody understands that Google doesn’t want its name associated with a bad product.

cloudsecurity.org: Looking at the Cloud Computing landscape beyond Google, what are your thoughts on the current state of Cloud Computing and Security?

GvR: It’s obvious that Cloud Computing is only just taking off. The next few years will be very exciting.

cloudsecurity.org: Lastly, what are some of your favourite App Engine apps?

GvR: There are too many to enumerate. If you insist on a highlight, well, I like Rietveld (http://codereview.appspot.com), a tool for collaborative code review which I (largely) wrote myself. It is open source and includes some essential components from Mondrian, a similar internal tool which I created before I joined the App Engine team.

Thanks

My thanks to Guido for his time and sharing his views.

Bruce Sterling (11/20/2001):

You know what I want? I don't want a National ID Card. I want a Global Coalition Visa.

Like it or not, we've got a huge global diaspora now. It is a fact of life. Nations with stupid and corrupt politics have seen their clever people brain- drained away, to places where the cops don't shake you down twice a day. And jet-setters go everywhere. And properly so. If you're in a true global society, then you spend a lot of your time among aliens. Quite often you are the alien. You might notice that even Al Qaeda is a genuinely multinational group. They gravitated to wicked, lawless places like Sudan, Chechnya and Afghanistan, where the locals shoot you if you ask for a badge.

But what about all us bright, shiny, world-trading jet setters, huh? There are thirty percent fewer Yankees in Europe this Christmas, and that is bad. Let me pose the problem this way. If I am going into a Japanese restaurant in Japan, I would rather like to be able to haul out some gizmo and flash it at my fellow civilians, and have these kindly people understand with a high degree of likelihood that I am not a mass murderer. On the contrary, I am quite civilized, and I should be brought a beer immediately.

A platinum VISA card and a five-hundred-dollar suit will almost do that, but those are too easy to forge and steal, plus they are not very democratic. The UN should get together on this. We should have a high level summit about digital hardware support for the crippled tourist economy. Fear and ill treatment shut down tourism faster than anything short of open warfare. That is bad for all of us. Killing off tourism harms our civilization and impoverishes our cultures. People in civilized states shouldn't routinely treat one another as criminal suspects. I don't want to get done-over for three hours every time I get off a plane in London. When I go to London, I go with empty suitcases. I don't plan to stay, but I am better news for the London economy than a lot of the people who live there.

They should know all that that before I get off the plane. My arrival is excellent news for Britain, so I should be treated that way. If this is a new kind of war, I don't want to be the evil guy hunkered down in the bunker; I want to fly with the boys from Air Assault. I want one of those handy crypto-style Friend-or-Foe IDs.

These people who normally meet me whenever I am an alien, they don't need to know my nationality, my home address or my shoe size. They just need to know that, despite being alien, I'm sort-of okay.

I want a democratic, citizen-to-citizen device that will bridge those social barriers and language barriers. I think we could invent devices and means of verification that would strengthen the global social fabric that terrorism wants to rip. It wouldn't be easy or simple, but it's not beyond our ingenuity. Our social capital sustains all civilized societies, and it is all about trust. So let's invent new methods of trust.

I added bold to the last sentence because I think this is the mission statement for building out federation systems.

Consider a virus that does the following: Once installed, the code would periodically download pornography onto the computer, encrypt it, and then store it on the disk. Not too much, and not too often, just a few pictures or small videos. After several weeks of doing this, it would un-encrypt the data, move it to “My Documents” or some subdirectory, and then uninstall itself. It could be programmed to remove signs that it was present, such as scrubbing log files to further hide from detection.

The computer could be infected randomly through a hostile web site or it could be targeted through an injection attack via some insecure service. It could even be targeted by a co-worker who installed this on your machine when you were at lunch, or loaned you an infected memory stick. A virus of this type could be subtle, and use so minimal CPU, network, and disk resources so as to go unnoticed both by the owner of the computer and the IT department.

Now what you have is presumed guilt. If the downloads are discovered by IT, or someone like the malicious co-worker were to proactively mention to HR “I saw something that looked like …” on or after the date the virus uninstalled itself, a subsequent search would reveal pornography on the machine. Odds are the employee would be fired. It would be tough to convince anyone that it was anything other than the employee doing what they should not have been doing, and “innocent until proven guilty” is a legal doctrine that is not applied to corporate hiring/firing decisions.

I was discussing this scenario with our former Director of Marketing at IPLocks, Tom Yates, and he raised a good point. We routinely use Occam’s Razor in our reasoning. This principle states that the simplest explanation is usually the correct one. And the simple explanation would be that you were performing unauthorized browsing with your computer, which could have negative legal consequences for the company, and is almost always a ‘fire-able’ offense. How could you prove otherwise? Who is going to bring in a forensic specialist to prove you are innocent? How could you account for the files?

I have had a home computer infected with a BitTorrent-like virus storing such files on a home computer in 2003, so I know the virus part is quite feasible. I know that remote sessions can be used to instigate activity from a specific machine as well. It is a problem to assume the person and the computer are one and the same. We often assume that you are responsible for specific activity because it was your IP address, or your MAC address, or your account, or your computer that was involved. Your computer is not always under your control, passwords are usually easy to guess, and so it is a dangerous assumption that the official user is responsible for all activity on a computer. Almost every piece of software I have ever downloaded onto my machine takes some action without my consent. So how would you prove it was some guy looking at porn and not spammers, hackers and/or the malicious co-worker?

Consider a virus that does the following: Once installed, the code would periodically download pornography onto the computer, encrypt it, and then store it on the disk. Not too much, and not too often, just a few pictures or small videos. After several weeks of doing this, it would un-encrypt the data, move it to “My Documents” or some subdirectory, and then uninstall itself. It could be programmed to remove signs that it was present, such as scrubbing log files to further hide from detection.

The computer could be infected randomly through a hostile web site or it could be targeted through an injection attack via some insecure service. It could even be targeted by a co-worker who installed this on your machine when you were at lunch, or loaned you an infected memory stick. A virus of this type could be subtle, and use so minimal CPU, network, and disk resources so as to go unnoticed both by the owner of the computer and the IT department.

Now what you have is presumed guilt. If the downloads are discovered by IT, or someone like the malicious co-worker were to proactively mention to HR “I saw something that looked like …” on or after the date the virus uninstalled itself, a subsequent search would reveal pornography on the machine. Odds are the employee would be fired. It would be tough to convince anyone that it was anything other than the employee doing what they should not have been doing, and “innocent until proven guilty” is a legal doctrine that is not applied to corporate hiring/firing decisions.

I was discussing this scenario with our former Director of Marketing at IPLocks, Tom Yates, and he raised a good point. We routinely use Occam’s Razor in our reasoning. This principle states that the simplest explanation is usually the correct one. And the simple explanation would be that you were performing unauthorized browsing with your computer, which could have negative legal consequences for the company, and is almost always a ‘fire-able’ offense. How could you prove otherwise? Who is going to bring in a forensic specialist to prove you are innocent? How could you account for the files?

I have had a home computer infected with a BitTorrent-like virus storing such files on a home computer in 2003, so I know the virus part is quite feasible. I know that remote sessions can be used to instigate activity from a specific machine as well. It is a problem to assume the person and the computer are one and the same. We often assume that you are responsible for specific activity because it was your IP address, or your MAC address, or your account, or your computer that was involved. Your computer is not always under your control, passwords are usually easy to guess, and so it is a dangerous assumption that the official user is responsible for all activity on a computer. Almost every piece of software I have ever downloaded onto my machine takes some action without my consent. So how would you prove it was some guy looking at porn and not spammers, hackers and/or the malicious co-worker?

There are also a lot of errors and misconceptions. With its aggressive advertising campaign and a CEO who publishes his Social Security number and dares people to steal his identity -- Todd Davis, 457-55-5462 -- LifeLock is a company that's easy to hate. But the company's story has some interesting security lessons, and it's worth understanding in some detail.

In December 2003, as part of the Fair and Accurate Credit Transactions Act, or Facta, credit bureaus were forced to allow you to put a fraud alert on their credit reports, requiring lenders to verify your identity before issuing a credit card in your name. This alert is temporary, and expires after 90 days. Several companies have sprung up -- LifeLock, Debix, LoudSiren, TrustedID -- that automatically renew these alerts and effectively make them permanent.

This service pisses off the credit bureaus and their financial customers. The reason lenders don't routinely verify your identity before issuing you credit is that it takes time, costs money and is one more hurdle between you and another credit card. (Buy, buy, buy -- it's the American way.) So in the eyes of credit bureaus, LifeLock's customers are inferior goods; selling their data isn't as valuable. LifeLock also opts its customers out of pre-approved credit card offers, further making them less valuable in the eyes of credit bureaus.

And, so began a smear campaign on the part of the credit bureaus. You can read their points of view in this New York Times article, written by a reporter who didn't do much more than regurgitate their talking points. And the class action lawsuits have piled on, accusing LifeLock of deceptive business practices, fraudulent advertising and so on. The biggest smear is that LifeLock didn't even protect Todd Davis, and that his identity was allegedly stolen.

It wasn't. Someone in Texas used Davis's SSN to get a $500 advance against his paycheck. It worked because the loan operation didn't check with any of the credit bureaus before approving the loan -- perfectly reasonable for an amount this small. The payday-loan operation called Davis to collect, and LifeLock cleared up the problem. His credit report remains spotless.

The Experian credit bureau's lawsuit basically claims that fraud alerts are only for people who have been victims of identity theft. This seems spurious; the text of the law states that anyone "who asserts a good faith suspicion that the consumer has been or is about to become a victim of fraud or related crime" can request a fraud alert. It seems to me that includes anybody who has ever received one of those notices about their financial details being lost or stolen, which is everybody.

As to deceptive business practices and fraudulent advertising -- those just seem like class action lawyers piling on. LifeLock's aggressive fear-based marketing doesn't seem any worse than a lot of other similar advertising campaigns. My guess is that the class action lawsuits won't go anywhere.

In reality, forcing lenders to verify identity before issuing credit is exactly the sort of thing we need to do to fight identity theft. Basically, there are two ways to deal with identity theft: Make personal information harder to steal, and make stolen personal information harder to use. We all know the former doesn't work, so that leaves the latter. If Congress wanted to solve the problem for real, one of the things it would do is make fraud alerts permanent for everybody. But the credit industry's lobbyists would never allow that.

LifeLock does a bunch of other clever things. They monitor the national address database, and alert you if your address changes. They look for your credit and debit card numbers on hacker and criminal websites and such, and assist you in getting a new number if they see it. They have a million-dollar service guarantee -- for complicated legal reasons, they can't call it insurance -- to help you recover if your identity is ever stolen.

But even with all of this, I am not a LifeLock customer. At $120 a year, it's just not worth it. You wouldn't know it from the press attention, but dealing with identity theft has become easier and more routine. Sure, it's a pervasive problem. The Federal Trade Commission reported that 8.3 million Americans were identity-theft victims in 2005. But that includes things like someone stealing your credit card and using it, something that rarely costs you any money and that LifeLock doesn't protect against. New account fraud is much less common, affecting 1.8 million Americans per year, or 0.8 percent of the adult population. The FTC hasn't published detailed numbers for 2006 or 2007, but the rate seems to be declining.

New card fraud is also not very damaging. The median amount of fraud the thief commits is $1,350, but you're not liable for that. Some spectacularly horrible identity-theft stories notwithstanding, the financial industry is pretty good at quickly cleaning up the mess. The victim's median out-of-pocket cost for new account fraud is only $40, plus ten hours of grief to clean up the problem. Even assuming your time is worth $100 an hour, LifeLock isn't worth more than $8 a year.

And it's hard to get any data on how effective LifeLock really is. They've been in business three years and have about a million customers, but most of them have joined up in the last year. They've paid out on their service guarantee 113 times, but a lot of those were for things that happened before their customers became customers. (It was easier to pay than argue, I assume.) But they don't know how often the fraud alerts actually catch an identity thief in the act. My guess is that it's less than the 0.8 percent fraud rate above.

LifeLock's business model is based more on the fear of identity theft than the actual risk.

It's pretty ironic of the credit bureaus to attack LifeLock on its marketing practices, since they know all about profiting from the fear of identity theft. Facta also forced the credit bureaus to give Americans a free credit report once a year upon request. Through deceptive marketing techniques, they've turned this requirement into a multimillion-dollar business.

Get LifeLock if you want, or one of its competitors if you prefer. But remember that you can do most of what these companies do yourself. You can put a fraud alert on your own account, but you have to remember to renew it every three months. You can also put a credit freeze on your account, which is more work for the average consumer but more effective if you're a privacy wonk -- and the rules differ by state. And maybe someday Congress will do the right thing and put LifeLock out of business by forcing lenders to verify identity every time they issue credit in someone's name.

This essay originally appeared in Wired.com.

There are also a lot of errors and misconceptions. With its aggressive advertising campaign and a CEO who publishes his Social Security number and dares people to steal his identity -- Todd Davis, 457-55-5462 -- LifeLock is a company that's easy to hate. But the company's story has some interesting security lessons, and it's worth understanding in some detail.

In December 2003, as part of the Fair and Accurate Credit Transactions Act, or Facta (.pdf), credit bureaus were forced to allow you to put a fraud alert on their credit reports, requiring lenders to verify your identity before issuing a credit card in your name. This alert is temporary, and expires after 90 days. Several companies have sprung up -- LifeLock, Debix, LoudSiren, TrustedID -- that automatically renew these alerts and effectively make them permanent.

This service pisses off the credit bureaus and their financial customers. The reason lenders don't routinely verify your identity before issuing you credit is that it takes time, costs money and is one more hurdle between you and another credit card. (Buy, buy, buy -- it's the American way.) So in the eyes of credit bureaus, LifeLock's customers are inferior goods; selling their data isn't as valuable. LifeLock also opts its customers out of pre-approved credit card offers, further making them less valuable in the eyes of credit bureaus.

And, so began a smear campaign on the part of the credit bureaus. You can read their points of view in this New York Times article, written by a reporter who didn't do much more than regurgitate their talking points. And the class action lawsuits have piled on, accusing LifeLock of deceptive business practices, fraudulent advertising and so on. The biggest smear is that LifeLock didn't even protect Todd Davis, and that his identity was allegedly stolen.

It wasn't. Someone in Texas used Davis's SSN to get a $500 advance against his paycheck. It worked because the loan operation didn't check with any of the credit bureaus before approving the loan -- perfectly reasonable for an amount this small. The payday-loan operation called Davis to collect, and LifeLock cleared up the problem. His credit report remains spotless.

The Experian credit bureau's lawsuit basically claims that fraud alerts are only for people who have been victims of identity theft. This seems spurious; the text of the law states that anyone "who asserts a good faith suspicion that the consumer has been or is about to become a victim of fraud or related crime" can request a fraud alert. It seems to me that includes anybody who has ever received one of those notices about their financial details being lost or stolen, which is everybody.

As to deceptive business practices and fraudulent advertising -- those just seem like class action lawyers piling on. LifeLock's aggressive fear-based marketing doesn't seem any worse than a lot of other similar advertising campaigns. My guess is that the class action lawsuits won't go anywhere.

In reality, forcing lenders to verify identity before issuing credit is exactly the sort of thing we need to do to fight identity theft. Basically, there are two ways to deal with identity theft: Make personal information harder to steal, and make stolen personal information harder to use. We all know the former doesn't work, so that leaves the latter. If Congress wanted to solve the problem for real, one of the things it would do is make fraud alerts permanent for everybody. But the credit industry's lobbyists would never allow that.

LifeLock does a bunch of other clever things. They monitor the national address database, and alert you if your address changes. They look for your credit and debit card numbers on hacker and criminal websites and such, and assist you in getting a new number if they see it. They have a million-dollar service guarantee -- for complicated legal reasons, they can't call it insurance -- to help you recover if your identity is ever stolen.

But even with all of this, I am not a LifeLock customer. At $120 a year, it's just not worth it. You wouldn't know it from the press attention, but dealing with identity theft has become easier and more routine. Sure, it's a pervasive problem. The Federal Trade Commission reported that 8.3 million Americans were identity-theft victims in 2005. But that includes things like someone stealing your credit card and using it, something that rarely costs you any money and that LifeLock doesn't protect against. New account fraud is much less common, affecting 1.8 million Americans per year, or 0.8 percent of the adult population. The FTC hasn't published detailed numbers for 2006 or 2007, but the rate seems (.pdf) to be declining.

New card fraud is also not very damaging. The median amount of fraud the thief commits is $1,350, but you're not liable for that. Some spectacularly horrible identity-theft stories notwithstanding, the financial industry is pretty good at quickly cleaning up the mess. The victim's median out-of-pocket cost for new account fraud is only $40, plus ten hours of grief to clean up the problem. Even assuming your time is worth $100 an hour, LifeLock isn’t worth more than $8 a year.

And it's hard to get any data on how effective LifeLock really is. They've been in business three years and have about a million customers, but most of them have joined up in the last year. They've paid out on their service guarantee 113 times, but a lot of those were for things that happened before their customers became customers. (It was easier to pay than argue, I assume.) But they don't know how often the fraud alerts actually catch an identity thief in the act. My guess is that it's less than the 0.8 percent fraud rate above.

LifeLock's business model is based more on the fear of identity theft than the actual risk.

It's pretty ironic of the credit bureaus to attack LifeLock on its marketing practices, since they know all about profiting from the fear of identity theft. Facta also forced the credit bureaus to give Americans a free credit report once a year upon request. Through deceptive marketing techniques, they've turned this requirement into a multimillion-dollar business.

Get LifeLock if you want, or one of its competitors if you prefer. But remember that you can do most of what these companies do yourself. You can put a fraud alert on your own account, but you have to remember to renew it every three months. You can also put a credit freeze on your account, which is more work for the average consumer but more effective if you're a privacy wonk -- and the rules differ by state. And maybe someday Congress will do the right thing and put LifeLock out of business by forcing lenders to verify identity every time they issue credit in someone's name.

---

Bruce Schneier is Chief Security Technology Officer of BT, and author of Beyond Fear: Thinking Sensibly About Security in an Uncertain World.

On May 13th, 2008 the Debian project announced that Luciano Bello found an interesting vulnerability in the OpenSSL package they were distributing. The bug in question was caused by the removal of the following line of code from md_rand.c

	MD_Update(&m,buf,j);
	[ .. ]
	MD_Update(&m,buf,j); /* purify complains */

These lines were removed because they caused the Valgrind and Purify tools to produce warnings about the use of uninitialized data in any code that was linked to OpenSSL. You can see one such report to the OpenSSL team here. Removing this code has the side effect of crippling the seeding process for the OpenSSL PRNG. Instead of mixing in random data for the initial seed, the only "random" value that was used was the current process ID. On the Linux platform, the default maximum process ID is 32,768, resulting in a very small number of seed values being used for all PRNG operations.

More info, from Debian, here. And from the hacker community here. Seems that the bug was introduced in September 2006.

More analysis here. And a cartoon.

Random numbers are used everywhere in cryptography, for both short- and long-term security. And, as we've seen here, security flaws in random number generators are really easy to accidently create and really hard to discover after the fact. Back when the NSA was routinely weakening commercial cryptography, their favorite technique was reducing the entropy of the random number generator.