Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software.  He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.”  I think that sums up the problem pretty well.  Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think.  The same thinking is used by people who install social networking applets and give them access to their personal data.  Someone, somewhere is taking care of the software security so I don’t have to.  It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives.  I’m safe in a plane or car because the government is looking out for me with standards and testing requirements.  I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space.  I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process.  I ask them if they have made security quality part of the requirements of the project and they say no.  Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers.  They have no evidence.  Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for.  So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems.  The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source.  Surely the developers writing the software are trained in secure software engineering.  Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers.  The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog.  He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment.  Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software. He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.” I think that sums up the problem pretty well. Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think. The same thinking is used by people who install social networking applets and give them access to their personal data. Someone, somewhere is taking care of the software security so I don’t have to. It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives. I’m safe in a plane or car because the government is looking out for me with standards and testing requirements. I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space. I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process. I ask them if they have made security quality part of the requirements of the project and they say no. Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers. They have no evidence. Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for. So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems. The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source. Surely the developers writing the software are trained in secure software engineering. Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers. The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog. He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment. Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

So I got my iPhone 3G on Friday morning and have been using it for a few days now. I have never used one before, don't use an iPod or even a Mac computer.  The iPhone was incredibily easy to use and without using and manuals quickly had a most everything working and downloaded a bunch of apps from the app store. 

Over all, the iPhone just is really nice to use and in many ways very easy, polished and intuitive. In other ways, it is still missing some key features in my book:

1. Sort and filter email be date, sender, etc.
2. Select more than one mail at a time to delete, move, copy.  Yes I know you can go to edit and select messages to work on, but you still have to select them one at a time. In Windows Mobile you can just run your finger over multiple messages to complete this.
3. Deleting duplicate contacts in bulk.  Doing them one at a time is just painful
4. A task manager. I would like to see some list that shows me which apps are running, how many resources they are using, battery usage and stuff like that.  Also to shut down running apps
5. Better calendar integration. I tried to click on and open calendar items, but just does not seem to work.
6. The battery sucks! I am not getting more than about 6 to 7 hours of battery time. I think I have to turn off the push for my Exchange email.  This is much less that I was getting on my Windows Mobile phone.

I do like the phone, the iPod MP3 and camera and the overall "feel" of the phone. Went to the Apple store in the maill (which was jam packed) and bought a rubberized case, but was unable to get a phone car charger for it yet.  I ordered one for 5 bucks on Amazon and will see it if works.

All in all, things are OK but I am going to withhold my final verdict for a while yet.

So I got my iPhone 3G on Friday morning and have been using it for a few days now. I have never used one before, don't use an iPod or even a Mac computer.  The iPhone was incredibily easy to use and without using and manuals quickly had a most everything working and downloaded a bunch of apps from the app store. 

Over all, the iPhone just is really nice to use and in many ways very easy, polished and intuitive. In other ways, it is still missing some key features in my book:

1. Sort and filter email be date, sender, etc.
2. Select more than one mail at a time to delete, move, copy.  Yes I know you can go to edit and select messages to work on, but you still have to select them one at a time. In Windows Mobile you can just run your finger over multiple messages to complete this.
3. Deleting duplicate contacts in bulk.  Doing them one at a time is just painful
4. A task manager. I would like to see some list that shows me which apps are running, how many resources they are using, battery usage and stuff like that.  Also to shut down running apps
5. Better calendar integration. I tried to click on and open calendar items, but just does not seem to work.
6. The battery sucks! I am not getting more than about 6 to 7 hours of battery time. I think I have to turn off the push for my Exchange email.  This is much less that I was getting on my Windows Mobile phone.

I do like the phone, the iPod MP3 and camera and the overall "feel" of the phone. Went to the Apple store in the maill (which was jam packed) and bought a rubberized case, but was unable to get a phone car charger for it yet.  I ordered one for 5 bucks on Amazon and will see it if works.

All in all, things are OK but I am going to withhold my final verdict for a while yet.

The towers are gone now, reduced to bloody rubble, along with all hopes for Peace in Our Time, in the United States or any other country. Make no mistake about it: We are At War now -- with somebody -- and we will stay At War with that mysterious Enemy for the rest of our lives. 

It will be a Religious War, a sort of Christian Jihad, fueled by religious hatred and led by merciless fanatics on both sides. It will be guerilla warfare on a global scale, with no front lines and no identifiable enemy. Osama bin Laden may be a primitive "figurehead" -- or even dead, for all we know -- but whoever put those All-American jet planes loaded with All-American fuel into the Twin Towers and the Pentagon did it with chilling precision and accuracy. The second one was a dead-on bullseye. Straight into the middle of the skyscraper. 

Nothing -- even George Bush's $350 billion "Star Wars" missile defense system -- could have prevented Tuesday's attack, and it cost next to nothing to pull off. Fewer than 20 unarmed Suicide soldiers from some apparently primitive country somewhere on the other side of the world took out the World Trade Center and half the Pentagon with three quick and costless strikes on one day. The efficiency of it was terrifying. 

We are going to punish somebody for this attack, but just who or what will be blown to smithereens for it is hard to say. Maybe Afghanistan, maybe Pakistan or Iraq, or possibly all three at once. Who knows? Not even the Generals in what remains of the Pentagon or the New York papers calling for WAR seem to know who did it or where to look for them. 

This is going to be a very expensive war, and Victory is not guaranteed -- for anyone, and certainly not for anyone as baffled as George W. Bush. All he knows is that his father started the war a long time ago, and that he, the goofy child-President, has been chosen by Fate and the global Oil industry to finish it Now. He will declare a National Security Emergency and clamp down Hard on Everybody, no matter where they live or why. If the guilty won't hold up their hands and confess, he and the Generals will ferret them out by force. 

Good luck. He is in for a profoundly difficult job -- armed as he is with no credible Military Intelligence, no witnesses and only the ghost of Bin Laden to blame for the tragedy.

One unintended lesson I take away from Hunter's life is how important patience is. Obama is a politician and may yet disappoint us all, but I gotta believe Hunter would be seriously impressed. If he had waited another couple of years, he may have seen a lot of the stuff he fought for in 1968 and 72 come to fruition. Sometimes you are just 36-40 years ahead of your time and you have to be ok with that and figure out how to deal if possible. (Note - it sure sometimes feels this way in software security). Speaking of security:

Security 

by Hunter S. Thompson (1955). 

Security ... what does this word mean in relation to life as we know it today? For the most part, it means safety and freedom from worry. It is said to be the end that all men strive for; but is security a utopian goal or is it another word for rut? 

Let us visualize the secure man; and by this term, I mean a man who has settled for financial and personal security for his goal in life. In general, he is a man who has pushed ambition and initiative aside and settled down, so to speak, in a boring, but safe and comfortable rut for the rest of his life. His future is but an extension of his present, and he accepts it as such with a complacent shrug of his shoulders. His ideas and ideals are those of society in general and he is accepted as a respectable, but average and prosaic man. But is he a man? has he any self-respect or pride in himself? How could he, when he has risked nothing and gained nothing? What does he think when he sees his youthful dreams of adventure, accomplishment, travel and romance buried under the cloak of conformity? How does he feel when he realizes that he has barely tasted the meal of life; when he sees the prison he has made for himself in pursuit of the almighty dollar? If he thinks this is all well and good, fine, but think of the tragedy of a man who has sacrificed his freedom on the altar of security, and wishes he could turn back the hands of time. A man is to be pitied who lacked the courage to accept the challenge of freedom and depart from the cushion of security and see life as it is instead of living it second-hand. Life has by-passed this man and he has watched from a secure place, afraid to seek anything better What has he done except to sit and wait for the tomorrow which never comes? 

Turn back the pages of history and see the men who have shaped the destiny of the world. Security was never theirs, but they lived rather than existed. Where would the world be if all men had sought security and not taken risks or gambled with their lives on the chance that, if they won, life would be different and richer? It is from the bystanders (who are in the vast majority) that we receive the propaganda that life is not worth living, that life is drudgery, that the ambitions of youth must he laid aside for a life which is but a painful wait for death. These are the ones who squeeze what excitement they can from life out of the imaginations and experiences of others through books and movies. These are the insignificant and forgotten men who preach conformity because it is all they know. These are the men who dream at night of what could have been, but who wake at dawn to take their places at the now-familiar rut and to merely exist through another day. For them, the romance of life is long dead and they are forced to go through the years on a treadmill, cursing their existence, yet afraid to die because of the unknown which faces them after death. They lacked the only true courage: the kind which enables men to face the unknown regardless of the consequences. 

As an afterthought, it seems hardly proper to write of life without once mentioning happiness; so we shall let the reader answer this question for himself: who is the happier man, he who has braved the storm of life and lived or he who has stayed securely on shore and merely existed?

A ship is safest at port, but thats not why we build ships.

• My presentation on buy vs build vs outsources
• "Why Replace Your Baby?"
• Log Management "Strategy:" Built ->Suffer->Suffer->Suffer

Q1: For those companies that have successfully implemented enterprise-wide logging, what were the big nasty surprises that they encountered?

A1: Here are a few:

• political boundaries within the organization: "these are our logs, and you are not getting them"
• privacy laws: some logs cannot be collected in some countries; some cannot cross the border, some cannot be seen by some people, etc. This is true mostly in EU, less in US.
• legal blocks: work with legal before deploying any org-wide log management; legal might try to prevent certain data from ever being created (for fear of being legally discovered later)
• log volume: underestimating log volume is common and pretty nasty
• related to the last one: vendors being "optimistic" about their tool scalability
• time synchronization (of course!), specifically, lack thereof.

Q2: For those companies that have successfully implemented enterprise-wide logging, what was their implementation approach?

A2: Typically, 2-3 vendor PoC or pilot first. Then with the chosen vendor: phased approach based on location + type of log source (e.g. firewalls, then routers, then OS, then proxies, etc) + network topology (e.g. DMZ, then internal) + log source criticality (e.g. critical servers first; the rest next). This might be handy to look at.

Q3: What kind of storage requirements have been experienced by those organizations who have successfully implemented enterprise-wide logging?

A3: Massive? :-)

Here is a simple example: PCI DSS is a bit more aggressive than NERC since it mandates 1 year of log retention vs NERC 90 days, so: 1 year worth of logs is = 365 days x 24 hours x 3600 seconds x 1 (one!!!) busy firewall with 100 log messages each second x 200 bytes per message average (e.g. valid for PIX and ASA devices) = 588 gigabytes / year of raw log data uncompressed (assuming 10x compression you'd get about 60GB of compressed log data per year)

Store it in RDBMS? Multiple it by 2-3. Have an index? Add about 30%.

The bottom line is: terabyte is the unit to measure logs.

Q4: At the organizations that have successfully implemented enterprise-wide logging, how logging impacted network and system performance?

A4: Too broad a question, so here are a few pointers:

• logging affects performance much more on some types of systems compared to other types: most painful examples are databases where some people (can't find a link...sorry) report performance loss of up to 40% if logging all SELECT statements and other data retrieval commands (you need to log selectively on these); in other cases (e.g. web servers) there is no performance loss and logging is "always on"
• log collection: agents impact system performance (long post on this subjects): a little when they run (everybody knows this) and A LOT when they crash (few people think about it - agent software memory leaks are not uncommon); unlike agents, remote agentless log collection barely affects system performance (unless you have one of the few esoteric cases)
• log transfer and network performance: look for compressed (logs compress really well), TCP-based transfers; syslogging over UDP uncompressed has a chance of doing a pipe saturation DoS on your network. Yes, people say "use a dedicated LAN," but this is definitely wishful thinking for many. Also, raw UDP syslog in large quantities over WAN = insanity :-)

Q5: What were some successful strategies for obtaining buy-in from system owners and operators in regards to turning logging on?

A5: OK, also too broad a question, but here are some pointers:

• provide them a useful service based on their logs (e.g. performance measurement, availability monitoring, compromise detection :-), or other security metrics, etc)
• help them with their compliance mandates (e.g. create reports that they can show to the auditors that "bug" them)
• give them tools to better solve their problems (e.g. allow access to a log management tool so that can investigate issues better, search the logs, check on their users, etc)

Q6: How the organizations that have successfully implemented enterprise-wide logging dealt with unusual devices (=log sources) that have no log management vendor support?

A6: They were in massive pain - if they choose a log management vendor wrong. You need to look for vendors that have "universal log source support" with NO requirement for a custom rules or custom collector/connector/agent development. Some vendors have generic text log collectors that can grab and analyze unknown logs. Typically this is done via some form of text indexing that works across all logs, including those from unknown, vertical, esoteric or custom-developed log sources

Hope it was useful!

You’ve read the headlines.  You’ve heard the buzzwords.  

Cloud Computing just seems like hype, right?  

“But it’s just another technology getting hyped to the max”.

The best case scenario is that your analysis is correct and you can go back to reading Slashdot and Daily Dave (you are reading Daily Dave aren’t you?).  You can pride yourself on your ability to recognise web hysteria and laugh at the losers that invested, wrote blog posts (!) and dared to take it seriously.

OK.  Now lets flip that around and just say for a moment you’re wrong - that Cloud Computing turns out to be a huge deal and takes off.  What could that mean for your day job?  No in-house servers to secure?  No in-house security operations to deal with? No in-house penetration tests to run?  No vulnerability assessment tools to run? No incident response where you actually ‘do something’?  

One scenario is you find yourself on a constant round of conference calls with 3rd parties trying to ‘pin down’ security in the cloud…  If you thought handling security issues associated with outsourcing was painful and slow, the Cloud will bring a multitude of competing providers that decision makers can switch from ‘digitally’ when the numbers ($$) make sense.

As the person responsible for your employer’s security arrangements, you may want to consider these 5 reasons for not dismissing Cloud Computing out of hand:

• Unless you work for an IT company, your employer did not go into business to ‘do IT’.  They are in business to sell a product or a service - in-house IT may have enabled that up to now but it was out of need rather than desire.  Cloud Computing has hit the cover of popular business magazines - its starting to get on the radar of CEO’s that ask questions like ‘how can I cut my costs?’, ‘how can I make my business more agile?’.  They may not switch overnight, but once the first goes in a given vertical, the clock is ticking.
• The temptation to contractually outsource security responsibility.  ”Our customer data got stolen from a cloud storage provider - not us - we don’t run IT!”.  Sure the buck stops with the org from a regulatory perspective but media coverage around recent data leakages involving 3rd party providers illicits a mixed reaction and thus diffuses the “reputation issues” to some extent.
• The skills you need to deal with Cloud Security may be different from the skills you have today.  Your “window” on Cloud security will be what the Cloud Provider gives you.  Beyond that you may be able to do an on-site audit from time to time but its a shared facility so no monkey in a cage pen-testing, scanning or filesystem forensic analysis.
• There’s a large cloud forming over the horizon.  The level of investment by providers doesn’t bear ignoring.  IBM, Google, Amazon, Microsoft and others are ploughing hundreds of millions of dollars building out data centers specifically for Cloud Computing.
• You may just end up working for the Cloud Provider!  This is something I believe will start happening in the next 2-3 years.  If you need a second opinion, go see Richard Bejtlich’s blog when he shared his own perspective.

What say you?  Hype or pending reality?