[SecurityRatty] tag: unix

The Economics of Finding and Fixing Vulnerabilities in Distributed Systems

Tue, 18 Nov 2008 19:47:55 +0000

The Economics of Finding and Fixing Vulnerabilities in Distributed Systems

Quality of Protection Keynote

Alexandria, VA

October 27. 2008

Gunnar Peterson

Managing Principal, Arctec Group

Blog: http://1raindrop.typepad.com

When Andy Ozment asked me over the summer to do this talk at QoP, I knew back in August that the topic I wanted to address was security and economics. So to that end I would like to start by thanking all of our friends on Wall Street and here in Washington DC for providing such a rich tapestry of recent events that I can speak to.

Like many people in this industry, my focus on security was fundamentally altered by Dan Geer's speech "Risk Management is Where the Money Is"[1], there are not many people who can call a ten year shot in the technology business, but Dan Geer did. The talk revolutionized the security industry. Since that speech, the security market, the vendors, consultants, and everyone else has realized that security is really about risk management.

Of course, saying that you are managing risk and actually managing risk are two different things. Warren Buffett started off his 2007 shareholder letter [2] talking about financial institutions' ability to deal with the subprime mess in the housing market saying, "You don't know who is swimming naked until the tide goes out." In our world, we don't know whose systems are running naked, with no controls, until they are attacked. Of course, by then it is too late.

So the security industry understands enough about risk management that the language of risk has permeated almost every product, presentation, and security project for the last ten years. However, a friend of mine who works at a bank recently attended a workshop on security metrics, and came away with the following observation - "All these people are talking about risk, but they don't have any assets." You can't do risk management if you don't know your assets.

Risk management requires that you know your assets, that on some level you understand the vulnerabilities surrounding your assets, the threats against those, and efficacy of the countermeasures you would like to use to separate the threat from the asset. But it starts with assets. Unfortunately, in the digital world these turn out to be devilishly hard to identify and value.

Recent events have taught us again, that in the financial world, Warren Buffett has few peers as a risk manager. I would like to take the first two parts of this talk looking at his career as a way to understand risk management and what we can infer for our digital assets.

Warren Buffett's evolution as an investor can be broken up into two parts. He began his career very much influenced by Ben Graham, who sought to buy "cheap stocks", comparing the price of the stock to value of the company's assets, and placing many, diversified bets on companies whose share price was below the total assets. Note that the businesses may have been of unremarkable quality, but when the price was right Graham would buy in, wait for it to rise and then sell. This was the dawn of value investing.

Buffett's later career departed from Graham's strict, statistical measures, where he sought to buy into companies that were selling at a fair price, but were also high quality businesses. We will examine high quality in Part 2 of this talk, but first we go to Part 1 which is asset value.

Why does a talk on finding and fixing vulnerabilities start with valuing assets? The reason is that vulnerabilities are everywhere, we are literally marinating in them. Interesting vulnerabilities are attached to high value assets. In a world that quite literally presents us with too much information, we need screens to sift out what is worth paying attention to. You can run your vulnerability assessment tool of choice on your system, and come back with hundreds or thousands of vulnerabilities, but which ones should you pay attention to and act on? The first part of answering this question is asset value.

When Warren Buffett was 19 years old studying at the University of Nebraska, he read Ben Graham's book "The Intelligent Investor", Buffett said he thought it was the best book on investing he has ever read and still feels that way today. In the Intelligent Investor Graham lays out the framework of value investing. Specifically, Graham talks about three concepts - Mr. Market, a stock is a piece of a business, and Margin of Safety.

Mr. Market is a fictional, teaching device invented by Graham. You imagine that you have a somewhat manic depressive business partner called Mr. Market. Every day, Mr. Market comes into the office and offers you quotes on companies, some days he is in a good mood and the prices are high, other days he is gloomy and prices are low. The market is a quote machine, for quoting prices, not a value assessment machine. Your job is to wait for the right price, and you are free to take as many passes and be as patient as you would like, Mr. Market will just show up the next day and throw out a new price.

Graham used Mr. Market to teach us the separation between a price of a stock, and the value of a company. The second big concept from Intelligent Investor is that buying a stock is buying a small piece of the underlying business. You are not buying a roulette chip, or a number that fluctuates in the newspaper every day, rather you are buying a piece of the company's existing and future cash flow. What the stock market says General Electric is worth yesterday, today or tomorrow is separate from GE's actual ability to generate cash flow.

The last big concept in "The Intelligent Investor" and the one seemingly most applicable to information security is the Margin of Safety. Graham's margin of safety involved calculating the intrinsic value of a business and then buying stock where the market cap of a company is less than its intrinsic value. So if a company has $100 million in assets and a market capitalization of $75 million, then an investor would get a 25% margin of safety. Ideally, Graham wanted to buy stocks that were selling for one half of their book value, i.e. with a 50% margin of safety. Graham said that buying stocks without a margin of safety, above their book value, speculation, not investing.

So price is readily available, but how do we calculate intrinsic value so that we can ascertain the margin of safety? Graham used quantitative statistical measures, relying heavily on the company's book value, like its hard assets. What would it take for a competitor to reproduce the company's assets - its factories, distribution system, and so on. The difference between the book value of the assets and market cap is the margin of safety.

What can we learn in information security from this quantitative approach? Where price and value are readily ascertainable we should build countermeasures and eliminate on vulnerabilities that give our assets a wide margin of safety. Since budgets are not unlimited we should prefer vulnerabilities that are cheap to find, cheap to fix.

First to the asset question, information security budgets like all IT budgets are crufty, they are not a reflection of today's top issues and priorities so much as an accumulating snowball of decisions, legacy contracts, and solution attempts to yesteryear's problems. Today the normal Information Security budget is just a legacy artifact from bygone years when the network was the purported greatest vulnerability. If you were around in 1995, you remember the great gnashing of gears as the enterprises opened up their networks, connected their back ends to the Web and began to transact business in the giant virtual space.

The security people huffed and puffed that it was dangerous but there was simply too much money to be made, so businesses went ahead. The security people would not go down without a fight and insisted on countermeasures. They got two - the network firewall and SSL. The firewall was used to separate the average Fortune 500s network of hundreds of thousands of machines, employees, consultants, and partners from the web at large. SSL was used to protect the network channel between the web server and the client browser. so the network firewall separated the network segments, and SSL in effect encrypted the last mile of many million complex transactions and computations.

In 1995, this seemed like a good security architecture. When we built out these security architectures, the eCommerce market was derided as a toy. Amazon famously lost money for years - losing a little on every transaction but making it up in volume. When the market is nascent, a quaint security architecture offers cost effective protection. But what about 2008? Those cute little eCommerce buggers have grown they even make profits now - market caps measured in the tens of billions, accumulating large cash hordes, no debt, and the largest ones are in better financial shape than the financial services players that kicked sand in their face in the dotcom era.

And its not just eCommerce, the "real" economy Fortune 500 types are all connected as well. Directly and indirectly the Web is seeping into all businesses. Major changes from when the security architecture of the web was built out. But has the security architecture changed to reflect these new business realities? Not a bit of it!

We can use the book value of the IT budget investments and the book value of the Information Security investments to see what kind of Margins of Safety Information Security groups are engineering.

Let's look at some market data, Gary McGraw reviewed the numbers [2] in software security for 2007, breaking down software security sectors like tools and services. Here is a summary of his findings on software security tools:

"One of the most important developments in the software security market can be seen in the tools space which, combined, almost doubled to $150-180 million. Top of list are two major acquisitions that closed in 2007: Watchfire's purchase by IBM (somewhere in the range of $120-150 million on 2006 revenue of $26 million) and SPI Dynamics's purchase by HP (for around $100 million on 2006 revenue of $21.2 million).

...

The black box space was flat in 2007, with IBM/Watchfire checking in at $24.1 million and HP/SPI Dynamics earning $22.3 million. Smaller companies in the space, including Cenzic, Codenomicon, WhiteHat and the like had combined revenues around $12.5 million (a growth of 25%, though Cenzic grew 16% and WhiteHat 52%). Most of the growth "hiccup" in the black box market can be attributed to the serious challenges posed by any acquisition. So far 2008 looks to be back on track from a growth perspective in the black box testing space. The global reach that IBM and HP offer are already making a big difference.

On a more positive note, static analysis tools for code review grew at a healthy clip in 2007 into a $91.9 million dollar market. Fortify was up 83% to $29.2 million. Klocwork grew over 60% to $26 million. Coverity grew over 50% to $27.2 million. Ounce Labs tripled their revenue to $9.5 million."

These are very nice growth numbers, what company doesn't want 83% growth? However, the let's look at the total picture and compare the software security countermeasures against other security mechanisms. Gary McGraw's estimate shows the software security space coming in at $150 Million total, yet we see a company like Checkpoint that won the network security war in 1995 with earnings of around $900 Million! One single network security vendor is 6 times bigger than the entire software security space, in what alternate universe does this make sense?

This is where we begin to see that decisions in the People's Republic of Information Security have no real risk management thinking, they truly are swimming naked and hoping the tide doesn't go out.

Let's look at network assets. Obviously Cisco is the biggest, they earned $39.5 Billion last year. Pretty stellar. So spending $900 Million (Checkpoint) to defined $39.5 Billion seems like a pretty good deal.

Except, let's compare software security spending - last year Microsoft earned $60 Billion, SAP $16 billion, and Oracle $22 Billion. So that is about $98 Billion in just three vendors and you are going to "defend" that with allocating $150 Million worth of software security tools?

On the network side we are buying $900 million of security countermeasures (Checkpoint firewalls) to protect $39.5 billion worth of Cisco gear, about 2.3% of the network investment goes to security.

On the software side, we are buying $150 million of security countermeasures (like static analysis and black box scanners) to protect $98 billion of software (you know the stuff that runs the whole business), roughly coming to about 0.2% of the software budget goes to security.

This is very disturbing. From a prioritization standpoint The People's Republic of Information Security is misaligned by an order of magnitude at least. Next time you read about a data breach, or see an auditor's report with thousands of findings you won't have to wonder how it happened. It happened because Information Security doesn't have its eye on the ball, it invests in network security not because those controls have greater efficacy (the whole point of networks is they are dumb), no, they invest in network firewalls because they bought a bunch in 1995, some more in 1998, and heck they just kept buying them, the Checkpoint rep kept showing up and taking CISOs out to play golf, contracts got renewed, and poof - there goes the security budget.

Consider that software security tools could grow 50% a year for five years and still be half of where Checkpoint is today.

The optimistic way of looking at all this data is that there is major room for growth for software security, if you take network security as a target for a mature industry and assume that 2.3% is a reasonable margin of safety, then the software security space should evolve to around 2% of the software space meaning that it should evolve into a $2 billion space around fifteen times larger than it is today. Unprotected assets will either be protected or will cease to be assets, VCs get your check books ready.

My friend Brian Chess has a nice way of looking at this he says 2007 was the turning point - "the first year there was a bigger market for products that help you get code right than there was for products that help you demonstrate a problem exists."

Now I am not suggesting that Information Security budgets have to be aligned with IT budget one for one, but I do think that looking at the overall IT budget is the starting point. If Information Security has a more cost effective security mechanism they should deploy it, but the starting point should be aligned to the business. Businesses spend most of their money on software, and there are very good reasons - competitive advantage, increased revenues and lower costs. Information Security spends most of its money on network security, and there is no good reason why, except that it was a seemingly good idea in 1995. You really don't have to go beyond the book value of IT investment as a whole versus Information Security to see a stunning disparity. Information Security's job is to deliver a Margin of Safety to the business, but they are not.

To deliver a real Margin of Safety to the business, I propose the following based on a defense in depth mindset. Break the IT budget into the following categories:

- Network: all the resources invested in Cisco, network admins, etc.

- Host: all the resources invested in Unix, Windows, sys admins, etc.

- Applications: all the resources invested in developers, CRM, ERP, etc.

- Data: all the resources invested in databases, DBAs, etc.

Tally up each layer. If you are like most business you will probably find that you spend most on Applications, then Data, then Host, then Network.

Then do the same exercise for the Information Security budget:

- Network: all the resources invested in network firewalls, firewall admins, etc.

- Host: all the resources invested in Vulnerability management, patching, etc.

- Applications: all the resources invested in static analysis, black box scanning etc.

- Data: all the resources invested in database encryption, database monitoring, etc.

Again, tally each up layer. If you are like most business you will find that you spend most on Network, then Host, then Applications, then Data. Congratulations, Information Security, you are diametrically opposed to the business!

Its not just about alignment for alignment's sake, its about applying controls as a way to have a Margin of Safety properly placed so that when not if there is a failure on a higher value asset you are relatively better positioned to deal with it.

The pure statistical approach can only take us so far. Buffett said he would be a lot poorer if all he did was listen to Ben Graham. Book value is great to see the diametric opposition mentioned above, but it doesn't really tell us much about the efficacy of the security mechanisms.

What we do get out of this statistical approach is a screen. The asset value screen filters out subjective opinion and narrows the field for where we need to dig in to do the high value, time consuming analytical work.

The second part of Warren Buffett's career and the second part of this talk leave behind pure statistical measures. In Warren Buffett's case he was joined by a guy named Charlie Munger who talked him out of the pure Ben Graham approach. Charlie Munger has a saying - "a great business at a fair price beats a fair business at a great price." Where Graham was focused on price and margin of safety, Munger wants a fair price but also a high quality business. This lead to Warren Buffett's company Berkshire Hathaway investing in companies like Coca Cola, Wells Fargo, and American Express, where the prices were far from dirt cheap (as Graham would have wanted), but the long term returns were outstanding.

In our world of Information Security, we start by aligning our priorities with the business using the thumbnail defense in depth approach, but then we would like to invest in high quality, effective controls.

To get at the notion of control quality and effectiveness, I am going to start part 2 of this talk with a brief history of software. The first web software was just static HTML, but web software really got interesting when developers started creating dynamic websites using CGI an PERL.

Once websites were hooked up to company databases and were not just serving static content, the security people realized they needed a security architecture, and they sprung into action. What they came up was was model that divided the world into "good stuff" which was comprised of all their networks, systems, and data; and then there was everything else the "bad stuff" on the Internet. So job one of the early days Internet security architecture was to separate all your good stuff (i.e. your network) for the bad stuff (the Internet). To do this the security people used a sophisticated tool called Visio to draw a flaming brick wall on the network diagram, and this flaming brick wall was supposed to keep the good stuff and the bad stuff separate.

The security people also realized that the data and session tokens that they served up from their Web server would have to traverse the "bad" neighborhood called the Internet, so they added one more security mechanism to secure the last mile of the transaction - SSL between the browser and the Web server.

And this was the state of the art security architecture used circa 1995 to protect the earliest dynamic web applications.

What happened next was that the dotcom boom started to happen and businesses realized they could make some real money on the Web, the web apps started to get more sophisticated, more personalization, richer session experiences and so on. This led the Java people to create JSP and the Microsoft people to create ASP, and of course the PERL people to create even greasier PERL scripts, all of this in the effort to pooling resources and sessions on the Web server. The security people defended this new application programming model with network firewall and SSL.

Around 1998, developers began building out more distributed N tier or 3 tier applications that separated the business logic layer, the presentation layer and the data access layer. Among other things, your web application could seamlessly integrate data from multiple back ends systems. Let's say you have pricing data in Oracle, order data in SAP, and customer data in a Mainframe. You write separate data access objects, apply business logic in the middle tier and then you tie it all together in a friendly user interface. At this point the web applications are beginning to integrate across departments and geographic boundaries, huge critical chunks of the business are now connected to the web. How did the security people defend this part of the business? They applied the same 1995 security architecture - network firewall and SSL.

Around 1999-2000 timeframe businesses relied on web applications for major parts of the revenue, and the apps were built in different technologies like Java and Microsoft technologies, but the customer didn't care (still doesn't), the customer wanted (and still wants) data access and functionality. So to integrate the disparate technologies, SOAP and XML were deployed so that Microsoft could talk to Java and so Websphere could talk to Weblogic and so on. And, oh yes, SOAP and XML were used to connect B2B networks so partners in a supply chain and business process can exchange data and interoperate. SOAP and XML present a fundamentally new programming model based on a message document style integration, where XML is used to mesh together data and functionality across platforms. SOAP and XML have no security model by default for authentication, authorization, and confidentiality. How did the security people deal with this? They kept the security architecture the same as they had in 1995 - network firewalls and SSL.

The software world did not stop innovating in 2000 of course, in the last few years we have seen Web services and XML form the basis of baroque and powerful SOAs and simple REST applications. We have seen Web 2.0 come on the scene, and entirely new networked applications built on top of that.

What we have not seen, is a single meaningful change in security architecture in 13 years. Developers have evolved, businesses have increasingly bet their entire business models on the web and they have increased security budgets. But what has the security architecture as its deployed in the field got to show for all of this? More firewalls and more SSL connections.

Since Information Security has proven incapable of evolving, it is time to learn from a discipline that has mastered innovation - software development, and yes, I will step back in case the lightning bolts hits.

What does software development focus on these days? Well, let's look at Service Oriented Architecture (SOA), all hype aside I look at SOA as a set of technologies that delivers three things:

Virtualization: we want Beijing, Bangalore and Boston to communicate.

Interoperability: we want our .Net stuff to talk to our java stuff.

Reusability: how many order/claim/pricing/customer systems does one company need?

To build out their SOA, developers separated the application interface from its implementation. So you can host the interface in a variety of locations, but its separate from the application logic and data.

This is also a useful trick for putting services like SOAP through the firewall. SOAP was designed as a firewall friendly protocol. When SOAP first came out, Bruce Schneier said calling SOAP a firewall friendly protocol is like having a skull friendly bullet. Which is a great line and explains why his books fly off the shelves, it does not explain, why security people think an architecture designed in 1995 is the one we should be using today. Maybe the problem is not that the developers figured out how to go through the firewall to get the data their customers want, maybe the problem is that the firewall is the sum total of the security architecture, and it never adapted.

A big part of this problem is that we have left Newton's world behind and entered Einstein's universe. Mainframes are Newton’s world, we have THE computer, THE price, THE record and so on.

As Pat Helland explained [4,5], Mainframes are Newron's world, but Distributed computing is Einstein’s world. More specifically in the Einstein world of distributed computing - "Computers don’t make decisions, computers try to make decisions." Our computers don't really make a decision, they say you can buy this book from Amazon at this price, we have it in stock and will deliver on such and such a date. But the warehouse runs out, the pallet gets dropped in the warehouse, your boo is crushed, and the package is stolen off your front step. The computer confirmed your transaction, but the real world intervened.

So we don't have iron clad decisions, instead its all about Memories (last time I checked your book was in stock), Guesses (we should be able to ship on this date) and Apologies (sorry the forklift ran over your book)

Translating this into security, security mechanisms don’t make policy-based decisions, security mechanisms try to make policy-based decisions

Some examples of memories, guesses and apologies in security

Memories

Security Policies - for example Triple A policy

Triple A policies can memorize a map of subjects, objects, and roles. They can even replicate these memories and play them back at runtime to try to make policy enforcement decisions.

Guesses

Security Policy Enforcement Decision

Unfortunately, while the policy enforcement decisions can be based on memorized logic, the decision itself is still a guess, even in the case of Triple A. Any guesses why? Because, the authentication process itself is a guess. It happens to be a guess that you then bind to a principal so it looks very official once you bind your guess to a Kerberos ticket or SAML assertion, but it still a guess.

Apologies

Giant Global Bank is sorry your account was compromised!

And this leads to lots and lots of apologies by companies with poor access control models.

Some additional examples of information security memories, guesses and apologies.

Example Memories - Triple A Security Policies, Audit logs, User account information , Authorization Logic - concrete mapping Subject, Resource, Condition, Action

Example Guesses - Security Policy Enforcement Decision Points, Authentication Logic, Monitoring, detection, fraud response

Example Apologies - Identity Management tools - provisioning, deprovisioning, Reimburse customer for fraud losses, Compensating Transaction - Giant Global Bank is still sorry your account was compromised!

The point of this is that security memories, guesses and apologies utilize different processes, different people, and different capabilities to be effective.

What trends can we identify to lead us toward better qualitative analysis based on the best practices of virtualization, interoperability and reusability.

Virtualization

Finding Vulnerabilities in a Virtualized World is a problem because applications are more configured than coded. Runtime behavior and structure not apparent due to weak typing and inversion of control.

Result - finding bugs becomes harder. Action - use screens to target finding time and resources

Fixing Vulnerabilities in a Virtualized World is a problem because how do I locate the controls when interfaces run in Beijing, Bangalore and Boston?

Result - synchronization and/or replication of security policy is problematic. Action - decentralized policy enforcement points and policy decision points.

Interoperability

Finding interoperable vulnerabilities

XSS - Javascript is an equal opportunity offender - interoperability for developers and attackers alike.

Fixing interoperable vulnerabilities

App servers, ESBs, and services are the attacker’s red carpet to your enterprise, right into your book of business. Interoperable access control can be leveraged across the enterprise.

Use XML signature for authentication and integrity

…

Use XML encryption to protect sensitive data, don't pass sensitive data in the clear

My Credit Card Number

Encrypt the data

…

XNQ0a4legiie5mWFxO6CQkk2hhldYNnKroObue/LXS/VYtvaTgMbCujhGExDi+vlkU//Qc2/T6mx0WVTmBMT3z8rogha8jD+nS9Zr2Bc3CwoTh2lh8wL3D0DEu91iwJT9JByLGXvt7v9lyuxK0ooDOYEClsH974CPmTs3tBC+GQ=

To ensure that these controls are applied use automated tools like static analysis to scan for security mechanism use and coverage.

In terms of reusability findings and fixes consider two bug findings

Session management bug: session state is passed around to every component, service and user. Makes for many high priority findings in audit report, also the fix is required on virtually every program

Data validation bug: Data access object (DAO) has a SQL injection hole. One major high priority finding in report. DAO used by many business logic classes, one fix location serves many classes

To bring these factors together, I generally use a scorecard index [6], so you can measure such things as transport security, message security, threat protection and so on. The hard work in developing the index is developing a useful scale. A scale for XML tokens could use the following

0: no token

1: hashed token

2: hashed and signed token

3: hashed and signed token from standard authoritative source

An example scale for XML validation could use:

0: no validation

1: schema validation

2: schema validation against hardened schema

3: schema validation against standard, hardened schema

These indexed scales are used to show maturity across the factors in the scorecard. The first part of the talk described value, the value assessment is used to focus time and effort on high value assets. The value assessment can be determined quantitatively. There is hard analytical work to qualitatively determine the scorecard, index, and scales, the quantitative value assessment is used to screen out high value targets for these endeavors. The scoring index is used to track progress and improve quality over time. In the best case scenario, automated tools are used to perform the checks described in the index, and once security is automated just like software developers we may see security innovation make progress in years not decades.

Thank you for your time.

1 "Risk Management is where the Money Is" by Dan Geer, http://catless.ncl.ac.uk/Risks/20.06.html

2 Berkshire Hathaway 2007 Shareholder Letter by Warren Buffett, http://www.berkshirehathaway.com/letters/2007ltr.pdf

3 "Software [In]security: Software Security Demand Rising, by Gary McGraw

http://www.informit.com/articles/article.aspx?p=1237978

4 "SOA and Newton's Universe" by Pat Helland, http://blogs.msdn.com/pathelland/archive/2007/05/20/soa-and-newton-s-universe.aspx

5 "Memories, Guesses and Apologies" by Pat Helland, http://blogs.msdn.com/pathelland/archive/2007/05/15/memories-guesses-and-apologies.aspx

6 "Web Servicres Security Checklist" by Gunnar Peterson, http://arctecgroup.net/pdf/WebServicesSecurityChecklist.pdf

Links for 2008-09-10 [del.icio.us]

Wed, 10 Sep 2008 20:00:00 +0000

Paul Melson's Blog: ArcSight User Conference 2008
* Logger 3.0 has adopted a more-ESM-like boolean filter interface. Big improvement over the chained-regex search in 2.5 and earlier. * Demo of Logger 3.0 shows that searches of data (no details on data set) are roughly 80x faster than a similar sized search on 2.5. (The claim is 100x faster, but I counted. Still, that's a significant improvement.) * Hugh has hinted that the slick, high-performance append-only storage stuff that Logger has is going to be integrated into ESM is some release beyond 4.5. That could mean the end of the Oracle / PartitionArchiver storage model.
Splunk Tames the Chaos Brought on by Virtualization : VMblog.com - Virtualization Technology News and Information for Everyone
Existing system management tools were not designed to handle the dynamic nature of virtualization. The Splunk for VMWare Management application includes a VMWare API for data input, over 25 pre-defined searches, alerts, and reports and dashboards specifically designed to monitor key metrics for the VMWare Virtual Infrastructure.
Dorian Software BLOG: Why Your HR Department Will Love Windows Vista, Even If Your IT Department Doesn't.
Event ID 4802 tracks whenever the screensaver is invoked after a group policy-determined idle time. Event ID 4803 tracks whenever the screensaver is dismissed by the logged-on user.
Moderately Idiotic Competitor
But the clever inside criminal is taking all the payroll data from the system that is either off the network or is temporarily down. When the machine comes back up, there is no record of the intrusion and the traditional "inside out" log management system tells the user there is no problem.
Last In - First Out: Presumed Hostile - Your Application is Out to Get You
Help - Eclipse SDK - Working with the Log4J Logging sample
Cartoon 2 from The Data Governance Institute ROI
Eccentric Engineer » Blog Archive » Conf Call Hem and Haw
It’s just a damned centralized-logging platform. Unix sysadmins have been doing those for years. This stuff is about as basic as tying your shoes. All this fluff seems like overkill…but it’s IT…and we have policies.
(ISC)2 Blog: Security metrics: more is not better
Are you Owned? | Roer.Com Information Security Blog
# list of all your profiles online, with your log in. # list of all your IM/e-mail and other communication tools, with log in # list of other sites/tools that requires you to log on. # The lists above should also include each sites URL or contact information for changing passwords, or in worst case shutting them down. # a friends-list who you trust, and who are willing to help you get back your own life online. The purpose is to have them help you rebuild your internet presence. Make sure you agree some way for them to be certain that they are communicating with you, and not someone else.
Industry View: Web Application Security Today - Are We All Insane? - CSO Online - Security and Risk
The problem has gotten so bad that industry sources say most websites hosting malware have been hacked, Google says 1.3 percent of their search queries return malicious content, and Vint Cerf (father of the Internet) approximates that one quarter of all PCs are part of a botnet. Firewalls are not working. Antivirus/spyware is not working, nor are weekly patching, user education, SSL, or "turning off the home computer" as recommended by the FBI cyber-crime website. In what has become an inside joke, every authority says to use these "best-practices" despite their ineffectiveness.
TaoSecurity: Schneier Agrees: Security ROI is "Mostly Bunk"

Rootkit Evolution

Mon, 01 Sep 2008 05:21:06 +0000

I saw my first rootkit in 2004, when I was still a rookie virus analyst. At that point I had some vague knowledge of UNIX-based rootkits. One day I stumbled on an executable for Windows that didnt se...

Network skill level gap is growing, but growth opportunities abound!

Mon, 25 Aug 2008 17:06:07 +0000

A recent IDC report sponsored by the Cisco Learning Institute reveals a huge networking skills gap is emerging in North America, which spells trouble for enterprises. Listen to this: “600,000 IT workers were needed to install, configure, manage and secure networks in North America in 2007, 14% of the total IT workforce.” However, IDC reports that another 180,000 engineers with wireless as well as traditional network engineering experience will need to be added by 2011 to keep pace with advances in technology that is transforming the role of the network.

The convergence of voice and video traffic are quickly transforming the growing complexity of networks at a torrid pace. IDC estimates that the skills gap in VOIP should grow to 19% by 2011.

This changing profile in the role of the network plays a key role in the skills shortage. Network enabled collaboration tools such as social networking apps and the Webex conferencing/collaboration solutions we use in our business each and every day are demanding a new set of IT skills to deliver business value.

My perspective is two-fold on this issue; the first is what I have seen in the resources we have attempted to hire! We give a very straightforward quick written/oral test to all new technical hires. This requires basic networking knowledge and some Unix commands. On average, (after filters from reputable recruiting firms, some with 5-10 years experience) less than 10% pass muster for the first filter we use in our hiring process. This is a troubling fact, which has cost us considerable time and effort to secure the right resources with competent skills. So I can say from our market assessment in a very strong technological job skills market, core Unix and networking foundation skills are slipping.

The second is that we as an IT Operations Management (ITOM) industry need to keep pushing hard to build better proactive and intuitive solutions to aggregate instrumentation from all Data Center tools, including more work around VOIP, video streaming, and collaboration so that we can ease this transition. If ITOM solutions become more proactive across the typical Cisco infrastructure that is commonly installed in the Data Center, we can free up some additional time for advanced “emerging technologies” training where existing IT workers can enhance their core skills and re-invigorate their careers. We have to do a much better job of getting our existing IT professionals trained on emerging technologies!

While there’s less that ScienceLogic can do around training, we certainly strive to do our part to enhance a day in the life of the networking engineers who use our solutions to simplify monitoring of increasingly complex networking, Wireless, VOIP, and collaboration needs.

Anton Security Tip of the Day #16: Virtually There - Journey Into VMWare ESX Log Analysis

Mon, 25 Aug 2008 08:11:00 +0000

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root." Notice the user name of the user who switched to root.

May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

[2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

May 25 07:03:48 esx1 sudo: jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

Technorati tags: security, tips, logging, log management

The Bot Hunter: An Event Processing Challenge

Fri, 15 Aug 2008 05:35:00 +0000

Recently we penned The Attack of the Spiders from the Clouds where we mentioned how cloud computing infrastructures can be used to stage malicous or accidential network attacks.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites. I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP. There will be no physical access to the server. No exceptions.
Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example, 10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
It should send an email to one or more email addresses with a message, for example: “New Bot Detected - Confidence 8″ with IP address, etc. in the message. Another example would be an email, “Bot Blocked” - with details, etc.
You cannot automatically block any traffic that is not a bot. Blocking one “non-bot” results in failure, no exceptions.
The Prize: The winner will get their logo (w/link) on this site in a block called “Bot Hunter Winner” (or something like that.)

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative! Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone. So, you will need to work with both long and short time windows. Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

The Attack of the Spiders from the Clouds

Thu, 31 Jul 2008 11:09:19 +0000

We have seen a lot of discussions of cloud computing in the news recently, as a technology to permit “users to access technology-enabled serviceswithout knowledge of, expertise with, nor control over the technology infrastructure that supports them.” This sound great doesn’t it?! Users with little to no IT expertise can log into the cloud and launch 8 instances of a server with the equivalence of 16 high performance CPU cores. However, as we all know, all things, including cool technologies have the potential for both good and evil, opportunity or threat; and cloud computing is no different.

It just so happens that I have been experimenting with Amazon Elastic Computing Services (EC2), documented in Computing in the Clouds with AWS over at The CEP Blog. The server over at The UNIX and Linux Forums has been experiencing some very hardware-limited, high load averages recently. We thought we should take a look at moving the forum server up to the clouds.

Then, a fellow system admin over at the forums suggested that maybe some rogue bots were causing high server loads; so I wrote a one-line command to do a bit of real-time spider hunting in the Apache2 logfiles. Surprise! I found there were a number of rogue, hungry spiders that would not follow our robots.txt directive not to crawl the site. One of the bots was from Russia, one was from China, and another one was from Korea. There were spiders from places I never heard of, all consuming precious resources and denying our users!

So, I did what any Linux admin would do. I used iptables to block the networks of these rogue, hungry, spiders (sorry I was not very kind to these cyber creatures). It probally comes to no surprise at this point in the story that four of the spiders were from the Amazon EC2 cloud. Here is a sample of the output from iptables -L:

root@www:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all — ec2-67-202-45-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-243-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-197-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-213-0.compute-1.amazonaws.com/24

Well, imagine a not-so-distant future dystopian world where criminals or terrorists want to launch a massive denial-of-service attack against some critical infrastructure, like the root DNS servers, or an attack against major financial institutions, military or e-commerce sites.

First, the bad guys create an instance of powerful operating system with a malicious network application, they test it, and they place it the cloud (without invoking the instance, paying a very small storage fee, no computing time fee) and they wait. Then, at the precise moment of their planned attack, they launch 128 instances each with the equivalence of whatever is the mega-platform at the time, and just blast away at their attack target(s). Even more damaging, they do this from many cloud computing infrastructures. (Note: The cost of the attack is minimal because the criminals are only charged a few pennies an hour for each running instance and the attack runs an hour or two.)

My experience with cloud computing, which is still maturing, is that cloud computing has great promise for both good and evil. The very real example of the “spiders from the clouds” is a harmless enough story of folks using a cloud computing infrastructure for web crawling, perhaps hoping to be the next Google billionaires.

One the other hand, cloud computing brings with it an emerging and growing danger for the misuse of the power of cloud computing infrastructures. The misuse could be malicious, or accidental, but never-the-less, the danger is real.

What an interesting world we have created! Would would have ever dreamed 10 years ago that we could be attacked by ……

#include

…. Spiders from the Clouds.

Reprinted by permission from The Attack of the Spiders from the Clouds by Tim Bass, CISSP

Computing in the Clouds with AWS

Fri, 25 Jul 2008 05:34:14 +0000

The admin team at The UNIX Forums have been considering moving the UNIX and Linux Forums to the clouds - the Amazon Web Services (AWS) cloud. Amazon EC2 is one option to scale the forums, which is a LAMP application.

Amazon EC2 allows us to rent dedicated servers (instances) on-demand to run applications, such as the forums. Then we can run and host on EC2 any Linux application; but unlike classic hosting where folks install your application and set up your server for you, Amazon Web Services provide only the infrastructure.

Here are some links about AWS:

Maybe you will elevate your event processing application to the clouds?

Developer fixes 33-year-old Unix bug

Wed, 09 Jul 2008 20:00:00 +0000

An OpenBSD developer has discovered and fixed a bug in the software that has been traced back to an AT&T version of Unix from 1975.

Virtualisation - Welcome Back to the 90s.

Thu, 03 Jul 2008 02:37:00 +0000

I've been thinking about this for a while but this blog post by Pascal Meunier pretty much sums up my feelings about Virtualisation.

Back in the 90s when the Internet was new-ish and just becoming important all the machines running it were Unix boxes. (Maybe not all, but most). And a 386 would typically run DNS, sendmail, telnet (shell accounts), ftp and apache. All on the same box.

Security wasn't so tight in those days but it was usually good enough and the box could happily do what it needed to do.

Along came Microsoft and produced the idea of "one box - one service". You can't seriously consider running your domain controller as a file server. What are you thinking? And to put mail on the same box? No way. In fact, your SQL server is running under significant load, chain a few together.

And companies would buy into this concept. Microsoft were happy - more licenses. All the PC guys were happy too - more money. More complexity - more jobs.

Essentially what has happened now is that Moores Law has kicked in and has caught up with the complexity of Microsoft's software to the point where one server box can run multiple applications on it. Imagine that. But Microsoft has planted the one-service-one-box concept so well that it is now part of IT law. File server and mail server on one box? But wait...whats this button over here....? Vir-vir-virtualisation.

And now we have the tools to allow us to once again run multiple applications on one server without having to admit that one-application-one-server never made sense.

To be fair - Virtualisation does have other advantages - running multiple Operating Systems for example, being able to easily move a virtual machine from one box to another (without configuration issues), being able to make a snapshot backup of a system.

But running multiple applications on one box is not a huge win.