We are regularly hearing from our security clients about their difficulties finding people with the right skills – or when they do finally find them, these people are too costly to employ because their skills are in such demand.

Indeed, the “unavailability of people with the right skills” was cited as a top challenge for security groups in both our enterprise and SMB surveys.

In comparing need for talent across 25 different IT roles, Forrester analysts came to the conclusion that information security experts are among the hottest roles in IT, sharing the top spot with information/data architects.

The skills shortage is likely to get worse before it gets better. We’re unlikely to see a significant spike in security experts’ salaries to attract those we need to hire: large changes in compensation for senior security personnel would run against the current of economic belt-tightening. Another typical approach to offsetting the shortage would be to train up: foster the career development and advancement of existing security personnel on our payroll. However, with all the outsourcing that is going on – and which will increasingly occur – there is a shrinking pool from which to find people with “the right stuff” worth championing their advancement.

We could look outside of security to others in IT, or even to co-workers in other departments or business groups. But given how poor a job IT Security does of marketing its value proposition, I don’t hold much hope for attracting non-security people.

What do you think? Are we about to hit a very big wall when it comes to skills and staffing? Are you presently feeling the pain of a skills shortage? Do you see such a shortage looming? What measures are you taking to acquire and nurture talent? Which ones are successful and why?

I welcome your thoughts on the topic.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.   I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

1. Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.  There will be no physical access to the server. No exceptions.
2. Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
3. Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
4. Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
5. Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,  10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
6. Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
7. If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
8. It should send an email to one or more email addresses with a message, for example:  “New Bot Detected - Confidence 8″ with IP address, etc. in the message.  Another example would be an email, “Bot Blocked” - with details, etc.
9. You cannot automatically block any traffic that is not a bot.  Blocking one “non-bot” results in failure, no exceptions.
10. The Prize:  The winner will get their logo (w/link) on this site in a block called “Bot Hunter Winner” (or something like that.)

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!  Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.   So, you will need to work with both long and short time windows.  Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

LAS VEGAS -- Last weekend, more than 9,000 hackers, freaks, feds and geeks gathered for the 16th annual DefCon, the world's largest computer security convention.

Wired.com brought you live coverage of the most newsworthy events at DefCon 16. Here are some photos from the lighter side of the conference.

Left: South Korean hackers compete in the Capture the Flag competition. The goal is to hack into and keep control of targeted servers.

Mr. Sinister and Dragon Cracker battle it out in a round of Guitar Hero -- one of DefCon's newest competitions.

Bringing-your-own-booze supply ensures optimal buzz at DefCon. Shortly after this picture was taken, hotel security escorted this backpack-hacker to his room.

Computer geeks from the National Institute of Standards and Technology set up a network secured with quantum encryption in a conference room at DefCon. The quantum-entangled photons are being used to encrypt a video stream across a line-of-site network.

A compact optical bench and an atomic clock (left) are used to secure a network with quantum encryption.

In the Lock Pick Pavilion, DefCon attendees Dustin, Jennalynn and Kunfoozball practice their lock-picking skills.

DefCon founder and organizer Jeff Moss, aka Dark Tangent, at the conference's closing ceremony Sunday.

A collection of black badges awaits the winners of the various competitions. These badges give their holders lifetime entry to DefCon.

One of DefCon's logos, the smiley-faced skull and crossbones, is welded inside a yellow sphere. The sphere is the primary stage of one of the most difficult competitions at DefCon: The Mystery Challenge.

Unbeknownst to attendees, this laptop is sniffing RFID tags and taking photos of their owners when they pass in front of the detectors. RFID tags are used in everything from building access to some credit cards.

At the closing ceremony, DefCon organizers turn off the lights while the attendees wave their high-tech badges back and forth.

Eye-Fi raises $11m in second funding round: I don't cover companies' financial dealings often, but Eye-Fi is always worth highlighting, as they appear to be the only smart entrant in the entire universe of cameras-with-Wi-Fi, and they're not even a camera maker. Camera makers have typically limited or straitjacked the onboard Wi-Fi. Eye-Fi's now three models of SD cards with Wi-Fi built in have a pretty wide range of controls and abilities. I tested out the Eye-Fi Explore recently, which pairs Wi-Fi GPS-like positioning from Skyhook with Wayport hotspot access, and the review appears in Saturday's Seattle Times. Eye-Fi's biggest challenge is better camera integration, so that cameras can handle power management in discussion with the card; camera makers have to not feel threatened by Eye-Fi's smart technology, though.

• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, let me bet my ass that "Not knowing what to log" problem will be licked in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what  the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, for esoteric log sources (and custom applications) might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?"  for reference)

Next, the problem of "Log volume" will  definitely get worse, much worse.  One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is NOT a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL (!!!) get since standards are being developed. We will see people struggling with all sorts bizarro log data in the coming years. Virtualization, web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into dark voodooistic experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. BTW, watch my blog for some uber-fun info on that!

"Getting the logs"  has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing  files remotely via secure protocols made application log collection easier (syslog-NG with TCP transfer and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services (thanks Eric!). However, in the future it might resurface as we try to collect logs from "weird" places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them (Yo, my t-shirt-making colleagues... :-))

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in  an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and cursing under ones's breath. Only then we can attempt to reliable handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so."  For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. My next poll will address that :-)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?

Blogger: Pete Lindstrom

On July 8th, Dan Kaminsky of IOActive announced a major DNS “vulnerability” in conjunction with a number of major DNS vendors. The announcement was off the charts in fanfare and attention, but what was the real impact on risk?

First, it is worth noting that this “bug” is more properly classified as a new attack technique invented by Dan. It combines two vulnerabilities that have been well-known for some time – the ability to guess non-random transaction IDs and the use of Additional RRs to insert new entries into the DNS cache. A fix against either of these vulnerabilities also negates the attack itself.

The fundamental question that determines the risk impact revolves around whether it is reasonable to expect fewer or more incidents that use this technique when comparing the period prior to disclosure -- or, more properly, before the date of Dan’s invention of the technique (this also assumes prior art) – with the period after invention/disclosure and into the future. If the disclosure reduces the number of those incidents, then risk is reduced; if the disclosure increases the number of those incidents, then risk is increased.

With that litmus test as our guideline, it is useful to break down the functional elements of risk and look at the impact on threats, vulnerabilities, and consequences (we will cover consequences, then vulnerabilities, and finally threat).

Consequences
Though the consequences are the same before and after disclosure, it is worth discussing the impact here, given that the implication was that the “entire web” could be taken down. The nature of the attack requires the following:

1. An attacker must convince/trick a user into making a DNS request for a domain that doesn’t already exist in their DNS server’s cache. The expectation here is that s/he can be easily tricked into doing this.
2. Then, the attacker must simultaneously attack the DNS server by guessing the transaction ID. According to Kaminsky, the request/attack phase can be done reliably in about 10 seconds.
3. The attack is DNS server-specific. Only users on the same DNS server are affected.
4. Propagation: once the cache is poisoned, anyone requesting that domain will be routed to a malicious server.

Without combining this attack with other attack techniques, there can be three results:

1. Spoofing of a single website for multiple, perhaps many, users using the same DNS server. Presumably, this would be followed by more traditional phishing and malware attacks.
2. Denial-of-service by rerouting traffic from a legitimate site thereby taking potential customers or “eyeballs” away.
3. Denial-of-service be rerouting traffic from a legitimate high volume site to a legitimate low-volume site thereby overloading the servers on the low-volume site.

Because of the point-to-point (user-to-website) nature of the attack, to do something that constitutes “taking over the entire web” is infeasible by a longshot.

The bottom line analysis for the effect on risk due to a change in consequences from pre-invention to post-invention: no change, and therefore no impact.

Vulnerabilities
These vulnerabilities have existed for years, and there have been workarounds for years. Along with this announcement, new patches were introduced in all major DNS server solutions. It is reasonable to assume that many DNS server implementations have been patched, though public accounts have suggested that number is in the 66%-75% range.

Bottom line analysis: the vulnerability level has been reduced, probably significantly, and the affect is positive for risk reduction. If 100% of DNS servers were patched, then overall risk would be reduced for this attack (assuming that there were actual attacks using this technique in the past.)

Threats
The real question regarding risk impact comes in the arena of the less-controllable manipulation of threat. The general threat equation revolves around an attacker’s willingness to attack, based on his/her own cost/benefit analysis that compares the cost to attack to the expected benefits, tempered by the potential for being caught and penalized.

Cost to attack – prior to disclosing the invention, there were likely few, if any attackers with “prior art” that mirrored this technique. It is anybody’s guess how many potential attackers might have figured it out eventually, but they would have had to come from the pool of folks with enough expertise to do so – I am going to guess 500,000 people.

After the disclosure, the hints provided in the press release, the podcast, the sorted stories, and the blog entries made it much easier to figure out. Let’s guess that 5 million people could execute the attack. With automated tools, that number goes up to 50 million.

These numbers are estimates that illustrate the nature of the exercise. You are welcome to fill in your own estimates and come to your own conclusions.

Bottom line analysis: a significant increase in threat and corresponding risk.

Net Effect
The risk manager's challenge is to weigh the decrease in vulnerable systems compared with the corresponding increase in threat, within the context of number of incidents and anticipated future incidents. Given the sheer size differential, it is difficult to conceive of a situation where risk is not increased.

Sometimes it "feels" like someone is taking action for the greater good, when that action actually creates a negative impact for all. For example, it is common for people to believe that raising prices of scarce resources during  times of trouble (e.g. gasoline in the hurricane Katrina aftermath) is unconscionable even though a majority of economists recognize that raising prices actually provides for the greater public good. Vulnerability discovery and disclosure, and attack inventions, might feel like the right thing to do, but the net result is almost always a negative impact.