The convergence of voice and video traffic are quickly transforming the growing complexity of networks at a torrid pace. IDC estimates that the skills gap in VOIP should grow to 19% by 2011.

This changing profile in the role of the network plays a key role in the skills shortage. Network enabled collaboration tools such as social networking apps and the Webex conferencing/collaboration solutions we use in our business each and every day are demanding a new set of IT skills to deliver business value.

My perspective is two-fold on this issue; the first is what I have seen in the resources we have attempted to hire! We give a very straightforward quick written/oral test to all new technical hires. This requires basic networking knowledge and some Unix commands. On average, (after filters from reputable recruiting firms, some with 5-10 years experience) less than 10% pass muster for the first filter we use in our hiring process. This is a troubling fact, which has cost us considerable time and effort to secure the right resources with competent skills. So I can say from our market assessment in a very strong technological job skills market, core Unix and networking foundation skills are slipping.

The second is that we as an IT Operations Management (ITOM) industry need to keep pushing hard to build better proactive and intuitive solutions to aggregate instrumentation from all Data Center tools, including more work around VOIP, video streaming, and collaboration so that we can ease this transition. If ITOM solutions become more proactive across the typical Cisco infrastructure that is commonly installed in the Data Center, we can free up some additional time for advanced “emerging technologies” training where existing IT workers can enhance their core skills and re-invigorate their careers. We have to do a much better job of getting our existing IT professionals trained on emerging technologies!

While there’s less that ScienceLogic can do around training, we certainly strive to do our part to enhance a day in the life of the networking engineers who use our solutions to simplify monitoring of increasingly complex networking, Wireless, VOIP, and collaboration needs.

• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, let me bet my ass that "Not knowing what to log" problem will be licked in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what  the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, for esoteric log sources (and custom applications) might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?"  for reference)

Next, the problem of "Log volume" will  definitely get worse, much worse.  One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is NOT a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL (!!!) get since standards are being developed. We will see people struggling with all sorts bizarro log data in the coming years. Virtualization, web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into dark voodooistic experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. BTW, watch my blog for some uber-fun info on that!

"Getting the logs"  has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing  files remotely via secure protocols made application log collection easier (syslog-NG with TCP transfer and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services (thanks Eric!). However, in the future it might resurface as we try to collect logs from "weird" places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them (Yo, my t-shirt-making colleagues... :-))

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in  an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and cursing under ones's breath. Only then we can attempt to reliable handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so."  For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. My next poll will address that :-)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?

Depending on who you talk to, he is:

a) a hero

b) a disgruntled worker

c) in need of a serious work/life adjustment

d) in need of $5 million and/or a better lawyer

e) all of the above

Surprisingly strong opinions, regardless of what you choose.

We chose to lighten things up a bit and, as we always try to do, figure out how to help our customers be proactive. So here it is, the Top 10 Signs Your Network Admin has Gone Rogue:

10) David Letterman has a Top 10 list called “Top 10 Signs Your Network Admin Has Gone Rogue”

9) Your Admin is the only one with the network device log-ins and refuses to share them with anyone else.

‘8) His presentations about network configuration include the words “Magic” and “Burn after reading”.

7) Instead of email, he forces everyone to use the Suggestion box placed outside of his door…and then places a very obvious nanny-cam hidden in a teddy bear right next to it.

6) He begins to grow out his sideburns and every question directed to him in meetings results in the same response, “Do you feel lucky today, punk?”

5) He has the mayor on speed-dial.

4) He starts wearing very big shoes to the office and accosts random people in the hallways asking if they think they could fill them.

3) He refuses to write router and switch configs to flash citing network security concerns.

2) He calls you and asks for a $5 million salary advance; caller id flashes “Department of Corrections”.

And #1: You’re the City of San Francisco

Enjoy your lock-down free weekend!

ShareThis

Like most of the service providers we talk to, they look to virtualization to provide immediate benefits to the business – e.g, cost savings from server consolidation and support for Green IT through cutting power/cooling requirements. And one more dimension to virtualization – Opus launched a new service, vClustr, which is a virtual dedicated server that provides the benefits of a fully managed dedicated server at a fraction of the cost…managed by EM7, of course.

We were happy to help Opus by working with them to implement our EM7 solution. Their growth plan was severely limited by inefficient processes and tools. As Opus grew rapidly in 2006, the tools they had in place were not easy to integrate as they were managed independently. There was a manual billing and ticketing infrastructure in place, and valuable engineer time was spent on maintaining what they had instead of enabling business growth. The company faced a choice, either grow by adding overhead and bodies or grow through automation.

Opus chose automation. They needed an automated solution to cover their immediate needs, and also enable them to scale processes for emerging technologies and future service offerings. Throughout their growth, Opus wanted to maintain their “customer first” philosophy and expand their green efforts.

By choosing EM7, Opus was able to replace their multiple, disparate tools with a single, integrated management system for networks, servers, applications, service desk assets and virtualization infrastructure. EM7 provided automated billing, ticketing, alerts and escalation options as well as a branded customer portal for transparency and self-service ticketing.

The results were tremendous. Opus Interactive recouped $130k per year of engineering resources. They automated critical operations to increase efficiency, enabled proactive monitoring and prepared for growth, while giving the business the processes and tools to grow the business without additional human capital resources.

We’re glad that we could help such a great company achieve their goals of providing an efficient “best-in-class” solution that combined superior customer service with a green philosophy.

Get the entire case study here.

ShareThis