The convergence of voice and video traffic are quickly transforming the growing complexity of networks at a torrid pace. IDC estimates that the skills gap in VOIP should grow to 19% by 2011.

This changing profile in the role of the network plays a key role in the skills shortage. Network enabled collaboration tools such as social networking apps and the Webex conferencing/collaboration solutions we use in our business each and every day are demanding a new set of IT skills to deliver business value.

My perspective is two-fold on this issue; the first is what I have seen in the resources we have attempted to hire! We give a very straightforward quick written/oral test to all new technical hires. This requires basic networking knowledge and some Unix commands. On average, (after filters from reputable recruiting firms, some with 5-10 years experience) less than 10% pass muster for the first filter we use in our hiring process. This is a troubling fact, which has cost us considerable time and effort to secure the right resources with competent skills. So I can say from our market assessment in a very strong technological job skills market, core Unix and networking foundation skills are slipping.

The second is that we as an IT Operations Management (ITOM) industry need to keep pushing hard to build better proactive and intuitive solutions to aggregate instrumentation from all Data Center tools, including more work around VOIP, video streaming, and collaboration so that we can ease this transition. If ITOM solutions become more proactive across the typical Cisco infrastructure that is commonly installed in the Data Center, we can free up some additional time for advanced “emerging technologies” training where existing IT workers can enhance their core skills and re-invigorate their careers. We have to do a much better job of getting our existing IT professionals trained on emerging technologies!

While there’s less that ScienceLogic can do around training, we certainly strive to do our part to enhance a day in the life of the networking engineers who use our solutions to simplify monitoring of increasingly complex networking, Wireless, VOIP, and collaboration needs.

Some of you may know I’ve been preparing to brush up on my *nix skills. A couple of our new solutions are running on Linux platforms and I feel compelled to understand any platform I’m working with inside and out… I know, it’s a bit OCD.

But to be honest, I haven’t really touched a Linux platform for about 10 years, since I was one of the three students running the Sun network over at NCSSM. I still remember the humorous ‘root’ ‘of all evil’ admin name that we used and the password, iaceo (in mixed caps), which was a Latin word for (I think) to lie dead. (Please correct me if you know what it means).  When you’re 17, these things are amusing.

I’ve kept my ls-ing and cd-ing over the years, but will be brushing up on the grep-ing and tail-ing ;)

So with any system, I think we all have our favourite commands that we use daily and are part of our daily arsenal. I’m working out mine but wanted to hear from you…

What are your 3 favorite Linux commands?

And is there 1 obscure one you really love (or hate)?

# # #

“Amazon EC2 public AMIs (Amazon Machine Image) generate unique SSH (Secure Shell) host keys each time you launch an instance. This enables you to get the host SSH keys from the console output and verify the host to which you are connecting.”

Important note: SSH host keys enable clients to verify the server identity (”are you really my server?”) and are separate from SSH user keys that allow the user to prove their identity to the server (”he really is Jeff”).

What does this mean?

It means that EC2 instances created from a public AMI after June 23rd have unique SSH host keys and thus are not vulnerable to a man in the middle attack against the SSH protocol, but only *if* you manually verify the host SSH key during your initial SSH connection.

OK, but I created my AMI before June 23rd - am I vulnerable?

According to Amazon, yes.  Every EC2 instance copied from a public AMI will have the same SSH host keys as the original AMI.  The only exception to this is if the original AMI creator spotted this problem and used a hook to force SSH host key regeneration upon first boot. This means that an attacker who say, uses a DNS cache poisoning attack, can intercept the communication between your SSH client and your AMI.

How can I fix my pre-June 23rd AMIs?

Regenerate the SSH host key.  The exact commands will depend on your operating system (hint: ssh-keygen).

Who is to blame?

Either the creators of the original AMI or Amazon - depends how you look at it.  If Amazon created the public AMI then it could be argued they are responsible.  However, anyone can submit a public AMI and Amazon makes no guarantee they are fit for use (Amazon do review the AMI listing according to their documentation).

Amazon can in fact make the argument they are acting in the interests of their users by implementing a shared solution to key regeneration (rather than requiring each user to manually regenerate the ssh host keys after booting an image).   That’s fine going forward but what of potential exposure to customers using the pre-June 23rd public AMI copies?

Just to be clear, its not the fault of SSH - ’secure channels’ require proper key management and the need for unique host keys is well documented.

Are there any mitigating factors?

Yes, if you have used security groups to limit SSH access to your AMI from IP ranges you trust (rather than the entire Internet).  You’ll still want to regenerate the ssh host keys sooner than later.

Is the Amazon environment vulnerable to Man-in-the-middle attacks?

I don’t know.  But that isn’t the real question - is the path between you and your AMI immune to MITM attacks and the answer is most definitely no.  If SSH on your AMI is only accessible from another AMI then its a fair question but its unlikely Amazon are going to show you their network diagrams ;-).  From experience performing MITM attacks, I would assume most networks are vulnerable (one of the reasons why we use SSH).

Why Didn’t Amazon Tell Me I’m Vulnerable?  They know from their logs what AMIs I use!

Didn’t they?  Whoops - naughty Amazon :P.

But seriously, Amazon are not responsible for the configuration of the public AMIs you use.  Its important not to confuse the AMI selection and cloning mechanism that Amazon provides, with the content of an AMI itself.

Does Amazon have a mailing list for customers to learn about new security problems (even if its not Amazon’s fault).

Not that I know of.   Right now you have to search forum posts and monitor documentation updates - which is time consuming and makes it easy to miss something.  I also can’t find an area on the AWS website where they collect security related items together (e.g. best practices, advisories, key management).   In my view, this is a shame as it probably undermines the effort that Amazon are putting into their security  (for some customers, if they don’t “see it”, it doesn’t “exist”).

A ‘Security’ link on the main AWS homepage pointing to those resources would go a long way to improving the visibility of the AWS security related information.

For June, we have an 802.1X hat-trick to blame for my slack blogging habits. Over the past few weeks, I’ve had back-to-back 802.1X implementations, one wired, one wireless and one with both. Two government customers and one commercial, not in that order. And I even did one semi-training-slash-semi-implementation-quick-start for another customer.

It’s been fun, but 1X is always challenging. The variety of components, the nature of the interactions and the ‘newness’ of actual implementations make it difficult to work from any type of cookbook or implementation guide. There are just too many variables.

When will it be easier? I think as 1X is more widely implemented in the real world, customers will become more familiar with the concepts and integrators will have more experience to make it go smoothly. For now, everyone has to just take it one step at a time and address issues as they arise. And, for now, I’ll enjoy the job security that 1X offers ;)

Luckily, I’ve had the opportunity to work with a variety of customers and a variety of environments and equipment while hammering out 802.1X. The experience and exposure has certainly given me a unique insight into the issues, complications and solutions that come along with a 1X project.

At present, I think we’ve successfully configured 1X on about a dozen different types of equipment, both switches and wireless APs and controllers, from a variety of vendors. It may not sound like much, but in the world of 1X, that’s quite a variety when you consider each manufacturer has their own ‘system’ for configuring 1X and the commands and procedures can vary greatly even from product-to-product from the same vendor.

Is the 1X streak over? Not at all. We have several customers with NAC and 802.1X projects that we had to queue up for after June 30. I’ll keep you posted!

# # #