So I was very interested when I heard that Opus Interactive, a customer of ours, had “joined the VMware vCloud initiative as a VMware Service Provider”. I talked to Eric Hulbert, CTO of Opus Interactive, to get some details directly from the source.

Eric shared our own caution about making “cloud-ready” announcements. There have simply been too many companies talking about cloud solutions that lack any substance – usually based on definitions of cloud computing that are hazy or just too broad. The backlash against the cloud hype is often quite justified. But in Opus’ case, there are real components that if they don’t add up to a “full” cloud computing solution just yet, are well on their way – and enabled by VMware’s program for service providers (VSPP).

Opus Interactive is serious about virtualization, which is an indispensable tool in their stated goal of creating a high-density micro-data center with the smallest footprint possible. They are 100% wind-powered and have already virtualized much of their data center, reducing the amount of hardware necessary to run the business and driving down costs to produce even more competitive advantage in a crowded marketplace.

VSPP for vCloud provides a rental model of VMware licenses – e.g., for Enterprise ESX or VDI. VMware Service Providers report on their customers’ virtual machines (vm) and pay only for what is actually used. This model lets Opus Interactive quickly spin up a vm to get a new customer up and running in about an hour and stay very cost competitive at the same time; Opus offers their vClustr entry-level virtual server for only $99.

Cost-effective, rapidly scalable computing “on-demand” based on shared resources, managed by “expert” third-parties, enabled by virtualization technology and pay-per-use vm licenses. Cloud computing? Instead of thinking about a single definition of cloud computing, perhaps it’s more relevant as the market matures to think about a continuum of cloud computing. And by that definition, Opus Interactive is providing cloud services, enabled by VMware’s VSP program. Next on the schedule, automated provisioning and perhaps in the future, API’s that make it even easier for application developers to test and deploy apps on Opus Interactive’s cloud platform – which, by the way, uses EM7 for its core management solution.

There is another reason why the article got my attention; my manager, Steve Lipner, is one of the few people to have designed and built a TCSEC A1 assured system and lived to tell the tale. None were sold, but they built one!

The NSA-directed project, the Tokeneer ID Station (TIS), involved building a low-defect system that conforms "to the Common Criteria requirements for Evaluation Assurance Level 5 (EAL5)" in a "cost effective manner." I'm all for this, because building high-assurance solutions is not cheap.

There's a paper with more technical detail about the project that is worth a read.

In my opinion, the project is only a science project, an experiment, for the following reasons:

• It's tiny. Weighing in at a little under 10 KLOC.
• It's only a very small portion of a much larger solution which has not been developed using the same rigor. This bit of context makes the solution as a whole moot. Call me cynical, but my question is "can the entire solution be built with same rigor in a ‘cost effective manner'?" Perhaps it can, but that is not what is presented.
• It sits on top of many operating systems (Windows, Mac OS X and Linux) that are not EAL5 certified. So it would be a little like having an EAL5 certified CharMap application running on EAL4 Windows Vista.
• It's written in a subset of Ada called SPARK, and SPARK skills are not common in the marketplace. Interestingly, SPARK makes use of annotations to help drive the static analysis process. While not a total analog, we also recommend Microsoft development teams use annotations (SAL) to help drive the required static analysis process.
• The application has a large number of dependencies that are not part of the project:

In the SDL we call these files ‘giblets' because they are components needed for your application to operate, but they do not belong to your team. Some of the files look old and highly vulnerable, such as libpng-3.dll from 2001! OSVDB lists 23 vulnerabilities since 2002 in libpng!

In summary, the TIS project is very interesting to a small number of important but specialized customers, such as the NSA, for whom this kind of research is critical. I too found it interesting, but the process is far from a set of "secrets to writing secure code" and the tools are certainly not within reach of day-to-day applications and not applicable to developing complete solutions.

• The four vendors fixed a total 585 vulnerabilities in 1H08. 26.8% affected multiple vendors and of those, only 8 were fixed on the same day – the rest had an average 35 day delay between the first available fix and the last available fix..
• Microsoft had the lowest average Days of Risk for all vulnerabilities fixed at 24.22 days, with the next closest vendor at 72 days.
• For desktop OS vulnerabilities, Windows Vista had the fewest vulnerabilities in 1H08 at 21. The next lowest number was Windows XP SP2 at 26.
• Windows Vista customers experienced full or partial mitigation for 46% of the 26 vulnerabilities affecting Windows XP SP2 in 1H08, but also experienced one additional vulnerability in new code.

In addition to these measurements for the vendors and products, the body of the report also provides weighted analysis which provides a lesser consideration for lower severity issues. Please read the full report for details.

We have been conducting polls of our customers and of IT professionals at technology trade shows for the past two years and the results are in: Pulling together all of the management pieces and processes is even more crucial in a virtualized environment.

So what does this mean for you? You will need to refine your incident and problem management processes with new technologies in order to reduce downtime and maintain end user performance. But of course even the most basic technologies are not well integrated even in today’s world.

I recently participated in a Gartner Conference and watched to my amazement a real-time electronic survey of the audience. To my disbelief, the audience, filled with 300+ people from Fortune 2000 companies provided real-time responses to the question:

What level of integration does your IT org have between event management and service desk applications?

• None: 10%
• Manual Phone call from IT ops to IT service desk staff member: 46%
• Manual click button on event manager to open trouble ticket: 20%
• Automated event management system automatically opens trouble ticket without requiring human oversight or approval: 24%

Unbelievable… still very few of the survey respondents have yet to formalize problem management systems with event management systems. For 56% of the audience the process is still manual!

Another interesting real-time survey question at the Gartner Conference was:

Who in your organization is responsible for critical problem processes and resolution?

• IT Service Desk 13%
• IT Operations 49%
• Process Team 12%
• Other 9%
• Responsibility not formalized 17%

Virtualization adoption and the speed with which things change in a virtualized environment require automation and will transform Incident and Problem Management. Clearly with this new technology we are required to re-think Organizational, Behavioral and Cultural Challenges required to take advantage of the opportunities that virtualization provides.

Incident and problem management processes and metrics must bridge organizational silos that have been the norm within IT. With virtualization, people have to work more closely together in the different silos than ever before. IT leaders need to break down the walls between the technology-centric silo mentalities.

Business Imperative Action Plan:

1. What can you do today? –Understand the impact of virtualization on incident and problem mgt. workload, provide technology training for helpdesk/service desk staff.
2. What can you do in the next 12 months?

Formalize problem management processes, metrics and personnel.
Invest in tools and processes for systems on virtualized servers.
Long term: On the Radar Screen!
Instill teamwork into all groups responsible for the virtualized environment service and support. Map components and configuration items directly to end user services.

Final Thoughts: Know the management pieces and ensure that they fit together. It’s great to buy new technology, but be demanding to ensure that your vendors show you have they will help to link all these pieces together - Change, Inventory, Incident, Problem, Server, Capacity, Performance, Configuration, Event, and Integrated Workflow.

Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

Developers have the objective of building a functional application. They are focused on building more functionality into applications. Moreover, building security creates more workload  for Developers which is a disincentive and moreover, Developers are rewarded for building more functionality than building more security. I have never seen a Developer in my professional life for being rewarded for building a secure application.

Hackers are focused on how to break the application. They look for weak links in application that will enable them to access application data. Developers usually follow process to build application, but Hackers have no process and all they have is multitude of possibilities. Hackers are innovative in trying various permutations in compromising the application.

A million dollar question is whether we can build secure applications when a Developer is focused on functionality but not on breaking the application?

There is a school of thought about Inside-out security where the application is built securely from scratch. Unfortunately,  this approach won't suffice because hackers traverse Outside-in. A little reflection will highlight the importance of vulnerability scanning and penetration testing of application. This will bring the perspective of what developers do not know already.

Building a secure application inside out is not enough. In order to address unknown unknowns (or blind spots of developers), penetration testing should be done. Both whitebox style penetration testing (where components of an application is known)  and also blackbox style penetration testing which mi micks an Hacker who may not have any knowledge of the application, should be carried out.

An application of higher level of security is not built just by Developers. It is built by integrative process of Developer mindset and Hacker mindset.  This is a constant struggle for years to come.