Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

I plan to blog about my progress (and share it) if there's not already a built-in solution out there.

The announcement is of course Good News — because once the PCeU is up and running next Spring, it should plug (to the limited extent that £2 million a year can plug) the “level 2″ eCrime gap that I’ve written about before. viz: that SOCA tackles “serious and organised crime” (level 3), your local police force tackles local villains (level 1), but if criminals operate outside their force’s area — and on the Internet this is more likely than not — yet they don’t meet SOCA’s threshold, then who is there to deal with them?

In particular, the PCeU is envisaged to be the unit that deals with the intelligence packages coming from the City of London Fraud Squad’s new online Fraud Reporting website (once intended to launch in November 2008, now scheduled for Summer 2009).

Of course everyone expects the website to generate more reports of eCrime than could ever be dealt with (even with much more money), so the effectiveness of the PCeU in dealing with eCriminality will depend upon their prioritisation criteria, and how carefully they select the cases they tackle.

Nevertheless, although the news this week shows that the Home Office have finally understood the need to fund more ePolicing, I don’t think that they are thinking about the problem in a sufficiently global context.

A little history lesson might be in order to explain why.

Back in 1930’s, Bonnie and Clyde and other US bank robbers were using the new-fangled automobile to flee across state lines — creating jurisdictional problems as a result. The US solution was to make bank robbery (along with auto-theft and other related offences) into federal offences rather keeping them as state-specific infractions. In particular this meant that the FBI could provide federal level policing (tracking down and killing John Dillinger for example).

We have the same jurisdictional issues dealing with cyberspace, with criminals in one country fleecing consumers in another while using systems hosted in a third. The Convention on Cybercrime addresses part of the problem by trying to ensure international consistency where eLaws are specifically needed (which of course is only the case for small parts of eCriminality, fraud is fraud whether eEnabled or not). However, there is limited inter-jurisdictional co-ordination for eCrime investigations — for example Interpol (often incorrectly perceived to be international police force) merely keeps a large database and passes faxes from one place to another.

In practice, most cross-border investigations are done as “joint operations” and the jointness is usually very limited — one force does all the legwork and a liaison officer in the other country deals with local paperwork. There’s usually a quid pro quo element to these joint operations, for budgeting reasons if no other.

What isn’t happening, or at least only in a handful of very specialised areas, is any international co-operation in setting priorities or selecting cases to pursue. Every country is doing its own thing about eCrime, and there’s a widespread impression that any criminal who can operate from “across the state line” is essentially immune from serious investigation.

We identified this problem last year when we (Ross Anderson, Rainer Böhme, Tyler Moore and myself) wrote a report on Security Economics and the Internal Market for ENISA. It’s not an easy one to fix whilst politicians (and populaces) are unwilling to see “foreign” police officers operating in their country, and the establishment of a truly international “cyber police force” seems equally unlikely.

Our policy proposal to tackle the issue harks back to WWII’s SHAEF, which has morphed into similar arrangements within NATO. In essence liaison officers from multiple forces would sit around a single table, working with a central coordinator, to set policy and decide which investigations to pursue. They would then communicate back to their own countries, who have specifically budgeted to provide appropriate assistance. So it’s very like “joint operations”, but the scheme is multi-laterial, and has a true command and control function in the centre — who will quickly learn to shy away from politically sensitive topics and make a real impact on eCriminality.

To summarise then, a welcome to the Home Office for finally finding a small amount of funding for some country-wide ePolicing; but it’s well past time to be working on world-wide initiatives.

The problem is self-signed certificates.

A CA is not a great solution:

Using a Certification Authority (CA) could solve the attack but at the same time introduces a new set of attack vectors:

1. The CA becomes a single point of failure. It becomes the juicy/high-value target for the attacker. Single point of failures are not good. Attractive targets are not good.

   Any person with access to the CA key can undetectably fake passports. Direct attacks, virus, misplacing the key by accident (the UK government is good at this!) or bribery are just a few ways of getting the CA key.

2. The single CA would need to be trusted by all governments. This is not practical as this means that passports would no longer be a national matter.

3. Multiple CA's would not work either. Any country could use its own CA to create a valid passport of any other country. Read this sentence again: Country A can create a passport data set of Country B and sign it with Country A's CA key. The terminal will validate and display the information as data from Country B.This option also multiplies the number of 'juicy' targets. It makes it also more likely for a CA key to leak.

   Revocation lists for certificates only work when a leak/loss is detected. In most cases it will not be detected.

So what's the solution? We know that humans are good at Border Control. In the end they protected us well for the last 120 years. We also know that humans are good at pattern matching and image recognition. Humans also do an excellent job 'assessing' the person and not just the passport. Take the human part away and passport security falls apart.

If you have ever wondered how security guards can possibly keep an unfailingly vigilant watch on every single one of dozens of television monitors, each depicting a different scene, the answer seems to be (as you suspected): they can't.

Instead, they can now rely on computers to constantly analyze the patterns, sizes, speeds, angles and motion picked up by the camera and determine -- based on how they have been programmed -- whether this constitutes a possible threat. In which case, the computer alerts the security guard whose own eyes may have been momentarily diverted. Or shut.

An alarm can be raised, for instance, if the computer discerns a vehicle that has been standing still for too long (say, a van in the drop-off lane of an airport terminal) or a person who is loitering while everyone else is in motion. By the same token, it will spot the individual who is moving rapidly while everyone else is shuffling along. It can spot a package that has been left behind and identify which figure in the crowd abandoned it. Or pinpoint the individual who is moving the wrong way down a one-way corridor.

Because one person's "abnormal situation" is another person's "hot dog vendor attracting a small crowd," the computers can be programmed to discern between times of the day and days of the week.

Certainly interesting.