Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

What’s the solution? Ars Technica suggests there might not be a good one, in part because malware distributors can go so far as to hire real people to do their dirty work:

Instead of trying to build better CAPTCHA-cracking programs, the malware industry went out and got itself some humans of its own. This effectively bypasses the primary security strength of the CAPTCHA system and leaves it entirely dependent on what we’ll call secondary security characteristics. CAPTCHAs are often complex (particularly these days), which does increase the chance that they’ll be misread (and returned incorrectly), while the font and display of the characters themselves are at least somewhat unfamiliar to the CAPTCHA crackers sitting on the other side of the world.

Sometimes those captcha phrases are pretty incoherent to me too. When I post over at Craigslist sometimes it says I’ve gotten its Captcha wrong, and I end up wondering if secretly I’m a bot?? Apparently not a very smart one either.

Steps are now being taken, rather faster since Dan Kaminsky came up with a really effective DNS poisoning attack, to secure DNS by using DNSSEC.

The basic idea of DNSSEC is that when you get an answer from the DNS it will be signed by someone you trust. At some point the “trust anchor” for the system will be “.” the DNS root, but for the moment there’s just a handful of “trust anchors” one level down from that. One such anchor is the “.se” country code domain for Sweden. Additionally, Brazil (.br), Puerto Rico (.pr), and Bulgaria (.bg) have signed their zones, but that’s about it for today.

So, wishing to get some experience with the brave new world of DNSSEC, I decided that Sweden was the “in” place to be, and to purchase “cloudba.se” and roll out my first DNSSEC signed domain.

The purchase wasn’t as easy as it might have been — when you buy a domain, Sweden insists that people provide their identity numbers (albeit they have absolutely no way of checking if you’re telling the truth) — or if a company they want a VAT or registration number (which are checkable, albeit I suspect they didn’t bother). I also found that they don’t like spaces in the VAT number — which held things up for a while!

However, eventually they sent me a PGP signed email to tell me I was now the proud owner of “cloudba.se”. Unfortunately, this email wasn’t in RFC3156 PGP/MIME format (or any other format that my usually pretty capable email client understood).

The email was signed with key 0xF440EE9B which was reassuring because the .se registry gives the fingerprint for this key on their website here. Rather less reassuringly footnote (*) next to the fingerprint says “.SE signature for outgoing e-mail. (**) June 1 through August 31.” (the (**) is for a second level of footnote, which is absent — and of course it is now September).

They also enable you to fetch the key through a link on this page to their “PGP nyckel-ID” at http://subkeys.pgp.net.

Unfortunately, fetching the key shows that the signature on the email is invalid.

Since the email seems to have originated in the Windows world, but was signed on a Linux box (giving it a mixture of 0D 0A and 0A line endings), then pushed through a three year old copy of MIME-tools I suppose the failure isn’t too surprising. But strictly the invalid signature means that I shouldn’t trust the email’s contents at all — because the contents have definitely been tampered with since the signature was applied.

Since the point of the email was to get me to login for the first time to the registry website and set my password to control the domain, this is a little unfortunate.

Even if the signature had been correct, then should I trust the PGP key?

Well it is pointed to from the registry website which is a Good Thing. However, they do themselves no favours by referencing a version on the public key servers. I checked who had signed the key (which is an alternative way of trusting its provenance — since the email had arrived to a non-DNSSEC secured domain). Turned out there was no-one I knew, and of 4 individual signatures, 2 were from expired keys. The other signature was the IIS root key — which sounds promising. That has 8 signatures, once again not people I know — but only 1 from a non-expired key, so perhaps I can get to know some of the other 7?

Of course, anyone can sign a key on a public key server, so perhaps it makes sense for .se to suggest that people fetch a key with as many signatures as possible — there’s more chance of it being signed by someone they know. Anyway, I have now added my own signature, using an email address at my nice shiny new domain. However, it is possible that I may not have increased the level of trust

When I walked up on stage and accepted the Inc 500 award, it hit me square in the face that this is a rare accomplishment, and even more difficult for a product company that started without the benefit of VC funding.

Dave with wife, Anne, at the awards ceremony
Over the 2 day period, I heard from some great speakers with entrepreneurial passion, many who never had accomplished making the list. It is so highly competitive and just plain hard to do.

I loved hearing some of the speeches during the conference and getting to know other entrepreneurs that attended the conference talk about how they created their niche and ultimately built a successful company from a good idea.

Because I enjoyed hearing some of what I like to call “golden nuggets of wisdom” so much, I thought in my conference wrap-up I would pass on a few to our blog readers:

Tom Peters – Author In Search of Excellence and The New World of WOW

“Only 7% of our great nation works for Fortune 500 companies. Small businesses and the entrepreneurs are the jet fuel that makes our country fly.”

“Brand is shorthand for a collection of experiences, memories of what it will be like the next time a customer deals with you. With the advent of blogs and consumer activism, Brand is impossible to fake; it is like the temperature in the room… it is there… it exists.”

Chester Elton – SVP Carrot Culture Group

“At the casino – they train the heck out of the Valet! Why do they spend 3 months on Valet training? Because he is the first and the last person to greet and interact with a visitor during their trip! Who is your company Valet?”

Paul Bennett – Chief Creative officer IDEO – speaking on — Creating a culture of optimism:

“You need to ditch B-B and B-C Need to become P-P Person to Person.”

“You don’t buy loyalty… you earn it… this is an interesting challenge, but small allows us to behave like human beings… Going off script and doing something human is a great place to start.”

“Stop obsessing about ROI and start obsessing about ROC! Return on Customer/Consumer is much more powerful than ROI!!!!”

“Happy people, unabashedly doing, happy things, makes for happy companies, which create happy businesses which enable happy cultures… IN WHICH THRIVE”

Marilyn Carlson Nelson – Chairman and CEO Carlson Companies – A family owned $40 Billion empire including TGI Fridays, Radisson Hotels…

“My leadership was tested terribly - after 9/11 the travel industry was particularly harmed. It was an extraordinary time for Carlson. “

“Put tactics around these strategic initiatives”

• Whomever you serve, serve with caring
• Whenever you dream – dream with your all
• Wherever you go, go as a leader
• And never, never give up
• Whatever you do – do it with integrity

“That builds trust, trust builds relationships and relationships build results.”

=============================================

Actually, I took about 40 pages of notes throughout the two days… So I can’t say that this will be my last summary post on the Inc 500/5000 conference, but I can say that the conference did leave a strong impression about how I can help shape the future of ScienceLogic in an even more positive way.

Interop NY 2008 was up considerably in size from the show in 2007.  According to Lenny Heymann, the GM of Interop, this is a trend that they expect to continue.  My personal experience was that the size of the vendors was also up this year.  I think there were so few startups that “Startup City” was pulled from the show completely.  In any case, the show floor was full and there was plenty of attendee traffic to go around.

Definitely helping out from a traffic and draw perspective was the addition of the Web 2.0 Expo - Interop was co-located with both Mobile Business Expo and the Web 2.0 show. It seems like that buzzword still hasn’t lost most of its luster.

From the InteropNet perspective, the main feeling was one of being rushed.  With the show only lasting two days, and the InteropNet team only having a couple of days of ramp up time, everything was compressed into a much shorter period than in Las Vegas.  While this would normally be a challenge, it’s an even bigger challenge at the Javits where the InteropNet team was allowed to do almost nothing ourselves because of union rules.  You’d be surprised how frustrated you can make a network guy who’s told that he has to stand there and watch the electrician plug things in, rather than just doing it himself.  The only thing faster than the InteropNet team getting the Interop NY network up, was my pedicab ride to the InteropNet Booze Cruise.

In any case, everything came off without a hitch, and EM7 performed flawlessly catching a couple of power outages that last day and alerting everyone before the batteries on the UPSes had a chance to run down.

Over the next couple of weeks I’ll analyze the data from the show to see how many tickets were handled, amount of bandwidth consumed, etc and we’ll do a comparison to Interop Las Vegas.

We’re (both ScienceLogic and me personally) looking forward to Interop 2009.

There are two classes of contraband at airport security checkpoints: the class that will get you in trouble if you try to bring it on an airplane, and the class that will cheerily be taken away from you if you try to bring it on an airplane. This difference is important: Making security screeners confiscate anything from that second class is a waste of time. All it does is harm innocents; it doesn't stop terrorists at all.

Let me explain. If you're caught at airport security with a bomb or a gun, the screeners aren't just going to take it away from you. They're going to call the police, and you're going to be stuck for a few hours answering a lot of awkward questions. You may be arrested, and you'll almost certainly miss your flight. At best, you're going to have a very unpleasant day.

This is why articles about how screeners don't catch every -- or even a majority -- of guns and bombs that go through the checkpoints don't bother me. The screeners don't have to be perfect; they just have to be good enough. No terrorist is going to base his plot on getting a gun through airport security if there's decent chance of getting caught, because the consequences of getting caught are too great.

Contrast that with a terrorist plot that requires a 12-ounce bottle of liquid. There's no evidence that the London liquid bombers actually had a workable plot, but assume for the moment they did. If some copycat terrorists try to bring their liquid bomb through airport security and the screeners catch them -- like they caught me with my bottle of pasta sauce -- the terrorists can simply try again. They can try again and again. They can keep trying until they succeed. Because there are no consequences to trying and failing, the screeners have to be 100 percent effective. Even if they slip up one in a hundred times, the plot can succeed.

The same is true for knitting needles, pocketknives, scissors, corkscrews, cigarette lighters and whatever else the airport screeners are confiscating this week. If there's no consequence to getting caught with it, then confiscating it only hurts innocent people. At best, it mildly annoys the terrorists.

To fix this, airport security has to make a choice. If something is dangerous, treat it as dangerous and treat anyone who tries to bring it on as potentially dangerous. If it's not dangerous, then stop trying to keep it off airplanes. Trying to have it both ways just distracts the screeners from actually making us safer.

There are two classes of contraband at airport security checkpoints: the class that will get you in trouble if you try to bring it on an airplane, and the class that will cheerily be taken away from you if you try to bring it on an airplane. This difference is important: Making security screeners confiscate anything from that second class is a waste of time. All it does is harm innocents; it doesn't stop terrorists at all.

Let me explain. If you're caught at airport security with a bomb or a gun, the screeners aren't just going to take it away from you. They're going to call the police, and you're going to be stuck for a few hours answering a lot of awkward questions. You may be arrested, and you'll almost certainly miss your flight. At best, you're going to have a very unpleasant day.

This is why articles about how screeners don't catch every -- or even a majority -- of guns and bombs that go through the checkpoints don't bother me. The screeners don't have to be perfect; they just have to be good enough. No terrorist is going to base his plot on getting a gun through airport security if there's decent chance of getting caught, because the consequences of getting caught are too great.

Contrast that with a terrorist plot that requires a 12-ounce bottle of liquid. There's no evidence that the London liquid bombers actually had a workable plot, but assume for the moment they did. If some copycat terrorists try to bring their liquid bomb through airport security and the screeners catch them -- like they caught me with my bottle of pasta sauce -- the terrorists can simply try again. They can try again and again. They can keep trying until they succeed. Because there are no consequences to trying and failing, the screeners have to be 100 percent effective. Even if they slip up one in a hundred times, the plot can succeed.

The same is true for knitting needles, pocketknives, scissors, corkscrews, cigarette lighters and whatever else the airport screeners are confiscating this week. If there's no consequence to getting caught with it, then confiscating it only hurts innocent people. At best, it mildly annoys the terrorists.

To fix this, airport security has to make a choice. If something is dangerous, treat it as dangerous and treat anyone who tries to bring it on as potentially dangerous. If it's not dangerous, then stop trying to keep it off airplanes. Trying to have it both ways just distracts the screeners from actually making us safer.

---

Bruce Schneier is chief security technology officer of BT. His new book is Schneier on Security.