Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

Making IT work as One

How does my company stay efficient while we’re using technologies around interoperability? How can innovation help my business?

Top business needs:

• Reduce cost
• Manage complexity
• Mitigate risk

Mixed IT environments are a reality for almost all organizations. Different environments, architectural strategies, desktop profiles, etc. There are benefits to having mixed source environments, although homogenous environments are ideal. On average 46,000 hours in an organization are spent on Sarbanes-Oxley standards.

Some considerations to make IT work as one:

• Strategy
• Solutions
• Ecosystem

Strategy

Actionable strategy is key. The emergence of three silos (applications, systems and infrastructure, and operations) are now moved into one. There is a lot of pressure to make these pieces come together.

Solutions

You need focused solutions to solve problems today while keeping an eye to the future. There are three main needs: the data center, end-user computing, and identity and security. This is also what is the most important to the market right now. The end goal is the agility of the data center.

Data Center Challenges

• Create an agile IT infrastructure
• Address power and space constraints
• Deliver performance, security and availability
• Manage hardware, software and labor costs
• Meet service level agreements

Data Center Solutions

• Workload management - green IT and server efficiency, unified physical and virtual environment
• Virtualization and Consolidation - business continuity and disaster recovery
• Enterprise Servers

End-User Computing Solutions

• Collaboration
• Enterprise desktops - Novell uses Linux and Open Office, interesting to note
• Endpoint management

Identity and Security Challenges

• Minimize risk, uncertainty and policy violations
• Provide timely and secure access to information
• Ensure, document and prove information security
• Reduce the cost of proving compliance
• Reduce the cost and complexity of governance

Identity and Security Solutions

• Identity and Access Management - user provisioning, role management, access management
• Compliance Management - Audit, Governance, Risk Management and Compliance (GRC), IT controls automation, Security, Information and Event Management (SIEM)

Ecosystem

The ecosystem is powerful. Companies should challenge partners for innovation and interoperability.

Community Innovation - open source and open standards

IT Landscape - Mixed IT Environments

• Consulting, systems integration vendors
• Application vendors
• Systems software vendors (Novell)
• Hardware, network vendors

How does your ecosystem help your company? How do your partners help? What is their role in the industry to help you? How are all the vendors in the industry helping you?

Marc’s initial shoplifting model in his post is based on John Colapinto’s concepts of matching a pattern of customer movements in the store with their estimated patterns of shoplifting behavioral patterns.    Marc’s asks how Coral8 might address this.   We are not ready to seek a vendor solution.  We do not yet have a workable detection model.

As indicated above, I don’t think the example situation cited by John and Marc is a viable model for automated processing.    Tracking the behavior of customer’s movements, by machine, would require some very sophisticated image processing technology that would be too expensive compared to any possible loss at most retails stores.    This type of behavioral pattern recognition. in retail stores, is performed by people (security personnel), not machines, observing people.  

To develop a machine pattern recognition application to detect retail shoplifting we need to build detection models that are economically feasible.  If we are going to use a model of shoplifting pattern recognition versus anomaly detection, we need to define the objects we must track.  

In the most simple model, we have merchandise-objects.   Stores normally (physically) track merchandise-objects only at the exit/entry points of the store using some electromagnetic proximity detection technology.   In this model, the detection configuration is a combination of simple alerting with humans watching the store (”minding the store”).    This is not complex event processing.

However, if we added another object to our model, the customer-object, then we start to get more “complex,” but we have not defined “complexity” yet because we have not defined the object properties, the possible states of the objects, and the relationships between the objects that are the basis for estimated situations.

Hence, model building is constrained by available resources, simple economics and risk (cost-benefit).  If we are detecting shoplifting in Walmart the cost-benefit model for implementing an automated shoplifting detection system would be different than at a top diamond store on 5th Avenue in NYC.   Protecting loss at a weapons-grade uranium respository follows a different model than protecting loss at a handicraft shop, naturally.

Like Marc, I find models to automatically detect shoplifting interesting, so permit me to close with a general discussion of shoplifting in the context of our CEP/EP reference model.

One approach would be do determine what objects will be represented in our model.   For example, if we are going to track merchandise, we need to model the ”merchandise-object”.  If we are going to track people, we need to define the properties of this “person object.”  If we are going to represent the store layout, we need to define all these objects (store-object, table-object, shelf-object, entry-object and so forth).  The model can get “complex” quite quickly. 

Editorial Note:  An object-oriented approach greatly assists complex model building because we can benefit from OO properties such as encapsulation and polymorphism.  For example, we can define a basic “person object class” and then create superclasses of this object for “customer-object”, “manager-object”, “or criminal-object.”

Generally speaking, each object we define will require a state-model, for example, in Marc’s example of a customer moving around the store, we would need to model the possible states (customer at the entrance, at table 1, at table 2, at shelf 1, in the bathroom, at the cashier, etc.)  Indeed Marc, this is complex event processing if we have modelled multiple objects and defined object-object relationships that indicate situations of interest.   For example, customer-object at table2 where merchandise-object has the property of  ”very expensive, high risk” and then customer-object changes state to “in bathroom”.  Of course, we need more key indicators, but you get the idea.

Right now, I am typing from the Taste from Heaven Vegetarian Restaurant in Chiang Mai and my battery is running low.  The owner of this excellent restaurant also runs the Elephant Nature Park, a non-profit organization advocating and acting on behalf of the rights of the mighty elephants in Thailand.  Would be great if we could also automatically detect the situation of “elephant abuse” by poachers and other crimes against nature.   Time to get back to my delicious mushroom salad, Northeastern Thai style.

As always, thanks for reading, time for me to get back to eating!

First of all, let’s ground the discussion a bit by translating “smart order routing” to “rule-based message routing” since in this application “smart”  translates to “using rules” and “order” translates to “message”.    Basically, Mark (and other “new on the routing scene” stream processing players) argue that rule-based message routing is CEP.  I will argue that routing is not even close to CEP.  Here is why,

Let’s take a look at a router on the backbone of the global Internet.   A backbone router has very sophisticated software developed over many decades.   These routers run sophisticated, mature algorithms to determine how to route messages (packets) and use these algorithms to build complex routing tables. 

In addition, these routers process messages (packets) from countless sources and route messages (packets) to countless destinations.  Using some of the terms in early posts (above), there is a great “confluence of events” processed by routers.    Futhermore, there are normally quite complex authentication, authorization and other security parameters managed in a router, all in real time.   Routers do much more, but I don’t want to get too deep into routing in this post.

My point is that, without any doubt, global Internet routers process very “cloudy” “confluence of events” with much more sophistication than order routing applications.    However, we do not call Internet routing “CEP”, regardless of how many connections are processed or how much sophisticated processing occurs.  The reason is because the “C” in “CEP” defines a complexity that is at a higher abstraction than messaging and routing.

If you study the literature on CEP, some of which I posted recently, CEP was envisioned to solve complex event processing problems “on top of the routing layer” because the routing layer is a mature technology layer.  We can route, pure and simple.  Of course, we are always seeking faster, more scaleable and more secure routing. 

I admire some of the startups in the CEP/ESP/EP space for working hard to make money and for aggressively positioning their products and attempting to build market share.   However, issues surface when these same companies seem to believe they are the first companies to work in the event processing or message routing space and that they can define whatever they want as “complex event processing” as long as it benefits their sales targets.

There is no doubt that a router does much more sophisticated event processing than the new rule-based stream processing systems running continuous queries across streaming data.  There is no doubt that a router processes a complex “confluence of events”.   However, we don’t call routers “CEP”. 

We do not call routers “CEP” because CEP is about a higher level of knowledge processing.  CEP was created to detect the “complex events” that happen above the mediation and routing layer.     The literature and original examples on CEP are quite clear on this.