Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

• ValidationSummary controls look at the ErrorMessage field to figure out what to display, so always use ErrorMessage in a verbose enough way that it will be helpful from a ValidationSummary control.
• If you need a shorter message to display inline (i.e., where the validation control is on the form, as opposed to the ValidationSummary) use the body of the control to define it.

In the past, I've used RequiredFieldValidator controls on my web forms to remind users that certain fields are required. I would set the ErrorMessage to something vanilla like, "This field is required", or even something simpler like "*" (an asterisk) if I didn't have much room on the form to display more prose for an error.

A friend was recently testing a new feature that I'd built for our sales team and she had a hard time seeing the little red asterisks that were showing up next to required fields. It felt to her as though she was pushing the submit button on the form but nothing was happening. It was clear that a ValidationSummary control would be helpful, especially if placed close to the submit button for the form.

I've been a bit lazy in the past about using ValidationSummary controls, partially because most of my forms are simple enough that they feel a bit redundant. But on a more complicated form, they can be very helpful to guide users back to the places on the form where there's problems.

So I threw one of those puppies on the form and immediately saw that there was a problem - my error message was set to "*", which meant that my validation summary was pretty useless - it just displayed a bunch of red asterisks! And in places where I'd used the prose, "This field is required", well that was pretty useless as an error message in the summary.

After a bit of research and experimentation, I discovered that the ValidationSummary control looks at the ErrorMessage property on each validation control in order to figure out what to display in the summary. So it's important to use ErrorMessage with a summary in mind! Don't use text like "*" or "This field is required". Be more specific so the user can find her way up to the problem field, as in, "PostalCode is required".

But if you make ErrorMessage verbose so that it's helpful in a summary, it may make your form really ugly when displayed inline next to the control being validated. The trick is to use the body of the validation control element to specify the inline error message. Then you end up with two messages: a verbose one that's used in your summary, and a more localized, brief message that shows up right next to the control being validated. Note the asterisk that's in the body of the RequiredFieldValidator below:

<asp:RequiredFieldValidator
      ErrorMessage="Zip/postal code is required"
      ControlToValidate='txtPostalCode'
      ValidationGroup='BasicInfo'
      Display="Dynamic"
      runat='server'>*</asp:RequiredFieldValidator>

I've learned a lesson from all of this. In the future when I use validation controls I'll always provide a summary-friendly message in the ErrorMessage field, and if I need something different (typically shorter) to display inline, I'll put it in the body of the validation control element.

Hope this helps!

Examples include:

• GSM mobile phones have an identifier for the phone (separate from the identifier for the user) that can be blacklisted when the phone is stolen.
• Some car radios will stop working when the battery is disconnected, and only start working again when a numeric code is entered. This is intended to deter theft of the radio.
• In Windows Vista, Bitlocker can be used to encrypt files. One of the intended applications for this is that if someone steals your laptop, it will be difficult for them to gain access to your encrypted files.

Ross told a story of what happened when he needed to disconnect the battery on his car: the radio stopped working, and the code he had been given to reactivate it didn’t work - it was the wrong code.
Ross argues that these reactivation codes are unecessary, because other measures taken by the car manufacturers - such as making radios non-standard sizes, and hence not refittable in other car models - have made them redundant.

I described how the motherboard on a laptop had needed to be replaced recently. The motherboard contains the TPM chip, which contains the encryption keys needed to decrypt files protected with Bitlocker. If you replace the motherboard, the files on your hard disk will become unreadable, even if the disk is physically OK. Domain-joined Vista machines can be configured so that a sysadmin somewhere within your organization is able to recover the keys when this happens.

Both of these situations suffer from classic usability problems: the recovery procedures are invoked rarely (so users may not know what they’re supposed to do), and, if your system is configured incorrectly, you only find out when it is too late: you key in the code to your radio and it remains a doorstop; the admin you hoped was escrowing your keys turns out not to have the private key corresponding to the public key you were encrypting under (or, more subtly: the person with the authority to ask for your laptop’s key to be recovered is not you, because the appropriate admin has the wrong name for the laptop’s owner in their database).

I also described what happens when an XBox 360 is stolen. When you buy XBox downloadable content, you buy two licenses: one that’s valid on any XBox, as long as you’re logged in to XBox live; and one that’s valid on just your XBox, regardless of who’s logged in. If a burglar steals your Xbox, and you buy a new one, you need to get another license of the second type (for all the other people in your household who make use of it). The software makes this awkward, because it knows that you already have a license of the first type, and assumes that you couldn’t possibly want to buy it again. The work-around is to get a new email address, a new Microsoft Live Account, and a new Gamer Tag, and use these to repurchase the license. You can’t just change the gamertag, because XBox live doesn’t let the same Microsoft Live account have two gamertags. And yes, I know, your buddies in the MMORPG you were playing know you by your gamertag, so you don’t want to change it.

ScienceLogic: How long have you been involved in InteropNet?

Katsev: I started at Coyote Point 3 years ago and InteropNet 2006 was my first “big” assignment.  This was the first time Coyote Point had put in a proposal to participate, so we were very excited when we were selected.

ScienceLogic: How long has Coyote Point been involved in Interop overall?

Katsev: We’ve been exhibiting at Interop for a number of years, and after seeing the InteropNet in action, we decided to submit a proposal in ‘06.  We were actually one of the first companies in the load balancing/traffic management space (we’ve been doing this for almost 10 years), so we have a lot of experience to share with InteropNet.

ScienceLogic: What is your role at Coyote Point?

My official title is “Engineering Project Manager”.  Basically, that means that I’m in charge of product releases and maintenance.  It sounds like a weird title for someone participating in InteropNet, but I’ve actually found it extremely useful since my position means that I don’t get to see our systems out in the field a lot.  We’ve added several features and have ideas for others just from my experiences at InteropNet.

ScienceLogic: What do the Coyote Point products do?

Katsev: Coyote Point makes a Traffic Management appliance called Equalizer.  What this means is that any traffic destined for a datacenter’s servers goes through our appliances and we make sure that the server which is best equipped to handle it, does.  Our systems sit between the clients and the servers and monitor the client traffic and the state of the servers.  If the clients start sending more traffic, we’ll balance it out so that no server is overloaded.  If one of the servers stops responding or starts responding very slowly, we’ll steer traffic away from that server.

ScienceLogic: In what way are your products being used as part of InteropNet?

Katsev: In the InteropNet, we’re utilizing a lot of our expertise:  We’re making sure that traffic is balanced and servers are redundant for show services such as DNS and SMTP.  We’re also using our geographic load balancing technology to ensure that the ScienceLogic EM7 appliances and some other internal NOC services are available from anywhere, with the lowest latency, with our SSL acceleration and GZIP compression technology.  Finally, we’re helping logistics in the NOC by allowing a physical separation between systems located in the NOC and those in an emergency rack outside of the NOC.  If either of these two locations were to fail, the network will continue operating without a glitch.

ScienceLogic: Are there any special considerations for Interop that cause you to deploy your systems there differently that any other place?

Katsev: Interop is definitely different than most of our customer installations.   One difference from a standard environment is that the network (at least this year) is one large flat network, with pieces carved out where extra security is needed.  Because of this, we can actually run our failover pairs of Equalizer systems in a non-standard configuration where the two peers are in different racks, or even on different floors.  That’s one of the things that I really like about InteropNet — it definitely brings new ideas to mind, which end up becoming ’special configuration’ white papers after the show.

ScienceLogic: Has InteropNet taught you anything that caused you to actually change your product?

Katsev: In addition to the failover configuration differences I mentioned above, participating in InteropNet has actually caused us to add several new features and allowed configurations.  One example is the “no-spoof” option for Layer 4 clusters.  Prior to the 2006 shows, we always ’spoofed’ the client’s IP address when talking to a server so that the server would see the client’s IP address instead of our own.  At Interop, we ran into a special configuration which would’ve been very difficult to set up in this manner, so our engineers added this feature, and it’s been very a very popular configuration with our customers ever since.

We have also had a couple of business relationships that extended outside of the show.  In 2006, we had a good experience using Spirent Communications gear to benchmark the network, so we ended up purchasing a couple of these systems to test our products.  More recently, we have found a way to bundle our Equalizer e350si load balancers with the ScienceLogic EM7 collector appliances to help ScienceLogic get the best performance in load balancing large quantities of syslog messages to be processed.  If it wasn’t for our participation in InteropNet, neither of these relationships would’ve happened.

ScienceLogic: What’s the best part of being involved with InteropNet?  What do you most look forward to?

Katsev: InteropNet is an amazing networking opportunity (no pun intended).  The group of engineers that put the network together every year is, well, amazing.  There is so much combined experience that any question instantly has several possible answers, and the best answer is chosen very quickly.  One of the ’sayings’ at Interop is “if you run into a problem, ask someone… we’ve probably seen that problem before… five times.”  One would think that being part of InteropNet is the same thing, year after year.  However, in the two years that I’ve been part of this (for four shows), there have been huge differences in the way that the network is designed and put together.  These are both because the vendors selected every year are different, and because the engineers who design the network change from year to year.  Somehow, though, when all is said and done, we have a network that works.

ScienceLogic: You don’t have to answer this one if you’re not comfortable… What would you like to see changed with the way things are done at InteropNet?

Katsev: This isn’t a cop-out… I really can’t think of anything I would do differently.  Sure, there are small problems that pop up sometimes, but every project has those, and the people at InteropNet are more than capable of figuring them all out.  In fact, I know that Interop started out as a show to test the interoperability of devices… but I’m still amazed that all of these devices actually talk to each other and “play nice” together.

ShareThis

The big difference between the two cities is the amount of time that the InteropNet team gets to produce a live, fully operational and redundant network. In Las Vegas, this was nearly a full week of time - a tight timeframe across 17 different vendors, but now we’re looking back at that timeframe as a luxury. In NY, we’ll be getting started Saturday morning, and the network needs to be delivered on Sunday morning for the registration desk and exhibitor move-in to begin. If you’re keeping score, that’s about 24 hours to deliver a working network. Sounds hard, but it’s even harder when you consider that this means four DS-3s from two different locations, 17 full and 7 half racks of network gear, all the fiber and copper that the network is delivered over, etc all have to get done. Good thing that with 2 and 3/4 kids, I’m not planning on much sleep, and I don’t think the rest of the team is either.

In order to try and get the network delivered in that short timeframe, we worked hard at Hot Stage to assure that everything is ready to go. With some luck, the work that we’ve done here will allow us simply to roll the network gear into place, run the cables, fire up and go.

Now, things never really work out that way but that’s what EM7 is going to be there for. We’ll watch in real time as the network elements come live and be able to let the other InteropNet vendors know if their gear isn’t behaving as expected or is not visible for all the areas of the network that it should be. We’ll keep track of all of this in the EM7 ticketing system so that after the show we’ll be able to analyze the behavior of the network and systems as we did after Vegas.

I’m looking forward to the show and once again working with some of the top engineers in the country on a complex and rapidly deployed network.  Speaking of which, we’re still looking for volunteers to help in the NOC.  Volunteers get to work with some really smart people, get an education that would be hard to get anywhere else, and get a trip to NY where your expenses (for things like hotel accommodations and food provided by the show) are taken care of.  Sound interesting?  Be sure and check out the application.

ShareThis

From Aut Disce, Aut Discede:

Consider for a moment the state of client-side bugs 5 or 6 years ago. Attacks such as this, a multi-stage miscellany of IE and Mediaplayer bugs that resulted in the “silent delivery and installation of an executable on the target computer, no client input other than viewing a web page” were reported with regularity. Gradually these type of attack gave way to exploitation of direct browser implementation flaws such as the IFRAME overflow and DHTML memory corruption flaws. So what has become of the multi-stage attacks - have they become redundant? The answer to this, which I’m sure you can guess, is a resounding “no” and will be emphatically demonstrated in my upcoming Black Hat talk “The Internet is Broken: Beyond Document.Cookie - Extreme Client Side Exploitation”, a joint double session presentation co-presented by Billy Rios, Nate McFeters and Rob Carter.

As a teaser for that, I’m going to revisit an old attack - pre-computed dictionary attacks on NTLM - and discuss how we can steal domain credentials from the Internet with a bit of help from Java. I’m going to split it into two posts. In this post we’ll apply the attack to Windows XP (a fully patched SP3 with IE7). In my next post we’ll consider its impact on Windows Vista.

For the full article read on.

Why are you still here? Go read it.

Article Link