But that credit card weirdness is nothing compared to the one I’m about to describe.

Before we do that, let’s take a moment to discuss the design principle of failing securely. The general idea is that if a security mechanism fails, it should fail closed. If your firewall crashes, it should block all traffic, not allow all the packets through. If the power source to your card key system is interrupted, it shouldn’t unlock all the doors. If the connection between your application server and your LDAP directory is severed, subsequent authentication requests should be rejected, not approved. This is not rocket science.

So back to credit cards. I had a conversation last night with an old friend who related a bizarre situation they had encountered during the QA process for one of their web applications. One of their tests involved repeatedly attempting a credit card transaction using a canceled/expired American Express card. Here’s what they saw in their logs, paraphrased by me:

Attempt 1: Denied
Attempt 2: Denied
Attempt 3: Denied
 .
 .
 .
Attempt 49: Denied
Attempt 50: Denied
Attempt 51: Approved

What the…? Approved? That can’t be right. So they ran the test again. Every time, after multiple consecutive rejected attempts, the transaction would inexplicably go through. The threshold wasn’t always 50, but the general pattern was consistent — keep trying and eventually it’ll work. Clearly, this had to be a bug in the code, but a deep-dive into the guts of the application turned up nothing. The application security group got American Express on the phone to see if they had any insight on this odd behavior. The answer? They didn’t concede the failure was on their end, despite log data showing the successful authorization codes.

My gut instinct would be that the application requesting the transactions wasn’t failing securely (e.g. network connection to AmEx timed out, so just approve the transaction). But that explanation wouldn’t account for authorization codes coming back.

So what in the world is going on here? Why would the system behave this way? Is it by design? I can’t think of a single legitimate use case for failing open like this. If this is actually a design decision by the credit card companies, I have no doubt that someone in our audience knows the rest of the story.

Be sure to check it out soon (you can preview bits of each course right now for free), as we're offering a limited-time early adopter discount that's good for the life of your subscription. Our online courses are told by the authors themselves, with names that you'll recognize, as many are MSDN Magazine contributing editors and book authors on their topics.

Courses we now offer online include:

WCF Fundamentals by Aaron Skonnard

Windows Workflow Fundamentals, by Matt Milner

WPF Fundamentals, by Ian Griffiths

Silverlight Fundamentals, by Ian Griffiths

ASP.NET 3.5 Fundamentals, by Fritz Onion

ASP.NET Ajax Fundamentals, by Fritz Onion

LINQ Fundamentals by Scott Allen

BizTalk Fundamentals by Matt Milner

BizTalk Server 2006 R2 Fundamentals by Jon Flanders

We'll be expanding this library of content in the months to come, as we continue to grow this online resource. I plan on adding modules on the Geneva family of identity products (Geneva Server, Geneva Framework, Geneva CardSpace) announced at PDC this week.

I've learned a lot of interesting tidbits as I helped to develop the back end infrastructure for Pluralsight On-Demand, and now that I'm not so crammed for time, I'll be sharing those insights here on this blog.

Congrats to all who helped bring this incredible resource to the public!

Aaron has more details if you want to know about pricing, customer feedback, and so on.

Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

Eye-Fi divided its Wi-Fi SD card line-up into three parts earlier in the year: Home, which transfers to a computer ($80); Share, which uploads to a computer and to Eye-Fi's servers, which relay them to gallery, print, and social services ($100); and Explore, which ties in Wi-Fi positioning and one year of a Wayport hotspot subscription for uploads ($130). I wrote a long review of the Eye-Fi Explore on 12-Aug-2008.

If you bought a Home, you can upgrade to the Share service for $10 per year, and if you bought either a Home or Share, you can add geotagging for $15 per year and hotspot access for $15 per year. It's a smart move, since original Eye-Fi card buyers already had a firmware upgrade that converted their card into a Share model; they'll now be able upgrade to the full featureset. This is something I thought the company was offering at launch months ago, and I speculated it would be easy to add.

Eye-Fi also added two new photo sharing services: Apple's MobileMe and AdoramaPix. I cannot think of any other firm that Apple has partnered with to allow direct MobileMe uploads, although this may be technically less a big deal than it sounds. But I believe it's unique--only the iPhone and iPhoto software can transfers images into MobileMe's galleries; I'll need to investigate further. It's a good feather in Eye-Fi's cap.

Finally, Eye-Fi says they'll release tweaked firmware on 5-Oct as well that will double the speed of photo transfers from their cards to a computer on the local network.

AT&T now says that any "FastConnect" subscription, even its DSL Lite offering of 768 Kbps down/128 Kbps up, qualifies for Wi-Fi Basic. The new statement reads: "AT&T Wi-Fi Basic service is FREE and already included if you subscribe to AT&T High Speed Internet, AT&T U-verseSM High Speed Internet, or AT&T FastAccess® DSL—all speed plans included.

There's still a $10 per month fee to upgrade to Wi-Fi Premier, which includes over 70,000 locations worldwide, along with the missing U.S. hotspots, but their Web site says that you have to have a 1.5 Mbps or faster connection to get the $10 per month upgrade. That may be out of date. That ordering page also says you need 1.5 Mbps or faster for free Wi-Fi, so that tends to confirm it hasn't been fixed. (It's even hosted at sbc.com, so perhaps that's part of the vestige of an older system, harder to update.)

Please note that iPhone subscribers still don't get free Wi-Fi on AT&T's Basic network.

Slacker licenses music directly from publishers, and includes a perpetual subscription in the cost of the player. Slacker creates stations that feed out an endless supply of music. The new models are $200 for a 4GB model with the ability to list 25 stations (up to 2,500 songs), or $250 for an 8 GB model with 40 stations (up to 4,000 songs). You can also sync your own music in MP3 or WMA format. For $7.50 per month, you can upgrade and store songs you're listening to, as well as avoid ads.

The G2 is already getting reviews as a much-improved upgrade from the first release. Like the Zune, there's no browser or other Internet features, and that might be a positive.

The G2 is tied into Devicescape's Wi-Fi home and hotspot authentication system, which lets Slacker G2 owners pre-program encryption keys or login information for hotspots that they frequent. Devicescape's software both retrieves and stores login information, allowing the G2 to be used in places that would otherwise require either tedious entry of a WPA passphrase, or be unavailable without a Web browser to handle the login.

The Zune doesn't include a Web browser or any Internet focused features; it's not an iPod touch. But you can use Wi-Fi to browse the Zune Marketplace for music and games, and download new songs in programmed channels, music selections created by a variety of artists and stations. Zune offers both music purchases and a subscription for unlimited music listening. The new models range from $149 for an 8 GB flash model to $249 for a 120 GB hard drive-based player.

The feature I'm most interested in is Buy from FM, which leverages the built-in FM tuner and very low-bandwidth data that's already pushed over analog AM/FM. (See my write-up of this feature from last week.) With Buy from FM, when you're listening to radio stations that participate, you'll be able to click a button and buy the song you're listening to if you're connected to a Wi-Fi network. Zune Pass subscribers can download the song at no additional charge. If there's no Wi-Fi network, the song download or purchase is queued.

Wayport's marketing head Dan Lowden said, "Obviously, it's cool because folks who already own a Zune device and just need to do an upgrade will be able to use this just as with any of the new Zune devices that they start selling as soon as possible." (Microsoft may have a little accounting work to do: Sarbanes-Oxley doesn't let you enhance a product in the market without a fee if you realize the revenue all at once.)

The benefit for Wayport is to have yet another hefty but undisclosed fixed sum underlying its fixed infrastructure costs. In the past, Wayport has done deals with Nintendo, ZipIt, and Eye-Fi to allow all devices in a category unlimited access at McDonald's locations. McDonald's obviously gets more customers, or existing customers who spend more time or visit more frequently.

A partnership with a hotspot operator means that Microsoft doesn't have to provide tools and their users endure frustration in joining a network. "We're experts enabling one click to get this network connected," Lowden said. He noted that Wayport has opened test labs to work with manufacturers in Japan, San Francisco, San Diego, and Seattle. "We're working with these guys from day 1 to make sure it's one click to get connected," he said. I'd also note that San Diego happens to be where Qualcomm's headquarters are located, not that Lowden gave me any tip-off there.

And I have to just say: burn, burn, burn on Apple. Despite Apple partnership with AT&T, which relies on Wayport to operate the AT&T-branded hotspot network and resells access to Wayport's own network, iPhone and iPod touch users have no inclusive Wi-Fi service. AT&T slipped a few times and ostensibly opened up their network or released details that iPhone users would gain free hotspot access--like all AT&T's fiber and all its standard and premium DSL customers.

As Wi-Fi becomes an expected part of any handheld gadget, the venues in which Wi-Fi is used multiply beyond cafes and hotels. Lifestyle locations--which could be clothing stores, nightclubs, ski resorts, and the tops of mountains suddenly become places where people want the same kind of access they have at home. Ultima thule is already unwired.

Apple added access for iPhone and iPod touch users to a subset of its iTunes Store over Wi-Fi--the awkwardly named iTunes Wi-Fi Music Store--more than a year ago, along with the ability to access that store at no cost from handhelds and laptops via Starbucks outlets in New York, Seattle, and throughout the San Francisco Bay Area. (Chicago and Los Angeles have been "coming soon" for a year, but the new AT&T/Starbucks deal may have delayed opening up those markets.)

Terrestrial AM/FM radio stations would like to figure out how to remain meaningful in a world of streaming Internet radio. Their latest strategy is to embed information that allows a listener to mark a song they want, potentially getting a piece of music sold in this fashion. With FM tagging, Zune players tap into an existing very low-data-rate encoding protocols that allow stations to push out their call letters and current song information. By adding a very short code, broadcasters can allow Zunes to look up the appropriate song.

At launch, 450 stations from major networks, including Clear Channel, Entercom, and others, will broadcast tagging details. Note that Microsoft includes KEXP, a Seattle independent and alternative radio station, in its sample image, for the new models. KEXP, given a boost a few years ago through significant short-term funding by Paul Allen--funding that involved changing its call letters to his Experience Music Project museum initials--has an enormous listenership over the Internet ironically enough. KEXP will be a programming partner creating channels of music for the subscription-based Zune Pass service. (Zune Pass is $15 per month, all you can eat.)

This option could allow Microsoft to ink partnerships with hotspot networks to brand them with Zune compatibility, lets radio stations promote something other than iPods that they would have a direct relationship with (and, potentially, some kind of revenue stream from?), and may be part of breaking Apple's digital music hegemony. May be. Nobody's gotten rich betting against Apple for the last several years. (Details of revenue sharing with radio stations hasn't been discussed.)

Apple opted for a partnership with HD Radio broadcasters and equipment makers that has a relatively elaborate process of tagging songs. HD Radio is digital AM/FM, a patented and licensed method that has provoked a lot of controversy, and has lagged enormously in the marketplace, despite well over 1,000 stations (including many public radio stations) broadcasting in this digital format, some for over three years.

HD Radio tagging requires an HD radio receiver with a Tag button; pressing that button stores the song's tag information. The radio must also have an iPod dock. Docking an iPod syncs the tag information, and the next time the iPod is sync with iTunes, you can see which songs were tagged. Kind of tedious compared to "press a button while listening to an FM station and buy the song over Wi-Fi." (I've been writing about HD Radio for years, and even launched a blog that's gone moribund; the technology is interesting, but Internet radio on mobile devices coupled with on-demand music purchasing over cell and Wi-Fi may simply make HD Radio unnecessary for listeners.)

Microsoft has a more compelling "marketing story" for this feature than Apple, that's for sure. On the other hand, do you really need to tag songs from stations that play only the most popular music in a given format?