Corporate custodians of confidential medical data should be closely monitoring events connected to a nightmarish computer security breach in the St. Louis region.

Express Scripts is one of the nation’s largest pharmacy benefits managers. The company, with headquarters in St. Louis County, handles approximately 500 million prescriptions per year for 50 million workers at 1,600 American companies. Early in October, it received an extortion letter, the details of which it released on Nov. 6.

The letter included personal information on about 75 Express Scripts clients — Social Security numbers, dates of birth and, in some cases, information about prescription medications. Whoever sent the letter demanded money from the company — the amount has not been disclosed — and threatened to use the Internet to reveal personal and medical information about millions of people if the demands were not met.

...

Beyond the scale of the problem for Express Scripts — and the potential impact on the company is enormous — the issue extends well beyond the mounting concerns about identity theft, a phenomenon with which most people have become at least somewhat familiar.

The greater problem is the unique nature of personal medical records, the importance of moving to computerization of such records to improve health safety and reduce costs and the irreversibility of the damage people can suffer if confidential medical information becomes public. The stakes are so high that a federal law establishes strict standards for maintaining the privacy of medical information and stiff fines for failing to do so.

Medical records of all kinds — paper and, especially, electronic — must be protected with the most sophisticated kinds of security systems available, including backup protections and automatic alerts of security violations. Yet Express Scripts learned of this breach in the “worst way,” as InformationWeek.com security correspondent George Hulme put it in an online report: “via an extortion letter.”

The Express Scripts breach raises many questions for all elements of the health industry: hospitals, clinics and doctors’ practices, benefits management firms, insurance companies, pharmacies, employers and government agencies:
Are they using the most advanced information security technology possible? Do they minimize the amount of data they collect and keep it only as long as necessary? Do they have strict protocols governing access to personal and medical data — and systems to enforce those protocols? If criminals were to hack into their systems, how would the companies know? How soon? And are the systems capable of instantly cutting off illegal access as soon as a breach is discovered?

Confronted with a grave breach of electronic security, Express Scripts has responded by contacting law enforcement, establishing an informational website, offering a substantial reward and hiring a private consulting firm to help clients who have privacy concerns and investigate situations that “appear to be tied to identity theft” and provide “identity restoration services.” There is no question that the company is taking the situation extremely seriously.
Given the ongoing criminal situation, information about how Express Scripts’ data systems were compromised — and whether it could have been avoided — has yet to be disclosed. But the American people have the right to expect that their sensitive personal and medical information is zealously protected and kept secure — not only by Express Scripts but also by every person or company entrusted with it.

"Customers and customer relationships...have tangible measurable value to businesses, and their value is much easier to communicate to those who fund projects. So in an enterprise risk management scenartio, their vlaue informs the risk management process...[For example, consider] a farmer deciding which crop to grow. A farmer interested in short term profits may grow the same high yield crop every year, but over time this would burn the fields out. The long term focused farmer would rotate the crops and invest in things that build the value of the farm and soil over time. Investing in security on behalf of your customers is like this. The investment made in securing your customer's data build current and future value for them. Measuring the value of the customer and relationships helps to target where to allocate security resources."

Now I’ve been “accused” of being a quant in the past (hi rybolov!) but in reality the only dogs I have in this fight are the model and the application of scientific method - and really, ethically speaking, I have to be tied to the latter while applying the former.

And I see a false dichotomy in this whole Quant vs. Qual thing.  We, as a profession, tend to create a political divide between the two which, if it even exists, I’d say is based more on our ignorance rather than our expertise.  After all, we are the profession that regularly multiplies across ordinal scales and uses wonderful models like R=VxTxI.   As someone  learning to deal in probabilities and rationalism, I have to recognize that this discussion is really just about the act of observation using different metrics of measurement.

But how we’re going about observing does not change the fact that there is measurement based on observation.  So if I’m working with you I can easily turn your qualitative scale into a quantitative one, and vice-versa.  Yes, Shrdlu, if we had the time, even your most seemingly Qual things could be Quant! (This flexible world view, btw, is an outcome of that new-fangled Bayesian thing).

COGNITIVE BIAS A-PLENTY

But back to what Rich is saying there about information security and risk - and he isn’t/won’t be the only one saying these sorts of things - we should try to understand what’s really going on rather than get caught up in the emotional hurricane.  Our profession suffers several forms of cognitive bias.  The nature of our jobs and what we do can cause us to be focused on the outcome and not the quality of the decision at the time it was made.  We want to bring in things from other professions that are useful, but at times we do view things outside our profession with false correlation to our own (unfortunately for those who write these sorts of articles, financial risk is completely different than operational risk).  We also have the tendency to focus on negative outcomes without acknowledging the positive outcomes (For example, I hear that Alan Greenspan’s new firm is up a couple of $billion in all this mess since he joined them, short sellers are doing quite well - must be because they have qualitative models or something -grin-).  The effect of these biases are compounded by the facts that proper correlation takes more work than we usually give it, and rational thought is not that easy when there’s a witch-hunt mentality.

What also floats in water? (link to Youtube)

WHAT SHOULD WE BE THINKING ABOUT?

So as you and I read opinions that seem to be the polar opposite of irrational exuberance (and there will be plenty between now and the election) we’ll have to ask ourselves, “what really failed here?”  At the risk (pun) of over-simplification:

• Was There an Error on the part of Probability Theory?

After all, Probability Science like all other fields of knowledge is always “advancing” as they say.  So perhaps probability theory is wrong somehow?

I’m personally disinclined to put the blame here, primarily because I would think that there would be evidence from other fields (like Quantum Mechanics) that something is amiss waaaaay before it hit a field like economics.

• Was There Error In The Model Used to Determine Risk?

Some people who understand real estate valuation and complex derivatives and financial risk want to put the blame here.  It’s a little too early to tell, but one thing is for sure - Financial risk is so different from operational risk I couldn’t begin to hazard an opinion on the subject.   But it would seem that this is really somewhere we might look.

• Was There Error In The  Scale Used (Quantitative vs. Qualitative)?

Honestly?  I find it extremely difficult to understand how this could be the source of financial ruin.

• Was There Error on the part of the Decision Maker?

What if all of the above were just fine, and the decision maker chose short term gain over long term stability?  What if this was (to simplify the matter greatly) a choice of “heads” over “tails” and the coin landed on tails?  What if the model represented the right risk (probability of negative outcome vs. positive outcome), but the complex derivative was sold to someone else who had poor “risk management” (ability to make a good decisions)?

Now I have no clue about complex derivatives, and I’m oversimplifying to be sure - chances are like most things, there are several problems that helped create the primary cause. But it seems to me that as we go into incident response mode for the economy, it’s more helpful to do so in a rational, logical manner.

OTHER THINGS WE MIGHT WANT TO CONSIDER

Consider the Source
Some authors (who I think tend to exploit outcome and hindsight bias,and then combine those with indirect ad hominem attacks in order to sell their books), are actually putting forth arguments against the use of analytics.  The source of this is a current epistemic debate between those who believe that only falsification is certain, and those who maintain that neither proof nor falsification are certain, there are only probabilities.    So before you go believing any “quadrants” of usefulness on faith - I encourage you to understand what is at the heart of the discussion.

We All Have to Live In The Real World
The sun will rise tomorrow, and someone will try to find the source of the problem and do a better job.  Now chances are, they’ll be doing it in a quantitative manner.  Chances are also that at some point their models will fail and we’ll need to build new ones.  And this will happen whether the field is cosmology, economics, meteorology, information security, or professional baseball.

WHAT ABOUT YOU, ALEX?

I’m far from certain and subject to change, but these days I lean towards Robin Hanson & MIchael Lewis w/regards to placing blame.

When you use a declarative data source to perform a parameterized update that contains string parameters, consider setting ConvertEmptyStringToNull='false' on your <asp:Parameter> elements, because it's true by default! In other words, if a text field contains an empty string, it'll be sent to your declarative data source not as string.Empty, but as null.

Now I don't know about you, but I don't like dealing with nulls if I can avoid it. Especially strings. Unless there's a clear need to have a null state, I avoid them like the plague not only in my database designs but also in my XML schema designs. Hopefully this helps somebody out!

In their eyes, we were not earning money for the bank. Worse, we had the power to say no and therefore prevent business from being done. Traders saw us as obstructive and a hindrance to their ability to earn higher bonuses. They did not take kindly to this. Sometimes the relationship between the risk department and the business lines ended in arguments. . . .

Tactfully explaining why we said no was not our forte. Traders were often exasperated as much by how they were told as by what they were told.

At the root of it all, however, was—and still is—a deeply ingrained flaw in the decision-making process. In contrast to the law, where two sides make an equal-and-opposite argument that is fairly judged, in banks there is always a bias towards one side of the argument. The business line was more focused on getting a transaction approved than on identifying the risks in what it was proposing. The risk factors were a small part of the presentation and always “mitigated”. This made it hard to discourage transactions. If a risk manager said no, he was immediately on a collision course with the business line. The risk thinking therefore leaned towards giving the benefit of the doubt to the risk-takers.

Collective common sense suffered as a result. Often in meetings, our gut reactions as risk managers were negative. But it was difficult to come up with hard-and-fast arguments for why you should decline a transaction, especially when you were sitting opposite a team that had worked for weeks on a proposal, which you had received an hour before the meeting started. In the end, with pressure for earnings and a calm market environment, we reluctantly agreed to marginal transactions.

• Would You Know a Good decision if You Saw One?
• Decision Making Under Uncertainty

ScienceLogic: Mike, what’s your background working with network and management system tools?

Mike Lawson: Before joining Apptis, I worked for the Air Force, mainly in satellite communications for almost nine years. I’m probably most familiar with HP OpenView and BMC Remedy. I managed a team that used them but wasn’t involved in tool selection; like many other federal IT workers, we didn’t have a choice of tools because there were existing enterprise licenses and maintenance contracts.

I also saw a large systems integrator do a full Remedy/Crystal Systems/OpenView installation. It took 6 weeks to stand up and customize to meet just the basic monitoring requirements, and it cost something like half a million dollars. At the time, I thought that wasn’t bad and was a pretty typical experience.

ScienceLogic: Coming from where you did, what’s your take on EM7?

Mike Lawson: Honestly, I didn’t believe that EM7 could really do all that it claimed. In many ways, it was the complete opposite of what I had seen first-hand with other monitoring solutions. Could it really cover that much functionality? At relatively much lower cost to the customer and without the licensing nightmare?

That quickly changed when I needed to understand the system enough to run it at a customer’s site. I went back over the training docs I received during my initial training class and jumped in; now, 6 months later, I’m the EM7 expert and can tell you that it delivers on all those promises. (But I still need to show people to get them to believe it too)

I preach the “EM7 gospel” and when anyone wants to talk monitoring, I ask about the universal pain points: cost, maintenance contracts and licensing, and then I explain EM7. The cost difference is real; the solution is based on capacity, so there’s no licensing and it’s easy to use. They are shocked to learn that they can buy multiple EM7 appliances and years of maintenance for what they paid for most other tools.

ScienceLogic: Apptis won the contract for monitoring aboard the USNS Mercy. We love that you’re using EM7 for one of the Navy’s hospital ships. Can you tell us more?

Mike Lawson: The USNS Mercy is a Military Sealift Command hospital ship. Some stats:

• 849 feet long (nearly the size of a football field)
• 12 fully-equipped operating rooms, a 1,000 bed hospital facility, digital radiological services, a diagnostic and clinical laboratory, a pharmacy, an optometry lab, a CAT scan and two oxygen producing plants
• Crew: 61 civilian mariners, 956 Naval medical staff, and 259 Naval support staff

The USNS recently departed on a five-month humanitarian mission in the Western Pacific and Southeast Asia in support of Pacific Partnership 2008. The partnership provides international medical, dental and engineering teams this summer to provide humanitarian support and conduct joint, combined, and cooperative Civil-Military Operations in order to improve regional stability and build partner capacity to respond to natural disasters and pandemic.

For the most part, the ship’s network is self-contained, but can also use a landline when docked. The network covers 400 devices, including Windows/Exchange servers and VMware for server virtualization. Prior to using EM7, none of the monitoring was integrated; each system was independently monitored through individual vendor-specific consoles.

Out of the box, EM7 provided integrated systems, application and network management for all network gear, applications and virtual machines in one solution. We didn’t have to do a lot of customization – EM7 includes best-practice based thresholds, event and monitoring templates and this covered what USNS Mercy needed to monitor.

ScienceLogic: You’re a systems integrator with a very useful “customer point of view” when it comes to looking at tools. From that perspective, can you share what you think are the biggest benefits that EM7 provides?

Mike Lawson: First of all, EM7 stands up right away. We’re talking days, not weeks. In contrast to the lengthy installation of OpenView and Remedy I witnessed during my military career, I was able to configure, customize, and implement the EM7 solution for the USNS Mercy in three days.

Second, it’s easy to train people on and the support is outstanding. This judgment is from first-hand experience. Right before the USNS Mercy departed on its latest voyage, the system administrator I had trained on EM7 left, so I had all of a day to train some new EM7 admins. I prepared a seven-page “cheat sheet” and over a 3-hour conference call, we walked through the entire EM7 solution; I haven’t gotten a support call since.

And when a problem did crop up with a device being discovered incorrectly, ScienceLogic was very responsive. We contacted ScienceLogic support on a Saturday and they created and emailed us a video to help troubleshoot the same day. Within 30 seconds of watching the video, the problem was resolved.

Finally, EM7 helps us be good stewards of the government’s money. This is very important to me personally and to Apptis as a company. Because EM7 is cheaper and deploys so quickly and easily, you might think that it’s just the opposite of what a system integrator would want to use. But that’s short-term thinking. We believe in deliver the most value for customers every time. It’s what creates trust and long-term relationships with our customers. Instead of that half million spent on standing up the solution and basic setup, I’d much rather (and I know the customer would rather) spend that on fine-tuning or extending the solution to do much, much more.

As a former government employee, I know what it’s like to use a tool that doesn’t fit my needs. EM7 proves that the best solution can totally break the old model of costly, lengthy installations. EM7 has the right model: the right solution and the right price delivered as an appliance that is easy to deploy, train on and use.

ShareThis

Don't worry, they won't be.

The key to understanding this is that this sort of contractual liability is part of a contract, and with free software -- or free anything -- there's no contract. Free software wouldn't fall under a liability regime because the writer and the user have no business relationship; they are not seller and buyer. I would hope the courts would realize this without any prompting, but we could always pass a Good Samaritan-like law that would protect people who distribute free software. (The opposite would be an Attractive Nuisance-like law -- that would be bad.)

There would be an industry of companies who provide liabilities for free software. If Red Hat, for example, sold free Linux, they would have to provide some liability protection. Yes, this would mean that they would charge more for Linux; that extra would go to the insurance premiums. That same sort of insurance protection would be available to companies who use other free software packages.

The insurance industry is key to making this work. Luckily, they're good at protecting people against liabilities. There's no reason to think they won't be able to do it here.

I've written more about liabilities and the insurance industry here.

This week Amazon’s Simple Storage Service (S3) suffered a major outage that affected several websites that rely on the service. This is actually the second major outage for Amazon S3 this year. As a result of these and other reported outages, some companies will come to question whether they should pursue these new cloud-based services in the future. I agree with Nick Carr, whether you’re a startup looking to rely on the cloud almost exclusively for computing power and storage capacity or you’re a brick and mortar company who may want to use SaaS services for CRM or an online backup service, these outages should not scare companies away from cloud-based services. Outages are inevitable; no one, not the most sophisticated internal IT shops on Wall Street, or the largest service providers can offer 100% availability all the time. Amazon threw everything it had to fix the problem and was able to address the outage in several hours. How well would you be able to execute on your disaster recovery plan if you had a major outage?

Instead of avoiding cloud-based services, organizations need to be savvier about security and resiliency of the service provider. In fact, your organization may already be in pursuit of these services. Online backup is becoming a viable alternate to premise-based solutions for PC backup as well as remote office backup. Next will be a number of services related to information management such as online archiving and online records management and more online storage offerings to support low cost storage. Further down the road, there will also be hosted, multi-tenancy Exchange solutions. Get involved in these discussions. Don’t take it for granted that the potential service provider has hardened data centers that meet Tier III or Tier IV classifications (these classifications describe data center site infrastructure and topology, Tier IV is the highest rating), that your data is replicated to another data center, that your data is encrypted in flight and at rest and that the service provider has strong security measures in place so that administrators can support the infrastructure but not access or even see your organization’s information. Organizations should have consistent processes before, during and after the contracts have been signed.

And, when you ask about SLAs regarding resiliency, keep in mind that there will be some downtime for routine maintenance and that some unplanned downtime is inevitable. Consider a service provider that might boast about 99.9% availability (8 hours/year outage for 24x7). What is the difference between the following?

· 8 AM to 4 PM on the last Friday of the quarter

· Biweekly outages of 30 min at 4 AM local time

Timing and duration are more important than total downtime/outage.

Get involved in these discussions but be careful not to come off as the obstacle or as the doomsayer. Quite the opposite, you want to be seen as the enabler. Help the organization understand some of the potential risks but then help the organization define its resiliency requirements, security requirements, and risk tolerance. When the organization knows this, it can more confidently go out and select the right service provider, negotiate the appropriate SLAs and be prepared ahead of time with contingency plans for any potential service outages.

LogLogic Lunch and Learn Presentation
- "Worst Practices" of Log Management
- Speaker: Dr. Anton Chuvakin, GCIA, GCIH, GCFA
- Wednesday, July 23rd, 2008 * 12:30pm - 1:15 pm

Want to learn all the embarrassing mistakes and pitfalls that await you on the path to log management nirvana? Attend "'Worst Practices' of Log Management" presentation by LogLogic's Logging Evangelist Dr Anton Chuvakin that covers all the things that can go wrong while planning, evaluating, deploying and running a log management solution. Insufficient planning, unrealistic expectations, choosing tools on price alone, lack of logging configuration guidance are among such "worst practices." Each common "worst practice" will be accompanied by suggestions to avoid the errors and do things correctly! Everybody touts "best practices", but this is the place to learn how to avoid the opposite - and have fun in the process.

if you want to meet, drop me an email/call or just show up for "lunch and learn." Unfortunately, I am going back right after my presentation tomorrow...