One of the more interesting session I went to yesterday was a talk by Chris Hoff called "The Four Horsemen of the Virtualization Apocalypse."  (If you've never read Hoff's blog, you should check it out at http://rationalsecurity.typepad.com/.)

I thought I was keeping a close eye on security and virtualization issues, but this talk illustrated how wide and varied the topic really is.  This was not about Blue Pill and it wasn't about having security monitors in the hypervisor - instead he focused on how virtualizing physical devices (e.g. switches, systems) will cause lots of problems for security architects and administrators.

Briefly, here are the four horsemen:

• Conquest - Translating your physical capacity planning implementation to virtual devices probably won't work.
• Death - Virtualized networks lack several physical attributes assumed by security applications and high-availability devices today - you'll probably have to re-architect it all to get the same functionality, which might not even be possible in your new virtual world
• War - Adding security VAs takes away precious resources that could have been used to dynamically add VMs.  It is a war of resources.
• Famine - With all of the redesigning and accommodation happening, security costs are going to eat into any savings you make on server consolidation.

Now, if you want to read the much more thorough version, see Hoff's original post here.

Okay, how does this all relate to the title of my post?  Not much.  However, much later on day one, things really started rolling.

After being crowded out of the Shadow Bar, a bunch of us ended up over at Casa Fuente (A cigar bar in Caesars forum).  Five minutes after arriving, someone spilled a drink in my lap, big fun!  It turns out that it was Stepto's birthday, and Hoff makes sure everyone has a drink and we all sing happy birthday to Stepto.  Check out part of it, courtesy of Jack Daniel:

Immediately after the toast, Jennifer Jabbusch knocks over a table, falls to the floor and begins having a seizure. Stepto rushes over, trying to help, and just about that time, she flips over and starts laughing - total fakeout! Everybody bursts out laughing.

Shortly after that, they closed for the night and kicked us out and we all headed over to Cleopatra's Barge. There weren't enough seats or tables for us, but I noticed that the "reserved" barge seating was empty. Drawing upon a clever technique (i.e. sometimes called "asking") I social engineered a waitress into letting us have the reserved area. Within mere minutes, several security geeks are on the dance floor, doing us proud.

This leads me to the Four Horsemen of Cleopatra's Barge.  (Though I was out there too, I am excluding myself since simply because I can.)

• JJ, for leadership
• Hoff, who owned the dance floor.
• Ryan Naraine, for getting low, low, low
• David, for letting his hair down.

Though our collective dancing does not signal the end of the world, it certainly capped an excellent day

ScienceLogic: How long have you been involved in InteropNet?

Katsev: I started at Coyote Point 3 years ago and InteropNet 2006 was my first “big” assignment.  This was the first time Coyote Point had put in a proposal to participate, so we were very excited when we were selected.

ScienceLogic: How long has Coyote Point been involved in Interop overall?

Katsev: We’ve been exhibiting at Interop for a number of years, and after seeing the InteropNet in action, we decided to submit a proposal in ‘06.  We were actually one of the first companies in the load balancing/traffic management space (we’ve been doing this for almost 10 years), so we have a lot of experience to share with InteropNet.

ScienceLogic: What is your role at Coyote Point?

My official title is “Engineering Project Manager”.  Basically, that means that I’m in charge of product releases and maintenance.  It sounds like a weird title for someone participating in InteropNet, but I’ve actually found it extremely useful since my position means that I don’t get to see our systems out in the field a lot.  We’ve added several features and have ideas for others just from my experiences at InteropNet.

ScienceLogic: What do the Coyote Point products do?

Katsev: Coyote Point makes a Traffic Management appliance called Equalizer.  What this means is that any traffic destined for a datacenter’s servers goes through our appliances and we make sure that the server which is best equipped to handle it, does.  Our systems sit between the clients and the servers and monitor the client traffic and the state of the servers.  If the clients start sending more traffic, we’ll balance it out so that no server is overloaded.  If one of the servers stops responding or starts responding very slowly, we’ll steer traffic away from that server.

ScienceLogic: In what way are your products being used as part of InteropNet?

Katsev: In the InteropNet, we’re utilizing a lot of our expertise:  We’re making sure that traffic is balanced and servers are redundant for show services such as DNS and SMTP.  We’re also using our geographic load balancing technology to ensure that the ScienceLogic EM7 appliances and some other internal NOC services are available from anywhere, with the lowest latency, with our SSL acceleration and GZIP compression technology.  Finally, we’re helping logistics in the NOC by allowing a physical separation between systems located in the NOC and those in an emergency rack outside of the NOC.  If either of these two locations were to fail, the network will continue operating without a glitch.

ScienceLogic: Are there any special considerations for Interop that cause you to deploy your systems there differently that any other place?

Katsev: Interop is definitely different than most of our customer installations.   One difference from a standard environment is that the network (at least this year) is one large flat network, with pieces carved out where extra security is needed.  Because of this, we can actually run our failover pairs of Equalizer systems in a non-standard configuration where the two peers are in different racks, or even on different floors.  That’s one of the things that I really like about InteropNet — it definitely brings new ideas to mind, which end up becoming ’special configuration’ white papers after the show.

ScienceLogic: Has InteropNet taught you anything that caused you to actually change your product?

Katsev: In addition to the failover configuration differences I mentioned above, participating in InteropNet has actually caused us to add several new features and allowed configurations.  One example is the “no-spoof” option for Layer 4 clusters.  Prior to the 2006 shows, we always ’spoofed’ the client’s IP address when talking to a server so that the server would see the client’s IP address instead of our own.  At Interop, we ran into a special configuration which would’ve been very difficult to set up in this manner, so our engineers added this feature, and it’s been very a very popular configuration with our customers ever since.

We have also had a couple of business relationships that extended outside of the show.  In 2006, we had a good experience using Spirent Communications gear to benchmark the network, so we ended up purchasing a couple of these systems to test our products.  More recently, we have found a way to bundle our Equalizer e350si load balancers with the ScienceLogic EM7 collector appliances to help ScienceLogic get the best performance in load balancing large quantities of syslog messages to be processed.  If it wasn’t for our participation in InteropNet, neither of these relationships would’ve happened.

ScienceLogic: What’s the best part of being involved with InteropNet?  What do you most look forward to?

Katsev: InteropNet is an amazing networking opportunity (no pun intended).  The group of engineers that put the network together every year is, well, amazing.  There is so much combined experience that any question instantly has several possible answers, and the best answer is chosen very quickly.  One of the ’sayings’ at Interop is “if you run into a problem, ask someone… we’ve probably seen that problem before… five times.”  One would think that being part of InteropNet is the same thing, year after year.  However, in the two years that I’ve been part of this (for four shows), there have been huge differences in the way that the network is designed and put together.  These are both because the vendors selected every year are different, and because the engineers who design the network change from year to year.  Somehow, though, when all is said and done, we have a network that works.

ScienceLogic: You don’t have to answer this one if you’re not comfortable… What would you like to see changed with the way things are done at InteropNet?

Katsev: This isn’t a cop-out… I really can’t think of anything I would do differently.  Sure, there are small problems that pop up sometimes, but every project has those, and the people at InteropNet are more than capable of figuring them all out.  In fact, I know that Interop started out as a show to test the interoperability of devices… but I’m still amazed that all of these devices actually talk to each other and “play nice” together.

ShareThis

What’s more interesting to me, now that I’ve digested the big secret, is how this whole situation has played out in the security community.

The security community has been polarized for the past two weeks, not so much over the technical details being withheld, but about Dan’s plea that people not speculate about the vulnerability. As many pointed out, the “bad guys” won’t stop trying to figure it out just because the “good guys” keep quiet. To be honest, my own lack of public speculation wasn’t because I agreed with the philosophy; I just wasn’t smart enough to figure out the vulnerability myself.

People implied — or stated outright — that Dan just didn’t want anyone stealing his thunder. Considering the timing of the release and the subsequent BlackHat talk, it’s obvious why such accusations were made. Personally, I think it’s a little of each. I believe the coordinated patch effort was undertaken with the best of intentions, but I also think Dan relished some of the glory and media attention as well. It’s hard to blame him for that; if you were in his shoes, wouldn’t you want some recognition too?

By many accounts, dealing with the DNS vulnerability from the operational side has been an exercise in frustration. Plenty of IT people wanted to patch but couldn’t get approval without being able to justify the operational risk. “Because Dan said so” is apparently not a convincing enough argument. Some wondered why the people who were responsible for creating the problem should be blindly trusted to implement an appropriate fix?

Ultimately, vulnerability disclosure is a minefield. No matter how you choose to disclose, somebody will always disagree.

P.S. If you didn’t figure out the title of the post by now, Nate was one of the unlucky few to draw the same timeslot at BlackHat as Dan Kaminsky.

What’s more interesting to me, now that I’ve digested the big secret, is how this whole situation has played out in the security community.

The security community has been polarized for the past two weeks, not so much over the technical details being withheld, but about Dan’s plea that people not speculate about the vulnerability. As many pointed out, the “bad guys” won’t stop trying to figure it out just because the “good guys” keep quiet. To be honest, my own lack of public speculation wasn’t because I agreed with the philosophy; I just wasn’t smart enough to figure out the vulnerability myself.

People implied — or stated outright — that Dan just didn’t want anyone stealing his thunder. Considering the timing of the release and the subsequent BlackHat talk, it’s obvious why such accusations were made. Personally, I think it’s a little of each. I believe the coordinated patch effort was undertaken with the best of intentions, but I also think Dan relished some of the glory and media attention as well. It’s hard to blame him for that; if you were in his shoes, wouldn’t you want some recognition too?

By many accounts, dealing with the DNS vulnerability from the operational side has been an exercise in frustration. Plenty of IT people wanted to patch but couldn’t get approval without being able to justify the operational risk. “Because Dan said so” is apparently not a convincing enough argument. Some wondered why the people who were responsible for creating the problem should be blindly trusted to implement an appropriate fix?

Ultimately, vulnerability disclosure is a minefield. No matter how you choose to disclose, somebody will always disagree.

P.S. If you didn’t figure out the title of the post by now, Nate was one of the unlucky few to draw the same timeslot at BlackHat as Dan Kaminsky.

To solve a complex problem in the blackboard-style, the intelligent agents cooperate as functional specialists, observing updates to the blackboard and self-actualizing in an event driven process) when there is new information to process.  Agents continually update the blackboard with partial solutions when the agents capabilities for processing match the state of the blackboard. 

The blackboard architecture is a distributed computing model for a metaphor describing how people work together to collaboratively solve a problem around a blackboard (whiteboard in todays lingo).   For example, one person is standing at the whiteboard working on a solution while three other people are sitting (or standing) around watching.   One of the observers sees new information on the whiteboard, thinks of how he (or she) can contribute, and then jumps up, takes the whiteboard marker from the person working, and adds to the solution.  This process is repeated in various scenarios.  

The blackboard architecture can be very effective in solving complex distributed computing problems, including event processing problems; however, scheduling the self-actuating agents can be a key challenge.   Another core challenge is how to model and manage the blackboard itself, especially in distributed blackboard architectures.  

John McManus, former CTO of NASA, wrote an excellent PhD dissertation in 1992,  Design and Analysis Techniques for Concurrent Blackboard Systems, at the College of William and Mary, addressing challenges in BB systems.

The table below lists two books that focus on blackboard architecture:

One of the thought leaders in blackboard architecture is Daniel D. Corkill a professor at the University of Massachusetts Amherst. 

Blackboard architecture is relevant to the field of event processing, and in particular complex event processing.   I will go into more details in future blog posts on this topic, including how blackboard architectures relate to grid computing, distributed object caching (of the blackboard), and CEP.

Well, no, security dweebs, we’re on a public policy kick, probably will be until the end of the year (more on that to follow, stay tuned), so you wouldn’t be so lucky.

The Plum Book’s official title is Government Policy and Supporting Positions and basically it’s a huge staffing chart for the Senior Executive Service–the political appointees.  Congress publishes the Plum Book after each presidential election, so for those of us who remember our civics lessons in high school, that would be every 4 years, and the last one was published in 2004.

In fact, you can see the last edition here.  Caveat:  it’s dry, like the uber-trocken Franken white wine that grows in the fields around where I used to live in Germany–so dry that it sucks the moisture right out of you.

Plum Pickin photo by Secret Tenerife

Now why do we care about the Plum Book?  Well, that’s a good question.  Have a look at some of the staffing plans in the plum book, and you’ll see something missing:  Agency CISOs.

Now, I’m not a rocket scientist on org charts, but it seems to me that unless you put CISOs up to where they’re answerable to the agency head, they’re just a cost center inside the IT department with no visibility to the decision-makers.  Once again, we’ve crippled our security staffs like the old-school way of doing things.

On another note, taking a quick straw poll of the agency CISOs that I know, I think about half of them are political appointees, and half of them are GS-15s.  So what’s the difference?

Well, political appointees (SES) are appointed by the President.  They make a better target because they have much more visibility from the higher-ups they are more political in nature.

GS-scale employees are civil service careerists.  Usually these are the guys who have moved up the ranks in the various agencies and know quite a bit of things.

Which is better?  Well, if you want survivability, then GS-scale is the way to go.  If you want to make the most difference, SES is the ticket.

Most of us will never get the choice. =)

Bookmark to:

The good news is that due to the nature of this problem, it is extremely difficult to determine the vulnerability merely by analyzing the patches; a common technique malicious individuals use to figure out security weaknesses.

The typical response I heard was “what do you mean, it can’t be reverse engineered? I’ll just look at the diffs!”

In hindsight, after examining the BIND diffs (yes, I did it too) and discussing with colleagues, all most people saw was UDP source port randomization and a better PRNG for generating the transaction ID, the latter of which would appear to be related to Amit Klein’s cache poisoning attack from about a year ago.

What Rich was really saying is that you can reverse engineer the patch until you’re blue in the face, but that won’t reveal the specifics of the vulnerability.

Dan’s blog post this morning appeared to confirm that interpretation:

DJB was right. All those years ago, Dan J. Bernstein was right: Source Port Randomization should be standard on every name server in production use.

There is a fantastic quote that guides a lot of the work I do: Luck is the residue of design. Dan Bernstein is a notably lucky programmer, and that’s no accident. The professor lives and breathes systems engineering in a way that my hackish code aspires to one day experience. DJB got “lucky” here — he ended up defending himself against an attack he almost certainly never encountered.

Such is the mark of excellent design. Excellent design protects you against things you don’t have any information about. And so we are deploying this excellent design to provide no information.

To translate the fix strategy into a more familiar domain, imagine large chunks of Windows RPC went from Anonymous to Authenticated User only, or even all the way to Admin Only. Or wait, just remember Windows XPSP2 :) This is a sledgehammer, by design. It cuts off attack surface, without necessarily saying why. Astonishingly subtle bugs can be easily hidden, or even rendered irrelevant, by a suitably blunt fix.

Nate McFeters appears to think that Tom Ptacek has figured it out. I’m going to go out on a limb and say that Tom didn’t figure anything out yet but still wanted to write a pithy blog post. I think that if Tom had figured it out, he would have written it down privately and posted the SHA-1 hash, as is the trendy thing to do these days.

Speculation aside, the title of Tom’s blog entry, Dan Kaminsky could have made hundreds of thousands of dollars with this DNS flaw!, does make an important point — Dan didn’t sell the details to ZDI, he used his influence and reputation to coordinate a massive vendor patch effort. That’s an admirable move.