We've probably all heard that you can't secure what you can't see and that statement is even more profound when it comes to virtual environments.  This is because it is extremely challenging to see what is going on at a micro vs. macro level within a virtual environments network.  The virtualization vendors such as VMWare and Citrix have provided embedded tools into their management consoles that show a macro level of visibility but its not enough to identify security events in the environment.  Take a look at the attached picture.  It simply shows VMWare's ability to monitor virtual network performance statistics from a bits per second perspective.

<-Click To Enlarge

With only this level of detail how can one determine which network applications are causing spikes.  Is it FTP traffic that is occuring at a high volume at an unuseal time of day?  If that were occuring, could that be indicative of either a breach or some sort of problem? What if FTP isn't even an authorized service in the virtual environment but there is a high volume of it?  Did someone install a rouge FTP service so they could steal information from the server at will?

These types of questions can't really be answered without a micro level of detail into the packets flowing in, out and within the virtual environment.  Now, what I am highlighting is not security in the traditional sense of prevention but using visibility as a means to first identify, then pin point the source of an issue so that it can properly be mitigated.  Having constant visibility can also ensure that other security products in the environment are performing as expected.  What if a Montego HyperSwitch with firewalling enabled is configured with many policies but someone forgot to create an FTP block policy.  One could think they are protected from rouge FTP services transmiting data out of the network, but without constant visibility monitoring, can you be certain?

Some vendors, namely Reflex Security will get you to believe that their IPS / IDS solution that is inline and running in the virtual environment is the right and only approach.  Or they will tell you to hang a virtual IDS off a span port in the virtual environment and you will at least have visibility into the attacks that are taking place.  Well, sure... You now have attack visibility but at the performance cost of your virtual environment.  Signature matching technologies are great, I'm a huge believer; however they don't scale very well in shared computing environments such as virtual ones.  IDS systems also don't typically track protocol and network service (FTP, HTTP, etc.) utilizations; which is another important part of visibility.

So, what do we do to gain visibility without the performance headache?  Well, for starters its probably best to put your IDS/IPS solutions in the physical environment where performance will be less of a concern.  In fact, you can span a virtual switch's traffic out to a physical NIC as easy as you can to a virtual one.  So why do it virtual and have to pay a 60% CPU utilization tax?  Another solution is to IDS inspect only the things you care about.  Why IDS inspect SSL traffic if you know your solution can't unencrypt SSL.  Its just a waste of compute cycles isnt it?  Policy based switching helps you with directing only the things you care about to an IDS (attack visualization product).  Montego's HyperSwitch also can help you with the traffic redirection of only the things you care about.

Another method of visibility which I tend to be a fan of is one of packet analysis (aka NetFlow).  NetFlow was invented by Cisco some time ago and has gained popularity in the physical world and definately has a use in the virtual world.  NetFlow is lightweight.  Let me say that again, its light weight!  It only sends a summation of packet detail to an analytical engine which can do some number crunching, packet comparison, etc. etc. to make some sense out of whats going on.  Lancope, an Atlanta based visibility company that provides Network Visibility, Security Visibility and User Visibility has this tool on their website that is a Netflow Bandwidth calculator.  You'll see from playing with this ( http://www.lancope.com/netflowcalculator.aspx ) calculator that it doesn't consume a lot of network bandwidth to transmit these network accounting records.  It also doesn't cause a lot of CPU overhead to send these records to an analytical engine sitting somewhere in the network.

Lancope's analytical engines have the ability to do the following for you within your virtual environment:

...and probably a slew of other things I'm not aware of.  A screen shot of their product is bellow:

<- Click to enlarge

You'll notice from the screenshot that you are able to visualize who is talking to who, how much traffic they have sent and received and something called a concern index (not seen on this screenshot).

Now, a concern index is a number that increases as Lancopes analytical engines monitor suspicious activity on a session.  A high counter can be indicative of a security problem.  Its another way of identifying (visualizing) compromised hosts (virtual machines) without having to do signature matching like a heavy weight IPS engine.  Example:  Lets say you have a VM that has a BOT on it and is "owned".  The Lancope product is monitoring this long life session.  Let's say that session is established for several hours or maybe even days or months.  Lets also say that the conversation appears to be mostly unidirectional from a public ip address not belonging to your enterprise.  Lancope would increase a the concern index on this since this server hasn't typically had this type of behavior.  Once the concern index reached a certain level it could then fire off an email, send you a text message or something saying:  Warning, Warning, Danger, Danger Will Robinson!!! You're virtual server may be infected with a BOT, please investigate immediately!!!

This example is VISIBILITY which helps you with SECURITY.  There are a number of other things you can do with NetFlow and Lancope products that have less to do with security and more to do with operational efficiencies.  Things like, helping you answer questions of:  How do I know what network applications are taking up the most bandwidth?  When should I move those applications over to a server with more horsepower?  When did these VM's vmotion over here and was there a traffic condition / CPU condition that caused that to occur?  I could go on and on but thats a topic for another blog entry.

So, my suggestion is to take a look at what NetFlow has to offer.  Montego Networks supports NetFlow transmission and Lancope supports NetFlow analytics and with both you can regain what was lost visibility.

I hope this was helpful to you all!

-John Peterson

We've probably all heard that you can't secure what you can't see and that statement is even more profound when it comes to virtual environments.  This is because it is extremely challenging to see what is going on at a micro vs. macro level within a virtual environments network.  The virtualization vendors such as VMWare and Citrix have provided embedded tools into their management consoles that show a macro level of visibility but its not enough to identify security events in the environment.  Take a look at the attached picture.  It simply shows VMWare's ability to monitor virtual network performance statistics from a bits per second perspective.

<-Click To Enlarge

With only this level of detail how can one determine which network applications are causing spikes.  Is it FTP traffic that is occuring at a high volume at an unuseal time of day?  If that were occuring, could that be indicative of either a breach or some sort of problem? What if FTP isn't even an authorized service in the virtual environment but there is a high volume of it?  Did someone install a rouge FTP service so they could steal information from the server at will?

These types of questions can't really be answered without a micro level of detail into the packets flowing in, out and within the virtual environment.  Now, what I am highlighting is not security in the traditional sense of prevention but using visibility as a means to first identify, then pin point the source of an issue so that it can properly be mitigated.  Having constant visibility can also ensure that other security products in the environment are performing as expected.  What if a Montego HyperSwitch with firewalling enabled is configured with many policies but someone forgot to create an FTP block policy.  One could think they are protected from rouge FTP services transmiting data out of the network, but without constant visibility monitoring, can you be certain?

Some vendors, namely Reflex Security will get you to believe that their IPS / IDS solution that is inline and running in the virtual environment is the right and only approach.  Or they will tell you to hang a virtual IDS off a span port in the virtual environment and you will at least have visibility into the attacks that are taking place.  Well, sure... You now have attack visibility but at the performance cost of your virtual environment.  Signature matching technologies are great, I'm a huge believer; however they don't scale very well in shared computing environments such as virtual ones.  IDS systems also don't typically track protocol and network service (FTP, HTTP, etc.) utilizations; which is another important part of visibility.

So, what do we do to gain visibility without the performance headache?  Well, for starters its probably best to put your IDS/IPS solutions in the physical environment where performance will be less of a concern.  In fact, you can span a virtual switch's traffic out to a physical NIC as easy as you can to a virtual one.  So why do it virtual and have to pay a 60% CPU utilization tax?  Another solution is to IDS inspect only the things you care about.  Why IDS inspect SSL traffic if you know your solution can't unencrypt SSL.  Its just a waste of compute cycles isnt it?  Policy based switching helps you with directing only the things you care about to an IDS (attack visualization product).  Montego's HyperSwitch also can help you with the traffic redirection of only the things you care about.

Another method of visibility which I tend to be a fan of is one of packet analysis (aka NetFlow).  NetFlow was invented by Cisco some time ago and has gained popularity in the physical world and definately has a use in the virtual world.  NetFlow is lightweight.  Let me say that again, its light weight!  It only sends a summation of packet detail to an analytical engine which can do some number crunching, packet comparison, etc. etc. to make some sense out of whats going on.  Lancope, an Atlanta based visibility company that provides Network Visibility, Security Visibility and User Visibility has this tool on their website that is a Netflow Bandwidth calculator.  You'll see from playing with this ( http://www.lancope.com/netflowcalculator.aspx ) calculator that it doesn't consume a lot of network bandwidth to transmit these network accounting records.  It also doesn't cause a lot of CPU overhead to send these records to an analytical engine sitting somewhere in the network.

Lancope's analytical engines have the ability to do the following for you within your virtual environment:

...and probably a slew of other things I'm not aware of.  A screen shot of their product is bellow:

<- Click to enlarge

You'll notice from the screenshot that you are able to visualize who is talking to who, how much traffic they have sent and received and something called a concern index (not seen on this screenshot).

Now, a concern index is a number that increases as Lancopes analytical engines monitor suspicious activity on a session.  A high counter can be indicative of a security problem.  Its another way of identifying (visualizing) compromised hosts (virtual machines) without having to do signature matching like a heavy weight IPS engine.  Example:  Lets say you have a VM that has a BOT on it and is "owned".  The Lancope product is monitoring this long life session.  Let's say that session is established for several hours or maybe even days or months.  Lets also say that the conversation appears to be mostly unidirectional from a public ip address not belonging to your enterprise.  Lancope would increase a the concern index on this since this server hasn't typically had this type of behavior.  Once the concern index reached a certain level it could then fire off an email, send you a text message or something saying:  Warning, Warning, Danger, Danger Will Robinson!!! You're virtual server may be infected with a BOT, please investigate immediately!!!

This example is VISIBILITY which helps you with SECURITY.  There are a number of other things you can do with NetFlow and Lancope products that have less to do with security and more to do with operational efficiencies.  Things like, helping you answer questions of:  How do I know what network applications are taking up the most bandwidth?  When should I move those applications over to a server with more horsepower?  When did these VM's vmotion over here and was there a traffic condition / CPU condition that caused that to occur?  I could go on and on but thats a topic for another blog entry.

So, my suggestion is to take a look at what NetFlow has to offer.  Montego Networks supports NetFlow transmission and Lancope supports NetFlow analytics and with both you can regain what was lost visibility.

I hope this was helpful to you all!

-John Peterson

Most event processing systems would be incomplete without the ability to process events in the form of messages.   Messages can be delivered in either a connection-oriented protocol or a connectionless protocol.   Most enterprise-class messaging systems have both.   Many messaging systems have features like guarenteed delivery, which are important to many applications.

On the other hand, you do not have to work with a messaging system or enterprise service bus (ESB) to process events, because the transport layer is independent from the event processing layer, theoretically.  Most enterprise-class event processing system architectures will use a combination of both asynchronous and synchronous messaging. 

To understand event processing I recommend you turn to network management and the practical use of Simple Network Management Protocol (SMNP) for a basic undertanding of event processing.   SNMP uses both synchronous event-based messaging, called polling, and asynchronous messaging, called traps.   Network management systems engineers use a combination of both polling and trapping in all enterprise-class operational NMS.  Optimizing polling and trapping is one of the tasks good NMS engineers do well. The same holds true in most distributed event processing architectures.  

For example, look at the CEP/EP reference architecture on this site.  You will notice that the mechanism for event transport is generic, represented as an event bus, but it does not specify the transport protocol.  If you are receiving raw events and comparing correlated results against a signature in a database, you are using both asynchronous and synchronous messaging.    In theory, you could build an event processing system with only connection-oriented protocols, but this would be an exeception, not the rule.

Event processing is generally associated with messaging because we generally represent event-objects as electronic messages.   In theory, we could call these cyber event-objects anything we want; for example, we could call them “packets.” However, packets are generally associated with the underlying Internet Protocol (IP) layer by network engineers.  

Moving up the stack, we think in terms of a complete message-object, which we generally call “a message.”  This message could be an SNMP event-object, an SMTP event-object (an email message), or an HTML request to a web server, to only name a few.    In fact, the basic unit of work at the application level of a distributed network application is what we call “a message.”  

So, in On Messaging and Events Opher asks, “Is event processing just fancy name to message processing ?”

Events are generally represented in some electronic format.  The event-object must be transported electronically in cyberspace, and the way that it is transported is in what network engineers generally call “a message.”   It make no difference what we call it, really; because whatever we call it, it is still binary data representing information we are interested in, hopefully in a format we can efficiently process.    Enterprise-class event processing systems are designed to work with myriad formats, protocols and transports.   One size does not fit all.

• Poison anybody’s DNS cache
• Expose how the actual exploit works

What it does is check whether your ISP’s DNS server is patched. Plain and simple. It looks for one thing — source port randomization. This does not give away the exploit, it checks for the existence of the sledgehammer fix that prevents the exploit from working.

More specifically, there’s some Javascript code that generates a random hex string which is used to create a URL, e.g. http://6313d97e498e.toorrr.com. Your OS then does a DNS lookup for that unique hostname. Your ISP’s DNS server asks toorrr.com’s DNS server (a server Dan controls) to resolve that funky DNS name to an IP address. It sends a few packets in the process. Dan’s server makes a note of the source port of each request and sends back the webserver’s IP address to your DNS server, which sends it back to you.

Now that you have the IP address, your browser can fetch the results page. The web page is generated dynamically by parsing the hex string out of the URL you requested, using Ajax to fetch the relevant port and TXID data stored on Dan’s server, and printing out a “safe” or “vulnerable” message such as:

Your name server, at 71.243.0.38, appears to be safe.
Requests seen for 6313d97e498e.toorrr.com:

71.243.0.38:45298 TXID=13926
71.243.0.38:45310 TXID=25412
71.243.0.38:45338 TXID=30829
71.243.0.38:45332 TXID=13934
71.243.0.38:45321 TXID=2701

That’s all. Nothing tricky. This particular DNS server is deemed safe because the source port varies from one request to the next.

Come to think of it, those source ports don’t really look that random, do they. For anybody “in the know”, is that amount of randomness sufficient to protect against the attack?

Agencies are supposed to demonstrate that they can:

• Transmit IPv6 traffic from the Internet and external peers, through the core (WAN), to the LAN.
• Transmit IPv6 traffic from the LAN, through the core (WAN), out to the Internet and external peers.
• Transmit IPv6 traffic from the LAN, through the core (WAN), to another LAN (or another node on the same LAN).

(Source: OMB IPv6 FAQs)

One year ago, the OMB reviewed the Enterprise Architecture Assessment Framework results and found that six of the twenty-four agencies were on track to achieve the June deadline. Two months ago, there was a good article by Carolyn Marsan Duffy about the status of compliance. Take a look at this article because it seemed like there was a lot of backpedaling going on about meeting the date – using phrases like “we don’t like the term mandate” and “more of a recommendation than a mandate.” At the time, only three agencies were in compliance.

Duffy just wrote an updated article, “Feds say they have aced IPv6 deadline”, and suddenly two months later, all lights seem green. As of June 24, ten of the twenty-four agencies sent emails to the OMB stating that “they have successfully transmitted IPv6 packets”. Fourteen still need to report in, but none have asked for an extension. And all of it was done through the regular tech refresh budget over the past three years. So if this is true, kudos to the feds!

Right around the time of the first not-so-rosy article, we ran a survey at FOSE, the big federal government IT show. We asked attendees if their agencies would be ready by the deadline:

• 33% said they would be ready
• 6% said they were already there
• 33% said they would NOT be ready
• About a quarter didn’t know

What was really interesting is that we asked this same question in 2007, and the audience was equally split (yes/no) on whether or not their agencies would meet the mandate – 1 in 5 (2007) instead of 1 in 3 (2008).

So what can explain these numbers? Surprisingly, out of the attendees we talked to, only 65% of them said that IPv6 is important to their operations, making it second to last on the list of IT priorities covered by the survey. Maybe the answer lies in the relative “unimportance” of the milestone – that just the network backbones (and the routers supporting them) be capable of passing IPv6 packets. The true test for government IT workers will be when actual IPv6 applications must be supported which will impact networks, systems, application and monitoring tools throughout the government.

So was this a nice checklist item for the Bush administration? This initial deadline is the only one for IPv6 mandates from the current OMB incarnation. Actually running IPv6 applications, that’s a whole ‘nother story, apparently for a new administration.

ShareThis

(Photo credit: The Tech Stop)

ScienceLogic: How long have you been involved with Interop?

Geoff Horne: Since about 1996.

ScienceLogic: How has it been changing? Does the show get more complex with new technologies or because of the constantly changing size of the show?

Geoff Horne: The technologies have changed. Every year there’s a different market environment. Since we build on customer needs, things change every year. Things like ScienceLogic for Network Monitoring, for how long have Network Management tools been completely web based? In general, it doesn’t really get any better or worse because every year we’re building it again. You don’t get the stability of a standard environment. The upside is that we’re always doing a full upgrade, a full technology refresh and not using old code.

ScienceLogic: Do those kinds of changes influence the types of vendors you look for for InteropNet?

Geoff Horne: The base categories don’t change. You always need to forward packets. You always need switches, you always need routers. We’ve tried to open it up to everyone that has products involved with networks to see if we have the time or space for it.

ScienceLogic: The kind of cooperation that you get between the vendors is what seems to be an unachievable nirvana for Enterprises. What’s the secret to getting 17 vendors to work together in such a short time? Enterprises would kill for that.

Geoff Horne: The honest answer is don’t trust the vendors. If they try and build something the way they want to, its not going to interoperate. You have to pull them out of their safety zone, make them do things that you think the product can/should do to ensure interoperability.

ScienceLogic: In a blog post prior to Interop Vegas 2008 you stated three major goals for InteropNet. They were Education, Monitoring and Statistics. How did you do against these goals?

Geoff Horne: I think we did pretty well. They’re 3 things we really didn’t have before. They’re things that just weren’t focused on the right way. For the first round of changing the focus, changing the way people look at the network (statistics rather than packets), it worked quite well, it gave people a much better idea as to what’s going on.

ScienceLogic: If we look at NY as take two for Interop 2008, are there things you are going to do differently based on lessons learned in Vegas?

Geoff Horne: We’re building more physical redundancy in the core network, geographic distribution of the infrastructure within the show. This will allow us to bring up chunks of the network independently. It isn’t something that we really thought of before. This helps us take the single point of failure (the NOC) out of the equation.

ScienceLogic: Are there any lessons learned from Interop that you think would help enterprises?

Geoff Horne: Visibility is key. Your network is significantly more functional when more people can see what’s going on. If the only guy that can see what’s going on is the guy with his fingers on the terminal, no one can make good decisions. You have to make people loosen up their control so that everyone can see and therefore make educated decisions.

ShareThis

The new compression technique, called variable bitrate compression produces different size packets of data for different sounds.

That happens because the sampling rate is kept high for long complex sounds like "ow", but cut down for simple consonants like "c". This variable method saves on bandwidth, while maintaining sound quality.

VoIP streams are encrypted to prevent eavesdropping. However, a team from John Hopkins University in Baltimore, Maryland, US, has shown that simply measuring the size of packets without decoding them can identify whole words and phrases with a high rate of accuracy.

The technique isn't good enough to decode entire conversations, but it's pretty impressive.

From the Last Hope:

For Immediate Release

THE LAST HOPE TO FEATURE HACKER RADIO

At The Last HOPE conference, hackers will broadcast their minds and their iPods.

In the center of the summer’s top hacker event will be a small isolation booth. “Radio Statler!” as the station is called, will send out a three day broadcast of all-original material. From the center of Manhattan, around the clock, discussions of the past, present, and future of technology, creativity, and humanity itself will be transmitted.

The first night of the conference, July 18th, the station will carry a program called Digital Music Night, hosted by Peter Kirn, editor of createdigitalmusic.com. The three hour live concert will feature a convergence of artists and musicians using custom, original tools for performing live in new and bizarre ways, including:

* Houseplants hooked up to live computer visuals and music
* A mutant trumpet, halfway between the digital and acoustic worlds
* Packets of data visualized as three-dimensional eye candy
* An animated digital art sketchpad controlled by Wii remote
* A set of digital gloves for gestural DJing
* A robotic drummer
* Computer-generated vocals that sing your spam folder to you
* Live digital art made from vintage game consoles and computers

The station will give additional talk and interview time to the conference’s speakers, broadcast the keynotes and other popular seminars, and offer attendees who don’t speak at the podium a chance to share their ideas. Many hackers who already do their own podcasts are being asked to contribute and do special programs for the conference.

Program and content submissions are still being taken, volunteers are being sought, and the organizers are looking for promotional sponsors to help cover the cost of broadcasting. More information can be found at http://radio.hope.net/ or by emailing projects@hope.net.

Damn, I’ll have to break out Garageband or maybe I’ll have to submit one of these tracks? HA!

What can we conclude?

First, good documentation never hurts :-) - indeed, the most popular information to look for when facing a new log record is documentation on what it means. While some software vendors are great in this regard, many other don't bother documenting their logs or document them only when customers complain.

Second, I was not sure that the second popular choice would be "Other logs from about the same time (this and other systems)."  This strongly points at huge value of cross-device log analysis (see this recent log entry on that),  where all the logs are consolidated and analyzed together (it goes without saying that time is synchronized OR at least corrected across those logs). Indeed, if you are confused about a log and documentation is not available, reviewing "what else was/is going on?" is smart. Trusting log time stamps across many systems is also key for that.

Third, having IP addresses in logs is great, but human-readable names are better: IPs in logs needs to be mapped to DNS or Netbios names. Indeed, given that often such names reveal where the system is, who might own it, what its function is, etc this information is not just a mapping, but true log information enrichment.

Fourth, so, what's next? The above 3 top responses are indeed universally useful, but the next choice digs deeper: flows, packets, connections and other network information does complement logs and is often studied in combination with logs (e.g. see a strange log entry then go see who connected to the system at that time or where the system itself connected to).

Fifth, next comes a group of pretty much everything else: other logs from the same system, logs about the same system as well as loosely defined 'similar' log entries. These come handy, but are not top choices. In fact,  from this I conclude that a lot of additional context information is needed to make sense of a confusing log entry.

Sixth, what was surprising? I thought that identity lookups (e.g. IP to real name or other user identity information) would score higher.  I also suspect that people were confused by "logs ABOUT the same systems" (what I meant is, for example, use firewall logs that mention the system which log we are now analyzing) and this should score higher.

Seventh, anything fun in the "Other" category? Yes, there were a few insightful ones: first, results of a Google search (supposedly for the info from the log entry in question)! Very true indeed. Also named were logs from the same daemon/program (how can I miss it?),  logs from previous incidents and information on the logging system owner.  All very useful indeed. Thanks for good ideas!

Finally, a brief message to people that work for a certain log-related vendor of ill repute who keep polluting my polls: if I catch you, I will kick you in the butt :-) Or, better, I will hammer you with a big and heavy log (you know, the wooden kind) over your miniscule heads ...

Past logging polls and their analysis: