ScienceLogic: Mike, what’s your background working with network and management system tools?

Mike Lawson: Before joining Apptis, I worked for the Air Force, mainly in satellite communications for almost nine years. I’m probably most familiar with HP OpenView and BMC Remedy. I managed a team that used them but wasn’t involved in tool selection; like many other federal IT workers, we didn’t have a choice of tools because there were existing enterprise licenses and maintenance contracts.

I also saw a large systems integrator do a full Remedy/Crystal Systems/OpenView installation. It took 6 weeks to stand up and customize to meet just the basic monitoring requirements, and it cost something like half a million dollars. At the time, I thought that wasn’t bad and was a pretty typical experience.

ScienceLogic: Coming from where you did, what’s your take on EM7?

Mike Lawson: Honestly, I didn’t believe that EM7 could really do all that it claimed. In many ways, it was the complete opposite of what I had seen first-hand with other monitoring solutions. Could it really cover that much functionality? At relatively much lower cost to the customer and without the licensing nightmare?

That quickly changed when I needed to understand the system enough to run it at a customer’s site. I went back over the training docs I received during my initial training class and jumped in; now, 6 months later, I’m the EM7 expert and can tell you that it delivers on all those promises. (But I still need to show people to get them to believe it too)

I preach the “EM7 gospel” and when anyone wants to talk monitoring, I ask about the universal pain points: cost, maintenance contracts and licensing, and then I explain EM7. The cost difference is real; the solution is based on capacity, so there’s no licensing and it’s easy to use. They are shocked to learn that they can buy multiple EM7 appliances and years of maintenance for what they paid for most other tools.

ScienceLogic: Apptis won the contract for monitoring aboard the USNS Mercy. We love that you’re using EM7 for one of the Navy’s hospital ships. Can you tell us more?

Mike Lawson: The USNS Mercy is a Military Sealift Command hospital ship. Some stats:

• 849 feet long (nearly the size of a football field)
• 12 fully-equipped operating rooms, a 1,000 bed hospital facility, digital radiological services, a diagnostic and clinical laboratory, a pharmacy, an optometry lab, a CAT scan and two oxygen producing plants
• Crew: 61 civilian mariners, 956 Naval medical staff, and 259 Naval support staff

The USNS recently departed on a five-month humanitarian mission in the Western Pacific and Southeast Asia in support of Pacific Partnership 2008. The partnership provides international medical, dental and engineering teams this summer to provide humanitarian support and conduct joint, combined, and cooperative Civil-Military Operations in order to improve regional stability and build partner capacity to respond to natural disasters and pandemic.

For the most part, the ship’s network is self-contained, but can also use a landline when docked. The network covers 400 devices, including Windows/Exchange servers and VMware for server virtualization. Prior to using EM7, none of the monitoring was integrated; each system was independently monitored through individual vendor-specific consoles.

Out of the box, EM7 provided integrated systems, application and network management for all network gear, applications and virtual machines in one solution. We didn’t have to do a lot of customization – EM7 includes best-practice based thresholds, event and monitoring templates and this covered what USNS Mercy needed to monitor.

ScienceLogic: You’re a systems integrator with a very useful “customer point of view” when it comes to looking at tools. From that perspective, can you share what you think are the biggest benefits that EM7 provides?

Mike Lawson: First of all, EM7 stands up right away. We’re talking days, not weeks. In contrast to the lengthy installation of OpenView and Remedy I witnessed during my military career, I was able to configure, customize, and implement the EM7 solution for the USNS Mercy in three days.

Second, it’s easy to train people on and the support is outstanding. This judgment is from first-hand experience. Right before the USNS Mercy departed on its latest voyage, the system administrator I had trained on EM7 left, so I had all of a day to train some new EM7 admins. I prepared a seven-page “cheat sheet” and over a 3-hour conference call, we walked through the entire EM7 solution; I haven’t gotten a support call since.

And when a problem did crop up with a device being discovered incorrectly, ScienceLogic was very responsive. We contacted ScienceLogic support on a Saturday and they created and emailed us a video to help troubleshoot the same day. Within 30 seconds of watching the video, the problem was resolved.

Finally, EM7 helps us be good stewards of the government’s money. This is very important to me personally and to Apptis as a company. Because EM7 is cheaper and deploys so quickly and easily, you might think that it’s just the opposite of what a system integrator would want to use. But that’s short-term thinking. We believe in deliver the most value for customers every time. It’s what creates trust and long-term relationships with our customers. Instead of that half million spent on standing up the solution and basic setup, I’d much rather (and I know the customer would rather) spend that on fine-tuning or extending the solution to do much, much more.

As a former government employee, I know what it’s like to use a tool that doesn’t fit my needs. EM7 proves that the best solution can totally break the old model of costly, lengthy installations. EM7 has the right model: the right solution and the right price delivered as an appliance that is easy to deploy, train on and use.

ShareThis

This happened recently to some Amazon S3 customers. There were complaints in the AWS forums about ‘S3 Corruption’. The first post in the forum was recorded at Jun 22, 2008 5:05 PM PDT (although in subsequent posts some people reported emailing Amazon prior to this):

we are having some serious S3 issues.

all data we store on S3 has gone through the same code path for months. starting a couple days ago a small percentage of the objects we are retrieving are not checksumming to the correct values. we hash and store objects by checksum and rehash the objects when we retrieve to ensure there is no data corruption. all the objects we’re having issues with were uploaded at approximately the same time period a few days ago.

we’ve stored 10’s of millions of objects in S3 and never encountered such problems. please let me know ASAP if you have any idea what could be going on here. thanks.

Amazon responded 6 minutes later (!) and started investigating. To troubleshoot they asked customers to email aws@amazon.com with the ‘Bucket-Name and few keys that you believe are having issues’.

Others weighed in reporting similar problems. Amazon provided status updates and on Monday Jun 23rd at 6:10pm PDT, provided the following explanation:

We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted. Based on our investigation with both internal and external customers, the small amount of traffic received by this particular load balancer, and the intermittent nature of the above issue on this one load balancer, this appears to have impacted a very small portion of PUTs during this time frame.

What are some of the takeaways?

• If you are directly using the AWS S3 API, make sure to calculate and send MD5 checksums along with actual data. Check status return codes - an HTTP 400 error code means ’something went wrong’ - respond appropriately.
• If you are relying on 3rd party tools to access S3, be sure to check with your software vendor that they are following the advice from Amazon to use MD5. If they are not then your data can get silently corrupted…
• Downloads, aka HTTP GETs, can also be affected. The thread in the forum continues and questions are asked as to whether the corruption caused by the loadbalancer was affecting both incoming and outgoing traffic. The conclusion was yes. If you are hosting media on S3, and the browser is using partial GET requests (to download in chunks) then the corruption will not be automatically detectable.
• If your business relies on Cloud Storage, are you prepared to wait a 36 hours for a resolution? This isn’t a swipe at Amazon, this is true for any provider. Check your SLA’s, check the trouble ticket resolution times, ask about availability of experts for troubleshooting etc.
• Cloud Providers will increasingly need to instrument their services such that they can ‘early detect’ negative operational events. In this case, Amazon has stated plans to use better logging and analysis to automate detection of unusual error patterns (i.e. anomoly detection).
• This incident - caused by an Amazon malfunctioning loadbalancer - did not make it onto the AWS status page at http://status.aws.amazon.com/. Taking Amazon at face value, this incident only affected a small number of transfers, relative to the total number of S3 transfers. But this begs the question, what level of outage or service problem needs to happen before Amazon will flag the issue on their status page? On a sidenote, based on the timestamps, 31 hours passed between the loadbalancer being taken out of service and Amazon providing the explanation on the forum.
• When Amazon update their S3 API documentation, it would be useful to have entries in the S3 API index for ‘checksum’, ‘MD5′, ‘integrity’ and ‘corruption’.
• Stepping back, will customers hold Cloud Service Providers to a higher standard than their own internal IT teams?

I’m sure there are more takeaways I didn’t cover. What say you?

###

Kudos for the heads-up on the S3 issue goes to my friend and colleague Jason Harper - network supremo and crypto-head. Thanks Jason!

If you’re planning to, or are implementing wired 802.1X, wireless security and/or NAC, the contents of this blog may save you hours of time and trouble.

Throughout the implementations I’ve done, for both wired and wireless 802.1X, I’ve developed a procedure for implementing and testing 802.1X each step of the way. Following these steps my seem to be tedious and unnecessarily time-consuming. But, if  you’re just starting with 802.1X, I’m offering a way to implement it in phased pieces that will give you the information to test, confirm and troubleshoot at each step.

To be honest, I frequently skip these steps, but I’ve done many 802.1X implementations and can usually hit the bullseye the first time (unless there’s buggy software or firmware- you guys know who you are). But, if something doesn’t work, I start right back at Number 1 here and I follow this procedure.

1) Configure wired 802.1X
First setup the basic wired 802.1X. Ideally, start with a Windows test, using XP SP3 or a later server edition and PEAP. Provision RADIUS, I recommend Microsoft IAS because it’s well-documented and well supported. Even if you have other future plans, if you’re using Active Directory, start with IAS. You’ll need to setup a test RADIUS group and policy and link to AD. Get a test switch, add it as a RADIUS client, and configure it to talk to your RADIUS. Set up some ports for 1X and enable it on the switch. I recommend testing with PEAP as the authentication method and a Windows credential pass-thru. Note- you’ll need to create a server certificate to use PEAP- a self-signed Microsoft cert is fine.

If this simple configuration doesn’t work, you have some troubleshooting options. First, view the system events log in the RADIUS/AD server and look for informational events from IAS. If the authentication request is making it from the client -> switch -> RADIUS, you’ll see something here. The something you see should tell you if the EAP method is mismatched, or if the credentials were wrong, etc. Your second line of troubleshooting comes if you don’t see any RADIUS log activity. If that happens, throw on a packet capture utility like Wireshark. You want to search for 2 things. First look for conversations from your Test Switch to the RADIUS server (filter on IP or MACs). If you see something here, see where the conversation drops off. If that comes up empty, it means the conversation is terminated between the Test Switch and Test Client. I have some neat tricks for troubleshooting I’ll share with you later.

2) Add in Wireless
If you’re planning to implement 802.1X for wireless, now is the time to throw 802.11 in the mix. It’s harder to sniff wireless traffic for troubleshooting, which is why I recommend starting with wired 1X. Keep it simple, and then start layering. Once you have the wired 1X configured, all you need to do is get your AP ready and configure it just as you did your switch- add it as a RADIUS client and configure it to talk to RADIUS. For wireless, you’ll need to configure encryption also. Note, I recommend (for testing) to begin with your primary VLAN.

If your wireless 802.1X isn’t working, follow our troubleshooting above and re-check settings based on the RADIUS event log contents. If nothing is making it to RADIUS, then most likely something is misconfigured in your AP/Controller and the AP isn’t communicating with the RADIUS server. You know the rest of it’s working (RADIUS, AD, Client) so you can narrow your troubleshooting scope. Once that’s working you can stop if wireless is your goal, or keep going if you’re layering on more security.

3) Replace with Custom Pieces
If you’re planning to use a different RADIUS server or a different supplicant, now would be a good time to start swapping out our vanilla configuration with custom pieces. Replace 1 piece at a time and re-test.

4) Add in NAC or Endpoint Integrity
Most NAC or EI solutions will integrate with your 802.1X infrastructure (if you want them to) and can be ‘consulted’ prior to authenticating and opening the secured port. My suggestion is to always get 1X working 100% before you add any type of integrity or compliance testing.

If you follow these steps, you can turn a complex configuration into a set of simple baby-steps. It may sound stupid, but I promise it’ll work for you every time!

# # #

Friday the 13th.

Well, it was apparently worse than I thought at Infosecurity Canada. I spoke with eight people that attended and all of them gave it a unanimous thumbs down. Too bad. I guess if they were better organized it wouldn’t have sucked that badly.

Click here to subscribe to Liquidmatrix Security Digest!.

And now, the news… (better late than never)

1. Ameritrade Settling Hacking Lawsuit | Wired
2. McKinnon’s last ditch appeal to be heard by Lords | Heise Security
3. Third time’s the charm? RIAA tries end run around old case | Ars Technica
4. AG creates Cyber Crimes Unit division: Conway hopes to target Internet predators | Medical Leader News
5. Outsourcing contracts must offer personal data security | Computer Weekly
6. Windows Inspection Tool Set Helps You Troubleshoot Your System | AppScout
7. Web Application Security: Don’t Bolt It On; Build It In | IT Business Net
8. Opinion: Breach laws fail to protect anyone | InterGovWorld
9. Hacking: A story untold | Burlington Free Press

Tags: News, Daily Links, Security Blog, Information Security, Security News

"The tool will assist in prioritization of security measures based on their risk reduction potential," said the statement of work accompanying TSA's formal solicitation, which was posted April 18.

The software tool would help TSA gather and organize information about specific transport modes and assist agency officials to make risk management decisions.

The contract, which will be issued by TSA's office of operational process and technology, envisions a one-year base period plus four one-year options. The chosen vendor will be expected to install the software, troubleshoot any hardware or software problems, consult on building risk assessment modules, attend classified intelligence meetings at TSA headquarters and maintain the software.

I don't think you have to be very good to qualify here. This automated system put Boise, ID, on the top of its list of most vulnerable cities. The bar isn't very high here; I'm just saying.

Q1: When you mention "forensics", are you speaking in term of legal forensic terminology - or in terms of incident investigation?

A1: When I say "forensics", I usually mean it in the legal sense. I call other investigations simply "incident investigations;"  forensics carries an extra burden of proof and seeks to establish facts, not just "good hunches."

Q2: Are there solutions that can handle 2-3 Terabytes of log data per minute?

A2: No. Easy, huh? :-) See this for a specific example.  Well, let me take this back: theoretically, you can always use a vendor that can handle a lot of data (like LogLogic) AND that has an ability to run a distributed operation across many appliances. The catch? You will need a lot of the appliances since 2-3 TB/minute is about 90 millions of log  messages/second (assuming an optimistic 200 bytes/message)

Q3: I have terabytes of log data but how can be analyzed all this data? Are there products that can process all this data and receive valuable information?

A3: Yes, but you need to ask one question first: analyze why (example reasons here)? To discover something "interesting" (my favorite reason)? To find some specific artifact that you need in the logs? Or for some other reason? Before anybody can answer a question about "are there tools to 'analyze this'?", you'd need to answer that dreaded "why" question.

Q4: We were told to log every access to every SQL database in our environment. Is this even feasible with the best products on the market?

A4: Yes, it is. However, one needs to be extra careful with this. Look at this post for options and ideas. It may turn out that logging every SELECT statement and then collecting those native database logs will not be the best approach (mostly for database performance reasons) and a dedicated tool will need to be used. Database built-in auditing are better used for selective auditing.

Q5: Once logs are captured, and centrally stored, who should be responsible for the management and review of those logs?

A5: Good question! Really, this is a very good question that a) is important to have answered  and b) does not have an "accepted," standard answer. It also depends upon what logs are those; let's assume the most complex scenario of a diverse set of logs from networks, systems and applications. So, the choices are: security team (sometimes: CIRT i.e. incident response team), some dedicated team in IT that provides "log services" (uncommon option, but growing in popularity) or some unit  in IT that is responsible for regulatory projects (if compliance driven). If your answer is nobody, then you will be in trouble :-) If you answer wrong, you might have to fight to access your own logs (example)

Q6: Most of the discussion so far is about how to get started. What about after the system is deployed? Products tend to focus on collection and not on action or response. Where are the tools heading in terms of usability, incident tracking, collaboration?

A6: That's a long story, really, and it is hard to provide a short answer to this. Yes, collection has been a focus of products in the last few years, but now we are at a point where analysis and various uses of the data will come to the forefront. At the very least, you should be able to run reports and searches on the logs that you collected.

Q7: Do vendors typically offer a template of which logs to collect based the desired use cases?

A7: They should, yes :-) In some cases what you have is a bit of a push-pull between a vendor and a customer: "Tell us what to do?" - "First, you tell us what you would like to accomplish?" - "No, really, you tell me what I should be looking to accomplish." - .... sometimes ad infinitum. Also, for some uses cases it is hard to come up with a credible list (see this discussion about PCI DSS here)

Q8: What are the biggest difficulties when the log management solution is going to to be integrated and deployed in an organization with a lot of different log sources?

A8: Political boundaries and "log ownership issues" (see some discussion here)  If you need to submit a paper form in triplicate to add a line to /etc/syslog.conf and then send more forms when something doesn't work right and you need to troubleshoot it (a real story), everything becomes painfully slow and inefficient.

Enjoy! Again, here is the link to the webcast recording.

What can we conclude from this?

First, a "duh" conclusion is in order! No matter how many times one can utter the word "compliance," logs are still most useful for mundane (one would hope! :-)) system administration. Yes, indeed, sysadmins are the primary consumers of logs - yesterday, today, and - likely! - tomorrow as well.

Second, I am saddened by the fact that application developers have not warmed up to logs, at least no en masse (and not according to this limited poll...). I am guessing when they start thinking of logging when creating their applications, they will be more aware of the fact that you can troubleshoot the applications using logs ...

Third, incident response team showing that low is some kind of fluke, I am sure. Everybody knows that logs are indispensable during incident response (yes, even if only a little logging was enabled or even logging defaults left in place, logs often reveal answers unobtainable via any other mechanisms)

Am I reading too much into this? Hey, maybe I am! :-) Then again, I am a former theoretical physicist - thus, I can explain anything!

Next poll coming soon!