On 13 August, jcorey asked about how to deal with those who firmly believe that the only answer to any security problem is to inspect everything at the edge. This is an important question, and I wanted to give Joe an answer. (You might have to scroll down when you click the previous link, it seems that linking to individual comments is broken.)

Today, 15 October, I wrote a little thesis as an answer to his question. I’m calling it out in a separate post because I want to make sure those of you with aggregators that don’t update when posts receive new comments still have a chance to reply with your thoughts. I’ll also repost it here:

jcorey-- You've nailed the biggest obstacle to deploying something like direct connect. Many security professionals have been taught that there simply is, and never will be, a process or technology that allows you to trust anything that originates from outside your corpnet. These professionals cling to this belief, and have been the cause that allowed the whole “detection” market to bloom.

Let me be clear: this total lack of trustworthiness is no longer absolutely true. Of course there will be times when unknown machines will be used by known and unknown people to access your information. But what about one particular subset -- known humans, with known portable computers -- can't we do something better than treat them as toxic invaders?

Indeed we can. And that's what I'm proposing with direct connect. The technology -- managed, of course, with the right processes -- exists so that you can extend the trust to known computers even though you don't trust the network they're connected to. This is because you have mechanisms that:

1. Allow you to configure the machine according to your requirements (domain join, group policy)

2. Dictate computer and user authentication requirements (IPsec policies, smart cards)

3. Limit what the users of these machines can do (UAC, non-admin, Forefront Client Security, Windows Firewall, even software restriction policies)

4. Validate the health of machines initiating incoming connections and remediate if necessary (NAP, System Center Configuration Manager)

5. Limit the threat of attacks against stolen computers (domain logon, smart cards, BitLocker with TPM)

With the robust authentication, validation, configuration, and control mechanisms available to you, I simply don't see that there's any need to fall back to “detection” now. Detection technologies were -- and remain -- necessary for the times when we have no clue about the health of client computers and when we had no way to gauge the intent of the users. But it is truly reflective of a head-in-the-sand mentality to assume that this is a complete description of what's capable today.

You know, someone once asked me what it takes to be a security professional. I answered that there are two primary elements: become a networking/packet wonk, and be willing to change your opinions when the right evidence comes along. Indeed, I suspect that many security folk have forgotten the need to keep their wonikness updated, which in turn makes them resist new ideas regardless of the strength of the evidence. I'm not very proud of what I just wrote, because I loathe generalities, but I'm not sure what else to think here. Sigh.

Joe’s question is important and strikes at the foundation of what it means to be a security professional today. I’m eager to continue this conversation, because it’s reflective of what I sense to be a radical shift in our jobs—we are, or should be, no longer the wolf-crying propeller-head who sits in the basement and twiddles with the firewall. Instead, our job should be defined as one who’s charged with protecting the organization’s information from attack, while maximizing its utility to authorized users, according to the principles of least privilege. Your thoughts?

• The user is logged on as a local administrator
• The user, or some code the user runs, enables IP routing

By default, all forms of IP routing (including NIC bridging) are disabled. Only local administrators (or group policy) can enable them. So the risk, actually, is minimal.

If you have a stroll through group policy, you'll discover this setting: "Prohibit installation and configuration of Network Bridge on your DNS domain network" (more here, here). This setting allows you turn a computer into a router that bridges two networks. The bridging works only when one of the interfaces is in the same DNS namespace it was in when the bridge setting was enabled, and it works only when the Windows firewall is disabled on both interfaces (never a good idea). Additionally, regardless of the group policy setting, the function doesn’t even appear as an option when the user is logged in as a non-admin. The group policy setting simply removes the option from people who are local admins of their computers. So here's a way you can remove the ability even for local admins to enable routing.

However, let me admit that I wish we did have a way to implement your request, but for an entirely different reason: IP address preservation. Consider what happens when I'm on my own corpnet in my office. I put my laptop in its dock, which is connected to the Ethernet. I never bother disabling my wireless (I'm lazy). So whenever I'm in my office I'm taking up two IP addresses: one on the Ethernet and one on the wireless. Such wasteful profligacy, I know! (Note this isn’t a problem for any Bluetooth adapter, which always uses APIPA in its default configuration; I can’t imagine a scenario where you’d want Bluetooth to use DHCP.)

If you agree with me that this is something we should address post Windows 7, not for "security" reasons but as a good general networking practice of being conservative with address allocation, please speak up. Now's the time for your input.

1. Our brilliant field engineer Dimitri McKay talks about the eternal topic of converting Windows event logs to syslog. Yes, Eric, we ALL know it is ugly, but that is the only way that actually works well across all systems ...
2. More on Windows and syslog: "Syslog ... 20 Years Later."  BTW, this is really not about syslog, but about Vista/2k8 finally getting an ability to natively centralize the event logs via event subscriptions ("It's only about twenty years behind schedule, if you're counting.")
3. Two fun pieces on correlation: 1 and 2. What often kills "a log correlation project"? "Whoever had worked on it had not had much time available to learn the way to properly configure the software" (from this)  and "correlation only really works when backed up by real data about what is the biggest problem in your environment, and how that problem manifests itself in the event logs." (from this) None of this is new, but a useful reminder nonetheless
4. Fun LogLogic podcast is here. The topic of this high-level discussion (CEO) is related to operational use for logs. I did one with them too; on logs and virtualization (will be up soon)
5. A couple of good posts on logging from Nemertes Research: "Sharpening Stones and Walking on Coals",  "Search or Destroy"
6. Reminder about a few useful Windows Vista and 2k8 events: 4802 (screensaver engaged) and 4803 (screensaver dismissed)
7. One person is wondering about the usefulness of logging after "experiencing" Linux auditd logging (kernel audit): "Logs are like a warm blanket; verbose logging means you can know what's happening on your systems if you keep up with the logs.  At the same time, logs become a burden very very easily, and they are easy to ignore." This post is a must read for us logging afficionados; producing too much log data is a sure way to make people hate you...
8. This also follows the same theme: people doubting the god-like power of logs :-) "So for an administrator to not care about logs was a shock." But would I argue that "log management is NOT a pain?" Now, would I? :-)
9. A classic about logging for application developers: "Building Secure Applications: Consistent Logging."  I am noticing a lot more discussions about logging in a developer community, e.g. see this and this (the latter, BTW, contains a lot of info on "why log" for developers). Overall, "getting logging right" is important (and will get more important in the future) and people need something NOW and cannot wait for the standards.  BTW, I am planning a mini-crusade on how to train application developers to include useful logging in their applications...
10. Finally, the "Is SIEM dead?" theme is continued in this fun post "Life after SIEM. Situational Awareness is next." Indeed, context is key for logs. BTW, if somebody mentions that I have "vendor bias", I will kick your ass! :-)

Enjoy!

The convergence of voice and video traffic are quickly transforming the growing complexity of networks at a torrid pace. IDC estimates that the skills gap in VOIP should grow to 19% by 2011.

This changing profile in the role of the network plays a key role in the skills shortage. Network enabled collaboration tools such as social networking apps and the Webex conferencing/collaboration solutions we use in our business each and every day are demanding a new set of IT skills to deliver business value.

My perspective is two-fold on this issue; the first is what I have seen in the resources we have attempted to hire! We give a very straightforward quick written/oral test to all new technical hires. This requires basic networking knowledge and some Unix commands. On average, (after filters from reputable recruiting firms, some with 5-10 years experience) less than 10% pass muster for the first filter we use in our hiring process. This is a troubling fact, which has cost us considerable time and effort to secure the right resources with competent skills. So I can say from our market assessment in a very strong technological job skills market, core Unix and networking foundation skills are slipping.

The second is that we as an IT Operations Management (ITOM) industry need to keep pushing hard to build better proactive and intuitive solutions to aggregate instrumentation from all Data Center tools, including more work around VOIP, video streaming, and collaboration so that we can ease this transition. If ITOM solutions become more proactive across the typical Cisco infrastructure that is commonly installed in the Data Center, we can free up some additional time for advanced “emerging technologies” training where existing IT workers can enhance their core skills and re-invigorate their careers. We have to do a much better job of getting our existing IT professionals trained on emerging technologies!

While there’s less that ScienceLogic can do around training, we certainly strive to do our part to enhance a day in the life of the networking engineers who use our solutions to simplify monitoring of increasingly complex networking, Wireless, VOIP, and collaboration needs.

ScienceLogic: Mike, what’s your background working with network and management system tools?

Mike Lawson: Before joining Apptis, I worked for the Air Force, mainly in satellite communications for almost nine years. I’m probably most familiar with HP OpenView and BMC Remedy. I managed a team that used them but wasn’t involved in tool selection; like many other federal IT workers, we didn’t have a choice of tools because there were existing enterprise licenses and maintenance contracts.

I also saw a large systems integrator do a full Remedy/Crystal Systems/OpenView installation. It took 6 weeks to stand up and customize to meet just the basic monitoring requirements, and it cost something like half a million dollars. At the time, I thought that wasn’t bad and was a pretty typical experience.

ScienceLogic: Coming from where you did, what’s your take on EM7?

Mike Lawson: Honestly, I didn’t believe that EM7 could really do all that it claimed. In many ways, it was the complete opposite of what I had seen first-hand with other monitoring solutions. Could it really cover that much functionality? At relatively much lower cost to the customer and without the licensing nightmare?

That quickly changed when I needed to understand the system enough to run it at a customer’s site. I went back over the training docs I received during my initial training class and jumped in; now, 6 months later, I’m the EM7 expert and can tell you that it delivers on all those promises. (But I still need to show people to get them to believe it too)

I preach the “EM7 gospel” and when anyone wants to talk monitoring, I ask about the universal pain points: cost, maintenance contracts and licensing, and then I explain EM7. The cost difference is real; the solution is based on capacity, so there’s no licensing and it’s easy to use. They are shocked to learn that they can buy multiple EM7 appliances and years of maintenance for what they paid for most other tools.

ScienceLogic: Apptis won the contract for monitoring aboard the USNS Mercy. We love that you’re using EM7 for one of the Navy’s hospital ships. Can you tell us more?

Mike Lawson: The USNS Mercy is a Military Sealift Command hospital ship. Some stats:

• 849 feet long (nearly the size of a football field)
• 12 fully-equipped operating rooms, a 1,000 bed hospital facility, digital radiological services, a diagnostic and clinical laboratory, a pharmacy, an optometry lab, a CAT scan and two oxygen producing plants
• Crew: 61 civilian mariners, 956 Naval medical staff, and 259 Naval support staff

The USNS recently departed on a five-month humanitarian mission in the Western Pacific and Southeast Asia in support of Pacific Partnership 2008. The partnership provides international medical, dental and engineering teams this summer to provide humanitarian support and conduct joint, combined, and cooperative Civil-Military Operations in order to improve regional stability and build partner capacity to respond to natural disasters and pandemic.

For the most part, the ship’s network is self-contained, but can also use a landline when docked. The network covers 400 devices, including Windows/Exchange servers and VMware for server virtualization. Prior to using EM7, none of the monitoring was integrated; each system was independently monitored through individual vendor-specific consoles.

Out of the box, EM7 provided integrated systems, application and network management for all network gear, applications and virtual machines in one solution. We didn’t have to do a lot of customization – EM7 includes best-practice based thresholds, event and monitoring templates and this covered what USNS Mercy needed to monitor.

ScienceLogic: You’re a systems integrator with a very useful “customer point of view” when it comes to looking at tools. From that perspective, can you share what you think are the biggest benefits that EM7 provides?

Mike Lawson: First of all, EM7 stands up right away. We’re talking days, not weeks. In contrast to the lengthy installation of OpenView and Remedy I witnessed during my military career, I was able to configure, customize, and implement the EM7 solution for the USNS Mercy in three days.

Second, it’s easy to train people on and the support is outstanding. This judgment is from first-hand experience. Right before the USNS Mercy departed on its latest voyage, the system administrator I had trained on EM7 left, so I had all of a day to train some new EM7 admins. I prepared a seven-page “cheat sheet” and over a 3-hour conference call, we walked through the entire EM7 solution; I haven’t gotten a support call since.

And when a problem did crop up with a device being discovered incorrectly, ScienceLogic was very responsive. We contacted ScienceLogic support on a Saturday and they created and emailed us a video to help troubleshoot the same day. Within 30 seconds of watching the video, the problem was resolved.

Finally, EM7 helps us be good stewards of the government’s money. This is very important to me personally and to Apptis as a company. Because EM7 is cheaper and deploys so quickly and easily, you might think that it’s just the opposite of what a system integrator would want to use. But that’s short-term thinking. We believe in deliver the most value for customers every time. It’s what creates trust and long-term relationships with our customers. Instead of that half million spent on standing up the solution and basic setup, I’d much rather (and I know the customer would rather) spend that on fine-tuning or extending the solution to do much, much more.

As a former government employee, I know what it’s like to use a tool that doesn’t fit my needs. EM7 proves that the best solution can totally break the old model of costly, lengthy installations. EM7 has the right model: the right solution and the right price delivered as an appliance that is easy to deploy, train on and use.

ShareThis

Q1: How do you handle variety of log sources? There are so many, almost beyond my capability.

A1: Sorry to ponder the meaning of "is" here, but what is meant by "handle"? It is really not that hard to collect logs from a large number of diverse sources (as long as the logs can be delivered via syslog or exist as files and can be collected). Now, there will certainly be challenges  when the volume of logs gets large, but if by "handle" you mean "collect + store", it is really not that hard, given the right tools. Now, if "handle" means "make sense of what all those logs are trying to tell you," it is a different story altogether.

Q2: You talked about the importance of logging; however for an intermediate or novice admin what are the starting steps .. what are the minimal logs they should start at once?

A2: Answered in "Log Management - Day 1" If you want a simple list of things to "enable today,"  I cannot really answer it since I know neither your needs, nor your environment. In other words, this is the "what is the meaning of life question?" :-)

Q3: What regulations, rules or guidance exist regarding sharing or visibility of logs to users?

A3: PCI DSS says in Requirement 10.5:  "Secure audit trails so they cannot be altered.
10.5.1 Limit viewing of audit trails to those with a job-related need
10.5.2 Protect audit trail files from unauthorized modifications
10.5.3 Promptly back-up audit trail files to a centralized log server or media that is difficult to
alter"

NIST guidance for FISMA also says something similar (for example, look in NIST 800-92 doc). Overall, log protection and security are mentioned in many other regulations as well.

Q4: Privileged groups membership monitoring in AD one of the most important from my point of view. However I did not find effective way to monitor/report on changes in those groups. Any recommendations?

A4: This is indeed a tricky one which might take more space to answer than I have here; it might also take you 'beyond logs.' One good source of information is Randy Smith's site and, specifically, his webinar on 'Active Directory "Logging Gap"' (here somewhere) - which covers how to audit things of that sort when then native logging is not sufficient.

Q5: How I can learn what exactly I need to log?

A5: OMG, this is a $1,000,000 question :-) Let me answer "how can I learn" part and not the "what exactly I need to log part,"  (also see discussion on "MUST-DO Logging for PCI?") as it is actually answerable. To learn what you need to log, first ask "Why?" (and then see this) - basically establish what you want to accomplish with logs, catalogue your systems, figure how to tweak the logging knobs - and then do it!

Q6: How granular should logging be? What is your recommendation for enterprise servers like domain servers and Windows servers?

A6: Again, too long to answer here in details (it will become a subject of a longer blog post later), but some pointers follow: here for Windows (MS site also have a few recommendations on audit policies)

Q7: What is "more control" and what is "less control" that you mention in the webcast? Can you give an example?

A7: OK, I did say that "sometimes when you implement more controls, you actually have less control." What do I mean? If you buy a firewall (a network security control) and then - over time, of course - configure it with 7800 rules (!) that are supposed to give you control over who can and cannot access your network, you will not gain control over your environment. You will actually be less in control of who is touching your network, compared to, say, having only 20 rules.

Q8: What about mandated NIST controls for government systems? Auditing is a specific control for Moderate and High risk systems. What list of events do you recommend for auditing?

A8: This is too long to answer here, but NIST 800-92 Guide is a really good source of such info ("Guide to Computer Security Log Management [PDF]") Also, see my presentation on NIST 800-92 Guide in the Real World.

Q9: The issue that many organizations get stuck on, is the monitoring process, and defining what exceptions to monitor for? Is there guidance / framework for this? How much of it is system specific and how much is applicable generally to all systems?

A9: I outlined some general ideas back in 2004 via this presentation (note to self - update that to be more 2008-relevant); it is mostly general, but also has pointers to specific system. Keep in mind that it is focused on security, not operational monitoring (which is often no less important - in fact, often MORE important)

Enjoy! Sorry for being brief with some of the answers - I am woefully late with this even as they are...

Other questions that I answered in the past:

• More Log Management Questions - Answered!
• Some Burning Logging Questions - Answered!

“FISMA Act encourages U.S. government agencies to configure their DNS servers to the DNSSEC security specifications set by the National Institute of Standards and Technology, and it has been reported that the federal government’s Office of Management and Budget (OMB) plans to begin enforcing DNSSEC requirements through an auditing process, setting the standard for DNS best practices.”

Yep, if you stamp FISMA on it, people will buy it, maybe in your PR department’s wettest and wildest dreams.  Guys, it’s been 6 years, that kind of marketing doesn’t work nowadays, mostly because we spent ourselves into oblivion buying junkware similar to yours and now we’re all jaded.

Now don’t get me wrong, DNSSEC is a good thing, especially this month.  But there is something I need to address:  FISMA requires good security management with a dozen or so key indicators, not a solution down to the technical level.  Allusions to OMB are just FUD, FUD, and more FUD because unless it’s in a memo to agency heads, it’s all posturing–something everybody in this town knows how to do very well.  OMB would rather stay out of mandating DNSSEC and maybe give a “due date” once NIST has a final standard.

My one word of wisdom for today:  anybody who tries to sell a product and uses FISMA as the “compelling event” has no clue what they’re talking about.

Bookmark to: