• The transport – processing Syslog events takes up a heck of a lot less processing power than collecting from a Windows box. Same with collecting data over an ODBC connection.

1. Our brilliant field engineer Dimitri McKay talks about the eternal topic of converting Windows event logs to syslog. Yes, Eric, we ALL know it is ugly, but that is the only way that actually works well across all systems ...
2. More on Windows and syslog: "Syslog ... 20 Years Later."  BTW, this is really not about syslog, but about Vista/2k8 finally getting an ability to natively centralize the event logs via event subscriptions ("It's only about twenty years behind schedule, if you're counting.")
3. Two fun pieces on correlation: 1 and 2. What often kills "a log correlation project"? "Whoever had worked on it had not had much time available to learn the way to properly configure the software" (from this)  and "correlation only really works when backed up by real data about what is the biggest problem in your environment, and how that problem manifests itself in the event logs." (from this) None of this is new, but a useful reminder nonetheless
4. Fun LogLogic podcast is here. The topic of this high-level discussion (CEO) is related to operational use for logs. I did one with them too; on logs and virtualization (will be up soon)
5. A couple of good posts on logging from Nemertes Research: "Sharpening Stones and Walking on Coals",  "Search or Destroy"
6. Reminder about a few useful Windows Vista and 2k8 events: 4802 (screensaver engaged) and 4803 (screensaver dismissed)
7. One person is wondering about the usefulness of logging after "experiencing" Linux auditd logging (kernel audit): "Logs are like a warm blanket; verbose logging means you can know what's happening on your systems if you keep up with the logs.  At the same time, logs become a burden very very easily, and they are easy to ignore." This post is a must read for us logging afficionados; producing too much log data is a sure way to make people hate you...
8. This also follows the same theme: people doubting the god-like power of logs :-) "So for an administrator to not care about logs was a shock." But would I argue that "log management is NOT a pain?" Now, would I? :-)
9. A classic about logging for application developers: "Building Secure Applications: Consistent Logging."  I am noticing a lot more discussions about logging in a developer community, e.g. see this and this (the latter, BTW, contains a lot of info on "why log" for developers). Overall, "getting logging right" is important (and will get more important in the future) and people need something NOW and cannot wait for the standards.  BTW, I am planning a mini-crusade on how to train application developers to include useful logging in their applications...
10. Finally, the "Is SIEM dead?" theme is continued in this fun post "Life after SIEM. Situational Awareness is next." Indeed, context is key for logs. BTW, if somebody mentions that I have "vendor bias", I will kick your ass! :-)

Enjoy!

Q1: How do you handle variety of log sources? There are so many, almost beyond my capability.

A1: Sorry to ponder the meaning of "is" here, but what is meant by "handle"? It is really not that hard to collect logs from a large number of diverse sources (as long as the logs can be delivered via syslog or exist as files and can be collected). Now, there will certainly be challenges  when the volume of logs gets large, but if by "handle" you mean "collect + store", it is really not that hard, given the right tools. Now, if "handle" means "make sense of what all those logs are trying to tell you," it is a different story altogether.

Q2: You talked about the importance of logging; however for an intermediate or novice admin what are the starting steps .. what are the minimal logs they should start at once?

A2: Answered in "Log Management - Day 1" If you want a simple list of things to "enable today,"  I cannot really answer it since I know neither your needs, nor your environment. In other words, this is the "what is the meaning of life question?" :-)

Q3: What regulations, rules or guidance exist regarding sharing or visibility of logs to users?

A3: PCI DSS says in Requirement 10.5:  "Secure audit trails so they cannot be altered.
10.5.1 Limit viewing of audit trails to those with a job-related need
10.5.2 Protect audit trail files from unauthorized modifications
10.5.3 Promptly back-up audit trail files to a centralized log server or media that is difficult to
alter"

NIST guidance for FISMA also says something similar (for example, look in NIST 800-92 doc). Overall, log protection and security are mentioned in many other regulations as well.

Q4: Privileged groups membership monitoring in AD one of the most important from my point of view. However I did not find effective way to monitor/report on changes in those groups. Any recommendations?

A4: This is indeed a tricky one which might take more space to answer than I have here; it might also take you 'beyond logs.' One good source of information is Randy Smith's site and, specifically, his webinar on 'Active Directory "Logging Gap"' (here somewhere) - which covers how to audit things of that sort when then native logging is not sufficient.

Q5: How I can learn what exactly I need to log?

A5: OMG, this is a $1,000,000 question :-) Let me answer "how can I learn" part and not the "what exactly I need to log part,"  (also see discussion on "MUST-DO Logging for PCI?") as it is actually answerable. To learn what you need to log, first ask "Why?" (and then see this) - basically establish what you want to accomplish with logs, catalogue your systems, figure how to tweak the logging knobs - and then do it!

Q6: How granular should logging be? What is your recommendation for enterprise servers like domain servers and Windows servers?

A6: Again, too long to answer here in details (it will become a subject of a longer blog post later), but some pointers follow: here for Windows (MS site also have a few recommendations on audit policies)

Q7: What is "more control" and what is "less control" that you mention in the webcast? Can you give an example?

A7: OK, I did say that "sometimes when you implement more controls, you actually have less control." What do I mean? If you buy a firewall (a network security control) and then - over time, of course - configure it with 7800 rules (!) that are supposed to give you control over who can and cannot access your network, you will not gain control over your environment. You will actually be less in control of who is touching your network, compared to, say, having only 20 rules.

Q8: What about mandated NIST controls for government systems? Auditing is a specific control for Moderate and High risk systems. What list of events do you recommend for auditing?

A8: This is too long to answer here, but NIST 800-92 Guide is a really good source of such info ("Guide to Computer Security Log Management [PDF]") Also, see my presentation on NIST 800-92 Guide in the Real World.

Q9: The issue that many organizations get stuck on, is the monitoring process, and defining what exceptions to monitor for? Is there guidance / framework for this? How much of it is system specific and how much is applicable generally to all systems?

A9: I outlined some general ideas back in 2004 via this presentation (note to self - update that to be more 2008-relevant); it is mostly general, but also has pointers to specific system. Keep in mind that it is focused on security, not operational monitoring (which is often no less important - in fact, often MORE important)

Enjoy! Sorry for being brief with some of the answers - I am woefully late with this even as they are...

Other questions that I answered in the past:

• More Log Management Questions - Answered!
• Some Burning Logging Questions - Answered!

ScienceLogic: How long have you been involved in InteropNet?

Katsev: I started at Coyote Point 3 years ago and InteropNet 2006 was my first “big” assignment.  This was the first time Coyote Point had put in a proposal to participate, so we were very excited when we were selected.

ScienceLogic: How long has Coyote Point been involved in Interop overall?

Katsev: We’ve been exhibiting at Interop for a number of years, and after seeing the InteropNet in action, we decided to submit a proposal in ‘06.  We were actually one of the first companies in the load balancing/traffic management space (we’ve been doing this for almost 10 years), so we have a lot of experience to share with InteropNet.

ScienceLogic: What is your role at Coyote Point?

My official title is “Engineering Project Manager”.  Basically, that means that I’m in charge of product releases and maintenance.  It sounds like a weird title for someone participating in InteropNet, but I’ve actually found it extremely useful since my position means that I don’t get to see our systems out in the field a lot.  We’ve added several features and have ideas for others just from my experiences at InteropNet.

ScienceLogic: What do the Coyote Point products do?

Katsev: Coyote Point makes a Traffic Management appliance called Equalizer.  What this means is that any traffic destined for a datacenter’s servers goes through our appliances and we make sure that the server which is best equipped to handle it, does.  Our systems sit between the clients and the servers and monitor the client traffic and the state of the servers.  If the clients start sending more traffic, we’ll balance it out so that no server is overloaded.  If one of the servers stops responding or starts responding very slowly, we’ll steer traffic away from that server.

ScienceLogic: In what way are your products being used as part of InteropNet?

Katsev: In the InteropNet, we’re utilizing a lot of our expertise:  We’re making sure that traffic is balanced and servers are redundant for show services such as DNS and SMTP.  We’re also using our geographic load balancing technology to ensure that the ScienceLogic EM7 appliances and some other internal NOC services are available from anywhere, with the lowest latency, with our SSL acceleration and GZIP compression technology.  Finally, we’re helping logistics in the NOC by allowing a physical separation between systems located in the NOC and those in an emergency rack outside of the NOC.  If either of these two locations were to fail, the network will continue operating without a glitch.

ScienceLogic: Are there any special considerations for Interop that cause you to deploy your systems there differently that any other place?

Katsev: Interop is definitely different than most of our customer installations.   One difference from a standard environment is that the network (at least this year) is one large flat network, with pieces carved out where extra security is needed.  Because of this, we can actually run our failover pairs of Equalizer systems in a non-standard configuration where the two peers are in different racks, or even on different floors.  That’s one of the things that I really like about InteropNet — it definitely brings new ideas to mind, which end up becoming ’special configuration’ white papers after the show.

ScienceLogic: Has InteropNet taught you anything that caused you to actually change your product?

Katsev: In addition to the failover configuration differences I mentioned above, participating in InteropNet has actually caused us to add several new features and allowed configurations.  One example is the “no-spoof” option for Layer 4 clusters.  Prior to the 2006 shows, we always ’spoofed’ the client’s IP address when talking to a server so that the server would see the client’s IP address instead of our own.  At Interop, we ran into a special configuration which would’ve been very difficult to set up in this manner, so our engineers added this feature, and it’s been very a very popular configuration with our customers ever since.

We have also had a couple of business relationships that extended outside of the show.  In 2006, we had a good experience using Spirent Communications gear to benchmark the network, so we ended up purchasing a couple of these systems to test our products.  More recently, we have found a way to bundle our Equalizer e350si load balancers with the ScienceLogic EM7 collector appliances to help ScienceLogic get the best performance in load balancing large quantities of syslog messages to be processed.  If it wasn’t for our participation in InteropNet, neither of these relationships would’ve happened.

ScienceLogic: What’s the best part of being involved with InteropNet?  What do you most look forward to?

Katsev: InteropNet is an amazing networking opportunity (no pun intended).  The group of engineers that put the network together every year is, well, amazing.  There is so much combined experience that any question instantly has several possible answers, and the best answer is chosen very quickly.  One of the ’sayings’ at Interop is “if you run into a problem, ask someone… we’ve probably seen that problem before… five times.”  One would think that being part of InteropNet is the same thing, year after year.  However, in the two years that I’ve been part of this (for four shows), there have been huge differences in the way that the network is designed and put together.  These are both because the vendors selected every year are different, and because the engineers who design the network change from year to year.  Somehow, though, when all is said and done, we have a network that works.

ScienceLogic: You don’t have to answer this one if you’re not comfortable… What would you like to see changed with the way things are done at InteropNet?

Katsev: This isn’t a cop-out… I really can’t think of anything I would do differently.  Sure, there are small problems that pop up sometimes, but every project has those, and the people at InteropNet are more than capable of figuring them all out.  In fact, I know that Interop started out as a show to test the interoperability of devices… but I’m still amazed that all of these devices actually talk to each other and “play nice” together.

ShareThis

• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, let me bet my ass that "Not knowing what to log" problem will be licked in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what  the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, for esoteric log sources (and custom applications) might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?"  for reference)

Next, the problem of "Log volume" will  definitely get worse, much worse.  One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is NOT a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL (!!!) get since standards are being developed. We will see people struggling with all sorts bizarro log data in the coming years. Virtualization, web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into dark voodooistic experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. BTW, watch my blog for some uber-fun info on that!

"Getting the logs"  has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing  files remotely via secure protocols made application log collection easier (syslog-NG with TCP transfer and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services (thanks Eric!). However, in the future it might resurface as we try to collect logs from "weird" places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them (Yo, my t-shirt-making colleagues... :-))

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in  an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and cursing under ones's breath. Only then we can attempt to reliable handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so."  For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. My next poll will address that :-)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?

Sometimes I’m the queen of analogies (likely a trait I inherited from my Dad). Quite often my analogies are pretty silly, but they almost always get the point across.

So I was trying to work out an analogy to explain how we can use logs, events and searching and why these are advantageous. I was in the shower and it hit me! And… here it is.  FYI- If you’re a techie, just stop reading now… (I warned you).

The analogy. Imagine a house… actually, imagine your house. Let’s say that your house is like a network. The house and all the major appliance and structures of the house are like infrastructure devices- switches and servers, for example. Of course, the people living in your house are users. In addition you have ‘gateways’ from your house to the outside world, in the form of doors, windows, vents, etc. These house gateways are like our WAN devices- firewalls, IDS/IPS and other gateway appliances.

Let’s say you live in the house with your spouse and family. You’re going to be the wife for now, so imagine you, your husband, three kids and a dog (only because that amuses me). Each of your house users have a key to get in.

Your major appliances- the TVs, refrigerator, oven, the family computers and alarm system are all creating logs when anything happens and they’re all giving their logs to the toaster. (The toaster is greatly under appreciated so I’m giving him a big role here- yes- your toaster is the Syslog server). The doors, windows and other ‘portals’ to the outside are also creating events and logging each time they’re opened, closed, locked or broken and, they too, are sending their info to the toaster.

Here’s where life in your house gets interesting. Let’s figure out what’s normal… it’s probably normal for your husband to come home, do some work on the computer while you cook, and then everyone watch TV. The kids are doing their homework, playing on the computer and probably rummaging around the fridge for an after-school snack. You see your syslogging toaster shows you…  

• the src= Refrigerator was opened multiple times in a short period of time between 3:43pm and 4:16pm by multiple users
• the src= Kids Computer was logged off the Internet at 4:30 by user: Kid2
• the src= Front Door was opened at 5:20pm by user: Husband
• the src= Oven was turned on Bake at 350 at 5:32pm by user: You
• the src= LivingRoom TV was turned on at 5:56pm by user: Husband
• the src= LivingRoom TV channel was modified multiple times in a short period of time between 5:56pm and 6:02pm (your husband was probably looking for the ball game)

These are all things you expect to see. So, what’s not normal? Some things your toaster may tell you that would be out of the ordinary…

• the src= Refrigerator was opened at 02:40am by user: Kid1
  What does this mean? Someone’s late-night snacking, no big deal.
• the src= Kids Computer was logged onto the Internet at 02:45am by user: Kid1
  Uh-oh, Kid1 is gallivanting on the Internet in the middle of the night un-chaperoned. Might need to check that out.
• the src= Front Door was attempted to be opened unsuccessfully 14 times in a short period of time beginning at 10:15am by user: UNKNOWN. The toaster logged the key code attempts tried by user UNKNOWN.
  Kids were at school, you were at work- someone’s trying to break in.
• the src= Front Door was opened the next day at 1:20pm by user: ROOT
  You were still not home- someone just broke into your house.

Maybe we want to be alerted when these things are happening, or have happened. With some log search and correlation tools, in conjunction with your toaster syslog, we can get immediate alerts when something unexpected is happening. We could tell the log search to keep talking to the toaster and immediately send us a text message if the toaster sees the front door or any windows being accessed between 09:00am and 3:00pm on any weekday, by any user. If the toaster saw something happening, we would know immediately and could take appropriate actions- maybe call the police to notify them of a break-in.

Now, back to the network. Now that you have an idea of how we can use logs and events in the house to identify what’s going on and spot abnormal activity, we can port that over to our network. Go back and again think of the house and its appliances as resources on the network. We can see when someone- inside or outside- is trying to or has successfully accessed something and we can alert, take action, or keep logs and reports for future use and accounting.

Replaying events. If you’re using a super-nifty tool, you may be able to replay specific events back in a visual format- almost like a video into the network. Let’s take our Kid1’s midnight snacking. If we replayed all the events that contained user= Kid1 from time 10:00pm (bedtime) to 07:00am (gettin’ up time) we could see Kid1 go from the bedroom down to the kitchen, opening the fridge, watching TV for a bit before going back to the room and surfing the Internet for an hour. We could actually ‘watch’ these events happening with a re-constructed timeline. A great example (and my favourite toy) to do this is Splunk’s Replay application.

That’s the basic gist of it all. There are some other detailed ‘things’ we can do with these technologies, and I may elaborate on those another time. We all have A.D.D. and this one is long enough already!

# # #

So my next iteration of fun reading on security, logging and other topics.

1. 0x000000 blog has a neat post on security, word definition and all. It reminds us that "security is forever" since it is about people, not broken technologies. A quote: "And so we will never able to secure other people, they have to secure them self. And we know that they can't." Same blog also have a fun (but a little bizarre with a little 80s feel) interview with Richard Stallman.
2. Along the same line, discussion about security industry longevity is here at Gunnar Peterson's blog: specifically, he debates Mike R's semi-humorous prediction that in 2012 there will be 0 "security professionals." Indeed, secure networks + secure OS + secure apps < security.
3. Also a very fun read comes from DarkReading: "7 dirty secrets of the security industry." Example quotes: "The goal of the security vendor is not to secure, it's to make money" , "Security vendors want businesses to buy what they sell, so they push specific products to block specific threats "; it also discusses another facet of compliance vs security.
4. Fun - and as usual heated - debates about the "AV is dead" and "anti-anti-virus revolt" happen here. Is blacklisting  AV dead now? More dead than before? :-) Or just "limited",  but still very useful? BTW, Matasano opines on the subject here as well, calling it not a revolution, but a protest.
5. The next  Carnival of the Security Catalyst Community - April 22, 2008; as always fun. Next carnival Apr 29 is here and the last (so far) one is here.
6. Really good look at logging for developers is here. "all too often logging gets treated as optional and not necessary. In this column we will cover the essentials of logging []for developers!] from a security perspective"
7. Latest stolen account prices are posted here by AVERT Labs guys. Account with $16,000 goes for about 700 euros (!) Also, Finjan reminds us that top corporations are all owned.
8. ISP data retention rears its (ugly?) head again. Good business for LogLogic or privacy nightmare?
9. A fun read from Tizor Blog: "How did the TJX data breach happen? Part 1: Anatomy" A must read, with diagrams, etc. "After breaching the TJX wireless system, the attacker was able to gain administrative privileges to the RTS servers located at the TJX corporate headquarters in Framingham, MA."
10. A very good read from Greg Shipley: "Risk Management: Do It Now, Do It Right." A lot of interesting bits about CSOs, security technologies evolution, etc. "The journey continues. We invested hundreds of millions of dollars in intrusion-detection systems without a solid understanding of their relative effectiveness and total cost of ownership. The IDS craze led to reinvestments in intrusion-prevention systems that even today are only partially enabled, and PKI is still a bad word in many IT circles. There's no shortage of disappointments on other product fronts."
11. "Data Classification Is Dead?"  Rich Mogul explains why data classification by the owners is never going to fly... "Enterprise content is just too volatile for static tags to really represent its value. Even those of you in defense/intelligence don’t *really* do granular data classification. " This is a good reminder to shoe that just spout the propaganda "first, need to classify data." Can you hope to do "DLP" without it? Also, read this one from Rich as well: not only you can't classify, you often don't know who owns what.
12. Hot, hot, hot! "Snake Bytes " on DarkReading. "We are all in the business of stopping just enough crime to keep us in business." Wow! Definitely a must read.
13. Marcus Ranum on logging in Start Trek (read the whole thread): "What do you expect from a starship that runs on Windows-24k? Microsoft added support for syslog in 2348 - citing customer demand - but still
    has no Enterprise-class log architecture." :-)
14. Piece on PCI and log management where a vendor makes an idiotic faux pas by saying that "less than 1% logs are of interest." In reality, all (OK, most) logs are of interest under the right circumstances. And we almost never know which ones we'd need.
15. A fun blurb from a lawyer on PCI. Good conclusion too: "Regardless, now is the time for merchants to begin engaging their legal teams to address PCI compliance, and opening the lines of communication between the lawyers and security pros." He also fights the checkbox mentality by saying that  "merchants should not view their internal security personnel or QSAs as “rubber stamps” of PCI compliance." I am happy to see this lawyer basically say that if you ignore PCI, your ass is  0wned :-)

On that happy note - see you next time! :-)