HANSEI - WHAT IS “RELENTLESS REFLECTION?” - And why we’re talking about it in the context of Risk Analysis.

Recall from yesterday’s post about how I got to thinking about the concept of Hansei-Kaizen, “relentless reflection” and “continuous improvement” and how we might apply that to risk management.  It’s a concept born of Toyota and is, in some way, the foundation for “Lean” production.

Call me biased, but I think that Hansei - the act of ‘relentless reflection’ made structured is the analytical function.  And I hate to debate (post-mortem) the father of Toyota quality success when he says that Hansei is the “check” in Plan/Do/Check/Act, but I think that Hansei also applies to the “Plan” of the P/D/C/A or Deming cycle.

You’ll recall the P/D/C/A cycle can be thought of even as an implementation of Scientific Method, in that it is Observation & Hypothesis Creation (P), Experiment (D), Analysis (Check), and Act (Revise/New Hypothesis, etc…).  Well then as such, the Hypothesis creation involves creating a model or creating an expected outcome for data using the currently accepted model.

So in our industry there is an opportunity for Relentless Reflection in both the Observation and Hypothesis (Plan) creation steps, and the Check step.  We create an estimate for control strength, or probable losses in the context of risk- then we go to Experiment step.  That hypothesis can be put it into production, have an audit, have a penetration test, whatever, in the context of the Do step.  BTW - using Hansei/Analytics in Plan is one way that strong analytical functions can really make penetration testing more useful - as a means to test the estimates and inputs into a model.  It’s Penetration Testing 2.0!  (<- tongue fully in cheek, yes)

Those who are versed in the reasons to merge Six Sigma and Lean together are probably already seeing where I’m going with this today.  But before you think that a simple DMAIC function is all that is needed to create proper “Hansei”, let me encourage you to keep reading.

Now if the analytical function can said to be “reflection”, why must it be relentless?

One word.  Change. There are essentially four separate “landscapes” or sources of change that we face (more on those tomorrow).  But anyone who has tried to manage system compliance, log management or policy exceptions knows that change is possibly the most difficult thing we security professionals must manage.  And when you think about it, there aren’t too many other business functions like information security where significant visibility and insight about the environment is needed for “complete” information (get bullish on Log Management is my recommendation).

HANSEI STEPS ADAPTED TO INFORMATION SECURITY

This is one of those quality control concepts that we can mangle adopt.  At Toyota, Hansei-Kaizen includes the following basic steps:

1. Initial problem perception
2. Clarify the problem
3. Locate area/point of cause
4. Investigate root cause (using an ask why 5 times approach)
5. Countermeasure
6. Evaluate
7. Standardize

Now it’s important to note that part of this includes the concept of Go See For Yourself, called “Gemba“.  Gemba can be translated as “the actual place” or “the place where virtue or truth is found.” At Toyota this might mean going to the shop floor to see the issue at hand in the production line.  But for us, that’s a problem because we live in the virtual world.  There’s usually not much use in hanging out in the wiring closets to try to see the problems.

But if you combine the concept of Gemba with the concept of “Nemawashi” –the process of discussing problems and potential solutions with all those affected- we can forge a similar concept using risk analysis.  That is discussing the issue and the risk associated with an issue (what some people would call “risk management”) with the business/LOB/data owner and let them accept authority and the risk decision.  We, the risk analyst, our goal is simply to perform items 1-5 (presenting countermeasure options that include transferring or accepting risk).  By going to the line of business and involving them, responsibility is shared.  Also, if you structure organizational behavior right, personal risk is transferred!

This sort of approach is also in harmony with concepts like “mutual ownership of problems,” or “genchi genbutsu,” (solving problems at the source instead of behind desks), and the “kaizen mind,” (an unending sense of crisis behind the company’s constant drive to improve).

One of the criticisms I have with the way most people try to implement DMAIC into “Lean”

REQUIREMENTS

Now to get this done, I really see three significant requirements.

1.)  A change in political structure.

2.)  Models that provide consistent, defensible analysis.

3.)  A Quantitative approach.  This means using actual units of measurement (not just amorphous percents, ordinal scales, etc.)  for risk and it’s subsequent factors.  Sure there are times when Q&D qualitative approaches are acceptable, but policy should be to have quantitative analysis whenever and wherever possible.

That last item - the quantitative approach - is really quite important.  And the reasons why will be discussed further in tomorrow’s post:

“What should we be reflecting about? & What is needed for reflection?”

P.S.  Your comments and suggestions, as always, are welcome.

P.P.S  Those who may be familiar with Lean/SixSigma/Kaizen sorts of mashups may be thinking - “hey, an Analytical step is built into SixSigma”.  Well, yes there is some prevision for analytical functions based on statistics, but I find SixSigma geared towards creating a State of Knowledge about operational processes, not towards creating a State of Wisdom for CISO’s around security & risks “big questions”.  In otherwords, the analytical function in DMAIC is in the context of Kaizen, and a different step than “reflective” analytics.

adaptive process management (n.) an element of resource and business process management, adaptive search and event processing. Sometimes referred to as “Level 4” event processing or process refinement.

application concept (n.) a definition of a set of properties that represent the data fields of an application entity. An application concept can describe relationships among themselves. For example, an order concept might have a parent/child relationship with an item concept. A department concept might be related to a purchase requisition concept based on the shared property, department_id. Application concepts can include an application state model.

application state modeler (n.) a UML-compliant application that allows you to model the life cycle of a concept instance — that is, for each instance of a given concept, you can define which states it will pass through and how it will transition from state to state. States have entry actions, exit actions, and conditions, providing precision control over the behavior of an event processing agent. Transitions between states also may have rules. Multiple types of states and transitions maximize the versatility and power of the application state modeler.

derived event (n.) an event that is created as a result of processing one or more other events.

complex event (n.) an event that is a situation-entity abstraction of two or more simple, derived or other complex events.

complex event processing (n.) CEP is a technology for extracting information from message-based systems. CEP is primarily an event processing concept that deals with the task of processing multiple events from an event cloud with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes.

event (n.) a instance of an event definition. It is an immutable object that represents a business activity that happened at a single point in time. Just as one cannot change the fact that a given activity occurred, one cannot change an event — events are immutable.

event aggregation (n.) the aggregation of simple, derived or complex events into higher levels of event abstractions.

event definition (n.) a set of properties related to a given activity that represents an important or interesting change of state in a human, system or computational activity. An event definition includes event properties such as event priority, event time to live (TTL), and a description of the payload, which is comprehensive information related to the activity that occurred. Events expire when the TTL has elapsed, unless the event processing agent has instructions to consume them prior to that time.

event channel (n.) a communications channel in which events are transmitted from event source to event receivers, typically received as electronic messages. Each channel can have multiple destination and. events can be configured to transmit to a default destination. JMS is an example of an event channel.

event cloud (n.) a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events. Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud.

event-driven (n.) the behavior of a human, system or computational entity whose execution or actuation is in response to events, typically received as electronic messages.

event-driven architecture (n.) an architectural style for distributed computing applications in which some of the components are event-driven and communicate by means of events.

event processing (n.) computing that performs operations on events, including modifying, creating and destroying events.

event-object (n.) an software object that represents an event, generally for the purpose of computer processing, that exhibits both encapsulation, inheritance and polymorphism.

event prediction (n.) computational activity where the impact of events, complex events, and situations caused by events identified, including both opportunity or threat. Sometimes referred to as “Level 2” event processing, impact assessment or predictive analytics.

event pre-processing (n.) computational activity where events are cleansed or normalized to produce semantically understandable data. Sometimes referred to as “Level 0” event processing.

event processing (n.) computational activities on events dealing with the association, correlation, and combination of event data and information from single and multiple event sources to achieve refined identity and situation estimates for observed event objects, and to achieve complete and timely assessments of opportunities, threats, and their significance. Event processing is characterized by continuous refinements of event estimates and assessments and by evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

event processing agent (n.) an EPA is a computational entity that performs event processing.

event processing network (n.) a set of event processing agents and a set of event channels connecting them.

event properties (n.) data representation of an event, typically by name-value pairs of type string, integer, real, boolean or a complex data type.

event refinement (n.) filter, identify and track events & make initial processing decisions based on association, correlation and state estimation. Sometimes referred to as “Level 1” event, or event-object, track and trace.

event stream (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

event stream processing (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

rule (n.) defines what triggers unusual, suspicious, problematic, or advantageous activity within an event processing agent and what the EPA does when it discovers these types of activities. Rules execute actions based on certain conditions on events, instances, or a combination of both. A rule includes a group of condition-rule statements and action-rule statements. The condition statements instruct the EPA what to look for in events, and action statements instruct the EPA how to respond when conditions are met. If all the conditions in a rule are satisfied by events or instances or both, the EPA fires the actions. The action might be to execute tasks, create an event instance, modify property values in an event instance, create and send an event, or something else.

rules engine (n.) a type of event processing agent that uses a declarative programming model to process events. Formally described as “an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet“. Informally, it can be any system that uses rules, in any form, that can be applied to data to produce outcomes.

rule language (n.) is an artificial language that is used to control the behavior of an event processing agent. Rules languages, like human languages, have syntactic and semantic rules to define meaning.

situation refinement (n.) identify situations, or complex events, based on event clustering, event-event relationships and relationship analysis and context. Sometimes referred to as “Level 2” event processing.

simple event (n.) an event that is not an abstraction or composition of other events.

virtual event (n.) an event that is imagined, modeled or simulated.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.   I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

1. Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.  There will be no physical access to the server. No exceptions.
2. Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
3. Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
4. Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
5. Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,  10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
6. Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
7. If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
8. It should send an email to one or more email addresses with a message, for example:  “New Bot Detected - Confidence 8″ with IP address, etc. in the message.  Another example would be an email, “Bot Blocked” - with details, etc.
9. You cannot automatically block any traffic that is not a bot.  Blocking one “non-bot” results in failure, no exceptions.
10. The Prize:  The winner will get their logo (w/link) on this site in a block called “Bot Hunter Winner” (or something like that.)

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!  Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.   So, you will need to work with both long and short time windows.  Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

Changing Behavior

1. Educate yourself.
At least every 6 to 12 months, surfers should browse the educational information provided by their operating system and security vendors and subscribe to any security-related newsletters they might offer. According to David Perry, familiarity with the latest threats, dangers, and recommended safety tips will allow surfers to make safe choices. “Until you know what’s out there, you’re just flying blind. Without an education, you’re wide open”.
2. Avoid suspect sites.
While criminals can infect even mainstream Web sites, sites such as gambling sites, adult Internet sites, and illegal file-sharing sites are far more likely to carry malicious code. Web sites that offer “something for nothing” frequently recoup their losses by infecting visitors’ PCs.
3. Lose Your Comfort Zone.

Recommended Technology

4. Use an updated virus scanning suite.
The most important component of any threat mitigation system is a virus scanning suite. In addition to detecting and removing known viruses and malware, modern virus scanning suites provide additional protections against new attacks by disabling their known protocols. For example, Trend Micro™ Internet Security encrypts keyboard traffic, protecting personal data from keyboard logging programs that might go unnoticed. Users should update their scanner and virus definitions as frequently as possible to ensure the best possible coverage.
5. Upgrade your OS and browser.
In addition to offering more features, Microsoft’s Internet Explorer version 7 and the latest Mozilla Firefox are both substantially more secure than previous-generation browsers. Users of older browsers should upgrade immediately to take advantage of increased security. Similarly, Windows Vista and Mac OS X are more secure than their predecessors, and users of older operating systems should consider upgrading, as well.
6. Disable scripting and “widgets.”
Many Web-based attacks use various scripting languages to run infectious programs in a browser or use downloadable “widgets” to execute infections locally. By disabling scripting and avoiding downloadable widgets wherever possible, surfers disable these common attack vectors.
7. Rate your Web pages.
Some available services rate the risk of Web pages in search results, allowing surfers to avoid unwanted content and hidden threats before viewing the pages. Rating applications (e.g., Trend Micro TrendProtect™) consume few system resources and run unobtrusively, so they are suitable for any Web-enabled personal computer.
8. Ask your provider.
Commerce companies, banks, and credit card associations are all interested in computer security, and many offer additional features. For example, Visa’s Verified By Visa program requires cardholders to enter a second password to identify themselves during a transaction, while businesses in Poland require cell-phone confirmation of credit card purchases. While nothing will be 100 percent effective, any additional security measure provided by a trusted source will increase protection, and surfers should adopt as many as possible.

This article provided for your reading pleasure by Trend Micro.

What I found is that the two providers were using entirely different means to format the events! The email provider calls ToString(bool, bool) on the event to ask it to format itself. But the EventLogWebEventProvider does its own formatting of individual fields of the event. Indeed, its ProcessEvent method has a big list of checks:

if (eventRaised is WebBaseErrorEvent)
    AddErrorStuff();
if (eventRaised is WebAuthenticationSuccessAuditEvent)
    AddLogonStuff();

So it seemed like a better approach would be to write my own provider. I left the event log provider alone, and I wrote a custom email provider to display errors in a more useful way. This also allowed me to drop some fields from the event report that aren't useful for us. And I was able to construct a much more concise and useful subject line (the subject line that SimpleMailWebEventProvider uses is rather clunky since it assumes it might be spitting out a whole bunch of buffered events in one go).

Not only does my provider include the stack traces for all of the exceptions in the chain, but in the subject line, I display the type of error that is at the root of the problem. So if I am formatting a TargetInvocationException, I drill into its InnerException chain until I find a different exception type, and display that exception type instead.

Oh, one other benefit of building the custom provider instead of using a custom WebEvent was that I was then able to remove the Error handler from global.asax. All I had to do was replace the SimpleMailWebEventProvider with my own provider, and I got the behavior I wanted. Now my email notifications include detailed stack traces.

I'll post the code for this provider once it's run for a little while in production and I'm satisfied that it works reasonably well.