Making IT work as One

How does my company stay efficient while we’re using technologies around interoperability? How can innovation help my business?

Top business needs:

• Reduce cost
• Manage complexity
• Mitigate risk

Mixed IT environments are a reality for almost all organizations. Different environments, architectural strategies, desktop profiles, etc. There are benefits to having mixed source environments, although homogenous environments are ideal. On average 46,000 hours in an organization are spent on Sarbanes-Oxley standards.

Some considerations to make IT work as one:

• Strategy
• Solutions
• Ecosystem

Strategy

Actionable strategy is key. The emergence of three silos (applications, systems and infrastructure, and operations) are now moved into one. There is a lot of pressure to make these pieces come together.

Solutions

You need focused solutions to solve problems today while keeping an eye to the future. There are three main needs: the data center, end-user computing, and identity and security. This is also what is the most important to the market right now. The end goal is the agility of the data center.

Data Center Challenges

• Create an agile IT infrastructure
• Address power and space constraints
• Deliver performance, security and availability
• Manage hardware, software and labor costs
• Meet service level agreements

Data Center Solutions

• Workload management - green IT and server efficiency, unified physical and virtual environment
• Virtualization and Consolidation - business continuity and disaster recovery
• Enterprise Servers

End-User Computing Solutions

• Collaboration
• Enterprise desktops - Novell uses Linux and Open Office, interesting to note
• Endpoint management

Identity and Security Challenges

• Minimize risk, uncertainty and policy violations
• Provide timely and secure access to information
• Ensure, document and prove information security
• Reduce the cost of proving compliance
• Reduce the cost and complexity of governance

Identity and Security Solutions

• Identity and Access Management - user provisioning, role management, access management
• Compliance Management - Audit, Governance, Risk Management and Compliance (GRC), IT controls automation, Security, Information and Event Management (SIEM)

Ecosystem

The ecosystem is powerful. Companies should challenge partners for innovation and interoperability.

Community Innovation - open source and open standards

IT Landscape - Mixed IT Environments

• Consulting, systems integration vendors
• Application vendors
• Systems software vendors (Novell)
• Hardware, network vendors

How does your ecosystem help your company? How do your partners help? What is their role in the industry to help you? How are all the vendors in the industry helping you?

In the case of phishing, it is relatively clear. The developers believe the PKI book. The PKI people believe in the efficacy of digital signatures to prove stuff. The cryptographers believe in the perfection of mathematics, and the security world believes in the completeness of their own learning. They are all wrong, but only at the large level of generalisations, not at the detailed level of particular claims. Any one of the claims, in isolation can be shown to be true. But, generalising these brittle claims to be solid building blocks is a completely different question. Few of the claims are strong enough to partake in a general model without severe support; the general model of secure browsing is the best evidence of how it is secure in name only.

How then is it built? By accident or by design, a series of claims meet together in a holy ring of righteous architecture. Each of the proponents claim loudly that their part is strong, but the ring has no strength. Eventually, one of the claims in the links is broken. For phishing, the browsers never did have the potential to show authenticity; not only did they not have the security strength to do it (c.f., Skype v. CSRF), they didn't even do it in practice (recall the lost padlock?), and their recent efforts to show authenticity (c.f. colour debate) reveal how far they are from understanding even the goal, let alone the implementation. Once that link was broken, and money was made, all the others revealed their weaknesses, as crooks systematically worked to breach the lot.

If we look at the wider financial collapse, now underscored by the nationalisation of the worlds biggest financiers of mortgages ($ 5.3 trillion.... or is it $ 5.4 ?), we see the same pattern. The bankers believed in their product. The originators believed in their origination, the securitizers believed in their free market and accurate price, and the holders believed in the assets. The CDO, the subprime, the other 100 special names, each was a contract. Each was clear in and of itself. But, when placed end-to-end, in a line, with a bunch of other agreements, the claims that were good in isolation were not strong enough to participate in the super-claim made of the overall edifice.
The financial system was built like a bridge; each piece rested on the previous one. And then, the clever architects bent the bridge around ... and around again, until the first piece met the last. The elegant keystone of finance was to finally lift up the first one to rest on the last.

Thus, the banks themselves invested their capital in their own product.

adaptive process management (n.) an element of resource and business process management, adaptive search and event processing. Sometimes referred to as “Level 4” event processing or process refinement.

application concept (n.) a definition of a set of properties that represent the data fields of an application entity. An application concept can describe relationships among themselves. For example, an order concept might have a parent/child relationship with an item concept. A department concept might be related to a purchase requisition concept based on the shared property, department_id. Application concepts can include an application state model.

application state modeler (n.) a UML-compliant application that allows you to model the life cycle of a concept instance — that is, for each instance of a given concept, you can define which states it will pass through and how it will transition from state to state. States have entry actions, exit actions, and conditions, providing precision control over the behavior of an event processing agent. Transitions between states also may have rules. Multiple types of states and transitions maximize the versatility and power of the application state modeler.

derived event (n.) an event that is created as a result of processing one or more other events.

complex event (n.) an event that is a situation-entity abstraction of two or more simple, derived or other complex events.

complex event processing (n.) CEP is a technology for extracting information from message-based systems. CEP is primarily an event processing concept that deals with the task of processing multiple events from an event cloud with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes.

event (n.) a instance of an event definition. It is an immutable object that represents a business activity that happened at a single point in time. Just as one cannot change the fact that a given activity occurred, one cannot change an event — events are immutable.

event aggregation (n.) the aggregation of simple, derived or complex events into higher levels of event abstractions.

event definition (n.) a set of properties related to a given activity that represents an important or interesting change of state in a human, system or computational activity. An event definition includes event properties such as event priority, event time to live (TTL), and a description of the payload, which is comprehensive information related to the activity that occurred. Events expire when the TTL has elapsed, unless the event processing agent has instructions to consume them prior to that time.

event channel (n.) a communications channel in which events are transmitted from event source to event receivers, typically received as electronic messages. Each channel can have multiple destination and. events can be configured to transmit to a default destination. JMS is an example of an event channel.

event cloud (n.) a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events. Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud.

event-driven (n.) the behavior of a human, system or computational entity whose execution or actuation is in response to events, typically received as electronic messages.

event-driven architecture (n.) an architectural style for distributed computing applications in which some of the components are event-driven and communicate by means of events.

event processing (n.) computing that performs operations on events, including modifying, creating and destroying events.

event-object (n.) an software object that represents an event, generally for the purpose of computer processing, that exhibits both encapsulation, inheritance and polymorphism.

event prediction (n.) computational activity where the impact of events, complex events, and situations caused by events identified, including both opportunity or threat. Sometimes referred to as “Level 2” event processing, impact assessment or predictive analytics.

event pre-processing (n.) computational activity where events are cleansed or normalized to produce semantically understandable data. Sometimes referred to as “Level 0” event processing.

event processing (n.) computational activities on events dealing with the association, correlation, and combination of event data and information from single and multiple event sources to achieve refined identity and situation estimates for observed event objects, and to achieve complete and timely assessments of opportunities, threats, and their significance. Event processing is characterized by continuous refinements of event estimates and assessments and by evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

event processing agent (n.) an EPA is a computational entity that performs event processing.

event processing network (n.) a set of event processing agents and a set of event channels connecting them.

event properties (n.) data representation of an event, typically by name-value pairs of type string, integer, real, boolean or a complex data type.

event refinement (n.) filter, identify and track events & make initial processing decisions based on association, correlation and state estimation. Sometimes referred to as “Level 1” event, or event-object, track and trace.

event stream (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

event stream processing (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

rule (n.) defines what triggers unusual, suspicious, problematic, or advantageous activity within an event processing agent and what the EPA does when it discovers these types of activities. Rules execute actions based on certain conditions on events, instances, or a combination of both. A rule includes a group of condition-rule statements and action-rule statements. The condition statements instruct the EPA what to look for in events, and action statements instruct the EPA how to respond when conditions are met. If all the conditions in a rule are satisfied by events or instances or both, the EPA fires the actions. The action might be to execute tasks, create an event instance, modify property values in an event instance, create and send an event, or something else.

rules engine (n.) a type of event processing agent that uses a declarative programming model to process events. Formally described as “an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet“. Informally, it can be any system that uses rules, in any form, that can be applied to data to produce outcomes.

rule language (n.) is an artificial language that is used to control the behavior of an event processing agent. Rules languages, like human languages, have syntactic and semantic rules to define meaning.

situation refinement (n.) identify situations, or complex events, based on event clustering, event-event relationships and relationship analysis and context. Sometimes referred to as “Level 2” event processing.

simple event (n.) an event that is not an abstraction or composition of other events.

virtual event (n.) an event that is imagined, modeled or simulated.

CEP describes an architecture, just like SOA describes an architecture and just like EDA describes an architecture.

For example, you do not buy an SOA.   An SOA describes an architectural style of programming via components that are involved as services in a distributed network architecture - a service-oriented, or service-based architecture.

The concept of CEP does not have “the A-word” like SOA and EDA, but none-the-less, CEP describes an architecture, not a product.   Do not make the mistake of thinking in terms of “buying CEP”, just like you do not buy an SOA or an EDA.  You think, plan and design in terms of CEP, just like you should do in an SOA or EDA.  These are constructs, not products.

In other words, for “true CEP” you need a number of components, some of the components might be in the architectural style of SOA, others might be in the architectural style of EDA.   Your solution architecture for solving a complex event processing problem might have request-reply transactions, or it might have fire-and-forget messages.  You might have a Neural Networking component for analytics and a rules component for filtering, mediation and scheduling.  You might even have a stream processing component performing as a high performance filter and pattern matcher on streaming data where the output is forwarded to a Bayesian Classifer for further processing.

My key message in this post is that CEP requires a number of technologies to solve complex distributed computing problems.   Do not be fooled into thinking that a single product is “CEP” no more than a single product is SOA or EDA.

The information security field is not yet fully mature; there is a lack of cohesive interoperable framework.   The rapidly evolving landscape adds to the existing problem. There are several examples: Intrusion Detection System (IDS) was quickly overtaken by Intrusion Prevention System (IPS).  On the Firewall arena: the focus has moved from perimeter security to end point security.  There are some security visionaries who are preaching inside-out security approach i.e. building products with information security in mind from the beginning.

Threats are moving higher up in the OSI stack making it harder to detect. Hackers are becoming more sophisticated – there are powerful free open source hacking tools available at their disposal. Security managers driving security initiatives without co-ordination can result in pieces of puzzle that don't fit well. Agency problem i.e. security managers thinking more about their personal advancement rather than security of the company is bad for the company’s security initiative. Security leaders who do not have a clear vision of

security at the component level, the administration level and the strategy level can only make information security even more convoluted. The CISO and acting CIO of US Dept of Veteran affairs resigned after the breach in May, 2006 where personal data of 26 million veterans and more than 2 million service members was stolen. This clearly demonstrates the accountability and visibility of security leadership.

The attitude of IT security leaders and security team members has a significant impact on security.  Reckless buying of information security technology can result in wasteful expenditure and very little gain in efficiency. Not understanding the business perspective of security issues or security perspective of business issues can lead to poor security decisions. Using security as a mechanism to gain control rather than using it as a tool to reduce risk can only diminish the perceived value of security initiative. Implementing security as an afterthought rather than building it into the framework not only result in poor architectural decision. Security investment is more like buying insurance. Thinking security as a vehicle providing an ROI can result in wrong expectation and lead poor decision. The business in which a company operates contributes largely to the perceived importance to security. Financial institutions usually have a higher bar on security because of the very nature of their business and their exposure legal liability. It is a good idea for many technology companies to emulate financial institutions to raise their information security bar.

It could be a pipedream to accomplish complete  information security but accomplishing a well managed information security program is an attainable possibility.

Complex event processing (CEP) is a new technology. It can be applied to extracting and analyzing information from any kind of distributed message-based system. It is developed from the Rapide concepts of (1) causal event modeling, (2) event patterns and pattern matching, and (3) event pattern maps and constraints. Complex event processing can be applied to a wide variety of Enterprise monitoring and management problems, from low level network management to high level enterprise intelligence gathering.

Applications of Complex Event Processing:

• Instant Insight  - hierarchical event viewing applied to the Enterprise IT layer. (coming soon)
  • Analysing business processes (paper in pdf format)
• Network Level Monitoring and Management (Powerpoint presentation)
• Cyber Security: Network Intrusion Detection
• Enterprise Monitoring and Management (coming soon)
• Modeling and Simulation of Collaborative Business Processes
• Business Policy Monitoring. (coming soon)
• Analysis and Debugging of Distributed Systems (coming soon)

Presentations:

• “Complex Event Processing: An Essential Technology for Instant Insight into the Operation of Enterprise Information Systems,” lecture at the Stanford University Computer Systems Laborary EE380 Colloquium series. Video of the lecture (duration: 60 minutes).

Publications:

• Complex Event Processing in Distributed Systems. David C. Luckham and Brian Frasca, Stanford University Technical Report CSL-TR-98-754, March 1998, 28 pages.Abstract: Complex event processing is a new technology for extracting information from distributed message-based systems. This technology allows users of a system to specify the information that is of interest to them. It can be low level network processing data or high level enterprise management intelligence, depending upon the role and viewpoint of individual users. And it can be changed from moment to moment while the target system is in operation. This paper presents an overview of Complex Event Processing applied to a particular example of a distributed message-based system, a fabrication process management system. The concepts of causal event histories, event patterns, event filtering, and event aggregation are introduced and their application to the process management system is illustrated by simple examples. This paper gives the reader an overview of Complex Event Processing concepts and illustrates how they can be applied using the Rapide toolset to one specific kind of system.
   
• Event Mining with Event Processing Networks. Louis Perrochon and Walter Mann and Stephane Kasriel and David C. Luckham, The Third Pacific-Asia Conference on Knowledge Discovery and Data Mining. April 26-28, 1999. Beijing, China, 5 pages.Abstract: Event Mining discovers and delivers information and knowledge in a real-time stream of data, or events. We show that the process of delivering knowledge by searching patterns in data and subsequent abstraction of found patterns can be applied in real-time to a complex, asynchronous system. Our event processing engine consists of a network of event processing agents (EPAs) running in parallel that interact using a dedicated event processing infrastructure. The agents can be configured at run-time using a formal pattern language. The underlying infrastructure (1) provides an abstract communication mechanism and thus allows dynamic reconfiguration of the communication topology between agents at run-time and (2) provides transparent, location-independent access to all data. These features allow dynamic allocation of EPAs to different threads and processes on different machines at run time.
   
• eJava - Extending Java with Causality. Alexandre Santoro and Walter Mann and Neel Madhav and David Luckham, Proceedings of the 10th International Conference on Software Engineering and Knowledge Engineering, June 1998, 10 pages.Abstract: Programming languages like Java provide designers with a variety of classes that simplify the process of program development. Some of these classes allow one to easily build multithreaded programs. Though useful, especially in the creation of reactive systems, multithreaded programs present challenging problems such as race conditions and synchronization issues. Validating these programs against a specification is not trivial since Java does not clearly indicate thread interaction. These problems can be solved by modifying Java so that it produces computations, collections of events with both causal and temporal ordering relations defined for them. Specifically, the causal ordering is ideal for identifying thread interaction. This paper presents eJava, an extension to Java that is both event based and causally aware, and shows how it simplifies the process of understanding and debugging multithreaded programs.
   
• Event-Based Execution Architectures for Dynamic Software Systems. James Vera, Louis Perrochon, David C. Luckham.
  Proceedings of the First Working IFIP Conf. on Software Architecture. 1999. San Antonio, Texas.Abstract: Distributed systems’ runtime behavior can be difficult to understand. Concurrent, distributed activity make notions of global state difficult to grasp. We focus on the runtime structure of a system, its execution architecture, and propose representing its evolution as a partially ordered set of predefined architectural event types. This representation allows a system’s topology to be visualized, analyzed and con-strained. The use of a predefined event types allows the execution architectures of different systems to be readily compared.
   
• Using Context-Based Correlation in Network Operations and Management. Louis Perrochon (work in progress, mail author for newest version)Abstract: Network operation consists to a large degree of reaction to activities happening in the network. Better knowledge of the network at any time allows more appropriate reactions. On the example of intrusion detection, we show how context-based correlation of such activities can provide a more detailed view of the network in shorter time. We first present how we model context and then describe the architecture of the Stanford University CEP context-based correlator. Correlation is specified as event patterns in a declarative language that allows us to specify what needs to be detected, instead of specifying how it should be detected. CEP introduces the concept of causal context to intrusion detection. The correlator is able to process events on-line, as they are generated and it can be reconfigured at dynamically. We then show how it increases detection rate, reduce false alarms, and detect large-scale attack patterns at an early stage.

Blogger: Ramon Krikken

Many different conference tracks, many different perspectives on 'security' and how to best implement it. I spent most of my time in the Service-Oriented Architecture (SOA) track, looking for little nuggets of wisdom to help with my upcoming SOA security overview, and I certainly did find some. There were - luckily - no huge upsets, but there were certainly lots of questions on how to to implement controls in a service-oriented environment. What was once only the question of what Web Services standards to use, has now evolved to discussions on everything from high-level architecture to the minutiae of security token translations.

One of the discussions in SOA security revolves around the location of controls. In general the architecture is best served if most controls, such as authentication and authorization, are externalized from the application code. It creates a separation of concerns, and usually makes management and auditing more straightforward. So some of the different infrastructure components, like web services modules and the XML gateways, support access control, encryption, and data validation features. Some vendors would like us to believe that pushing all this functionality into their well-packaged, standards-based solution is going to solve the 'security problem,' but does it?

It all works out well as long as we can - in the true spirit of service orientation - view the service as a black box, but that isn't necessarily possible from a security perspective. Certain functionality, like the compute-intensive XML schema validation, is an ideal candidate for infrastructure security, and so is service-to-service authentication. User authorization is all over the map depending on its granularity and requirements for data-awareness. With encryption it also depends on whether we're talking data transport or storage. Service-enabling legacy applications also throws us a curve-ball because of, amongst things, the need for identity and access token mapping that take us into the darkness of the black-box service.

In other words, both applying controls in service orientation, and applying service-oriented principles to security, aren't necessarily as straightforward as some may want us to believe. Security professionals probably already had a feeling this would be the case; we're a bunch of skeptics, after all. But if it's the case that enterprise architecture is far ahead of security architecture in SOA planning or implementation, then there may be some misunderstanding in the organization on how to secure the infrastructure and services. At the surface, and in the common case, the decision to put controls at the infrastructure level seems simple. The devil, it appears, is very much in the details that are invisible to us in some of the higher-level architectural discussions.

Fortunately, all is not lost. We may have thought that 'the SOA train has left the station, and security is not on board,' but it now appears - at least from Burton Group's research - that the train isn't necessarily all too far down the tracks yet. We need to work with the architects to create a security strategy that matures along with the other aspects of SOA implementation, work with the development team to overcome the challenges of building security into the SDLC, and most of all, work with ourselves to make sure we're able to apply consistent principles of information assurance no matter what the next best thing in SOA technology is. There is time to get things right, and the best time to start is now. 

Blogger: Ramon Krikken

Many different conference tracks, many different perspectives on 'security' and how to best implement it. I spent most of my time in the Service-Oriented Architecture (SOA) track, looking for little nuggets of wisdom to help with my upcoming SOA security overview, and I certainly did find some. There were - luckily - no huge upsets, but there were certainly lots of questions on how to to implement controls in a service-oriented environment. What was once only the question of what Web Services standards to use, has now evolved to discussions on everything from high-level architecture to the minutiae of security token translations.

One of the discussions in SOA security revolves around the location of controls. In general the architecture is best served if most controls, such as authentication and authorization, are externalized from the application code. It creates a separation of concerns, and usually makes management and auditing more straightforward. So some of the different infrastructure components, like web services modules and the XML gateways, support access control, encryption, and data validation features. Some vendors would like us to believe that pushing all this functionality into their well-packaged, standards-based solution is going to solve the 'security problem,' but does it?

It all works out well as long as we can - in the true spirit of service orientation - view the service as a black box, but that isn't necessarily possible from a security perspective. Certain functionality, like the compute-intensive XML schema validation, is an ideal candidate for infrastructure security, and so is service-to-service authentication. User authorization is all over the map depending on its granularity and requirements for data-awareness. With encryption it also depends on whether we're talking data transport or storage. Service-enabling legacy applications also throws us a curve-ball because of, amongst things, the need for identity and access token mapping that take us into the darkness of the black-box service.

In other words, both applying controls in service orientation, and applying service-oriented principles to security, aren't necessarily as straightforward as some may want us to believe. Security professionals probably already had a feeling this would be the case; we're a bunch of skeptics, after all. But if it's the case that enterprise architecture is far ahead of security architecture in SOA planning or implementation, then there may be some misunderstanding in the organization on how to secure the infrastructure and services. At the surface, and in the common case, the decision to put controls at the infrastructure level seems simple. The devil, it appears, is very much in the details that are invisible to us in some of the higher-level architectural discussions.

Fortunately, all is not lost. We may have thought that 'the SOA train has left the station, and security is not on board,' but it now appears - at least from Burton Group's research - that the train isn't necessarily all too far down the tracks yet. We need to work with the architects to create a security strategy that matures along with the other aspects of SOA implementation, work with the development team to overcome the challenges of building security into the SDLC, and most of all, work with ourselves to make sure we're able to apply consistent principles of information assurance no matter what the next best thing in SOA technology is. There is time to get things right, and the best time to start is now.