I arrived in Austin Sunday night to get ready for a factory tour on Monday, a kickoff dinner and then two days of briefings from Dell executives, including Michael Dell himself! Dell’s ISG business is growing at a very fast pace and continues to build momentum and focus within the broader organization.

We had a nice overview of the product roadmap, including some of the exciting enhancements Dell is making to their storage products such as the MD3000 and the new EqualLogic PS5000 series iSCSI solutions.

I really enjoyed the Council meeting and it reminds me all over again; what I admire about Dell is the way they and Michael Dell himself stay close to the customer. The entire purpose of this event is to “get it right” and determine meaningful ways to embrace change (including change in the manufacturing process) in order to make their customers more successful. Ah shucks, you may say that all companies behave this way… well I must tell you that is not true and at times, I find it difficult as we continue to grow to stay as close as I would like to all of our customers varying needs and directions.

This concept of gathering, internalizing and embracing customer feedback is a simple principle of Business Success stories. Always trying to improve the pace of change and build meaningful sticky relationships with customers. Dell’s very successful Ideastorm site where customers post product feedback and are active participants in the Dell community is a great example of how to do this right. No other hardware vendor that we have worked with or attempted to work with has ever gone to the extent of embracing change that Dell has during our 5-year relationship.

From the custom factory integration services to the attention to detail in the order and manufacturing, and logistics processes, Dell helps us execute for our customers and I must admit that we could not have built the business as quickly or efficiently without Dell!

So thank you Michael Dell for building a business that embraces change and is focused on helping your ISG customers succeed.

First of all, I learned the Thai alphabet nearly 20 years ago, so I have have a pretty good foundation for the Thai language.   I can read (slowly) and speak better than 99.99+ percent of all foreigners in Thailand; so, I thought it was time to redirect my career to a “new challenge” in the business climate of Thailand.   

This was no small decision.  Your career changes dramatically when you give up a successful consulting practice in the US and dive into business in a foreign land for a new challenge.  I can frankly tell you that often the challenge is sometimes overwhelming.    It is quite difficult as a foreigner to do business in Thailand.

First of all, it is not legal for foreigners to own land in Thailand.  Foreigners can ”own” land using a variety of legal loopholes, proxy owners and shell companies; but all of this is risky and not advised.  Foreigners lose a lot of money coming to Thailand and attempting to buy land.  Some get lucky, but the entire process of foreigners buying and selling land is quite risky.

Foreigners can own condos, under certain conditions, but this results in  inflated prices for condos in Thailand that are traded in an artificial market place.   Condos that are up-to-par with condos in the US can easily cost more than condos in major cities in the US.  Hence, the cost of living is not as cheap as some might believe.

Business can best be described as “protectism” where the government has placed many barriers to entry to foreigners working in Thailand.     Every foreigner must have a work permit and these work permits are expensive and time consuming to maintain.   If you own a business you must pay high professional service fees for auditors to perform annual and semiannual audits even if your business has no income yet.   Firms in Thailand charge thousands of dollars for these ”audits”.      

In addition, if you operate a business, you must have a place of business, so you are forced to lease office space.   Foreigners from the US must be paid a minimum of 50,000 Thai Baht per month, so the government will take 10 percent of that each month as their share of tax withholdings.   Therefore, to start a company, you will pay a lot of money in startup fees, permits, tax, leases, visas, etc.  The entire system is designed to secure money from you, even if you do not have a penny of incoming revenue.

Of course, generating incoming revenue can be quite difficult in a climate of protectionism.   In Thailand, it is easy when you are spending money.  When you are trying to generate income from Thailand, as a foreigner the challenge can seem overwhelming at times.   Many foreigners here give up because the barriers to business here are very high.

On top of all these challenges, which I have not described in detail, is the overall global business slowdown combined with a climate of political instability, which I am sure you have seen in the news.  

Most people I know say it is better to be a tourist here.   Being a tourist is completely different.  Money flows from you, so life in Thailand is fun and friendly, complimentary to the “Land of Smiles” you have heard about.     However, when you are working to have money flow the other direction, flow to you versus away from you, you don’t see the “Land of Smiles” as tourists experience.

Without getting into too many details, I can simply say that a foreigner doing business in Thailand experiences protectionism and, to a certain degree, discrimination, and sometimes I wonder if coming here for a “business challenge” was a good idea.    I was seeking a “new challenge” and I got more than I bargained for!

In a future post on business in Thailand I will discuss issues regarding how little value is placed in intellectual property in Thailand and how this adversely impacts professional services.    I will also touch on how this lack of regard for intellectual property impacts a consulting practice.   Also, I will touch on some cultural differences in how Thais appear to view teamwork, which is very different than in the US.

For those of you who aren’t familiar with XSS Filter, a brief summary is that it is a client-side defense against reflected cross-site scripting (XSS) attacks.  It works by recognizing that reflected XSS attacks inject script into the string that the browser sends to the targeted web server.  If the server doesn’t neuter or strip out the injected script, it gets sent back to the browser and executed in the context of the target web page.  Bad things then happen.  At a high level, XSS Filter remembers the string that the browser sent to the server, and looks at the server’s response to see if any of the script was actually in that string.  If it was, then XSS Filter decides that it got there because it was injected by an XSS attack and blocks the script from executing.  The rest of the web page renders as usual.  This is a vastly oversimplified sketch of XSS Filter – for details, see the post by David Ross, inventor of XSS Filter on the Security Vulnerability Research and Defense blog.

So what does XSS Filter have to do with the SDL?  Well, for almost nine years, since XSS was first discovered at Microsoft, we’ve been trying to figure out effective ways to reduce vulnerability to XSS attacks.  Our focus has been on improving the ways that web page developers code their pages, and we’ve developed a lot of tools and techniques for making web content safer from XSS attacks and for detecting XSS vulnerabilities in live pages.  The SDL requires the use of many of these tools and techniques, and we’re sure we’ve prevented a lot of XSS vulnerabilities from being introduced into Microsoft web pages as a result. 

But while we identify (and the SDL requires) measures that allow developers to avoid classes of vulnerabilities, we also look to identify more sweeping solutions that can either 1) eliminate classes of vulnerabilities, 2) reduce their severity, or 3) reduce the likelihood of attacks being successful.  The process usually starts from deep understanding of a class of vulnerabilities and attacks, and then we broaden defenses from there.  In the case of XSS Filter, David’s years of work researching XSS led him to come up with an approach that blocks many of the most common vulnerabilities to reflected attacks found on the web today.  The solution is compatible with existing web pages (doesn’t “break the web”) and thus we were able to enable it by default for users of Internet Explorer 8.  Because it’s a client-side mitigation, it will help protect users from attacks even though the sites they visit may be vulnerable to XSS. 

Our work on buffer overrun defenses follows a somewhat similar pattern – we started by prescribing coding techniques, banning the use of some APIs, and building tools that detect coding constructs that look like buffer overruns.  As we gained a deeper understanding of how buffer overruns can be exploited, we enhanced the /GS compiler flag and added ASLR in a quest to cause classes of exploits to fail even if a buffer overrun remains.  We’re not yet close to eliminating the SDL requirements for use of tools and coding techniques, but the SDL also requires the use of the mitigations to reduce the severity of vulnerabilities that slip past.  Will we ever get to the point where the mitigating technologies are so strong that we can relax the coding requirements?  Maybe not, but we will continue to introduce technologies that reduce the chances of a successful attack.

Similarly, in the case of XSS, even after IE8 ships, the SDL will continue to require the use of safe web site coding practices and tools such as the Anti-XSS library both to protect users of browsers other than IE8 and to provide protection in recognition of the fact that XSS Filter is a mitigation or defense in depth rather than a complete solution.  But we’ll also be keeping our eyes open (and doing active research) in the quest for an even more effective defense – whether client or server side – that eliminates XSS for good.

This post is a little far afield from the normal content of the SDL blog, but I thought it was important to provide a picture of the role of security science and security research in defining SDL requirements and in making major improvements in software security.  You can read more about our work in security science in the Security Vulnerability Research and Defense blog.

So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root."  Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo:     jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze  VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

To kick off our celebration of our first five years, we asked ScienceLogic founders Dave Link, Richard Chart and Chris Cordray for their thoughts and memories on events leading to today’s milestone. How and why did they set out on this venture? What happened along the way – expected and unexpected? Why were they successful in times when other new (and established) businesses have come and gone?

How did you three put together this team?

We all worked together at a large Managed Service Provider for a couple of years before leaving to start ScienceLogic, so we all knew each other and knew our collective strengths. More importantly, each of us had worked with network management tools on some level (sales and marketing, engineering and product development), and knew first-hand all of the customer pain points, from every perspective. So we left and began rapidly figuring out how to build a better network management solution based upon our real world operational experience..

Dave: One interesting aspect is that our areas of expertise don’t overlap, which has contributed to our success. Chris is excellent with developing the product front-end and interface, Richard handled the backend architecture and engineering and I focused on the technical business side of sales and marketing. Our roles have been to build a product that works well and that provides real value to operations teams that experience the same day to day frustrations that we felt.

Whose idea was it to start the company?

Dave: It was really a collective effort. We were all passionate about “getting it right” and not just starting a company. We knew the industry need and between us, we had the knowledge and skill sets to address all of the right aspects of developing a product and a building a business around it.

What process did you go through to get started?

Richard: From the beginning we knew the type of solution the market needed and we knew that we wanted to build it as an appliance. From different vantage points, we had each experienced the effects of long, difficult and expensive installations that still exist with traditional network tools. Every install has unique variations: there are always different server types, varying hardware and software versions, different patches installed, and on and on. Every installation was time consuming and unpredictable. We knew that an appliance model would address all of these variables and save a lot of time on how quickly customers could achieve immediate value.

The harder decisions were around actually starting the business, assessing the market and of course determining the product pricing.

EM7 completely flips the traditional model of complex, lengthy and expensive deployments. How did you convince others that the EM7 Meta-Appliance product was valid?

Dave: Yes, EM7 totally disrupts the traditional model for network management. While others take a narrow approach, we intentionally designed EM7 to focus on the broad problem – managing the data center. How do you cover a variety of technologies and make sure they work seamlessly together? The vision was to make it easier, not harder, for customers.

Chris: I have to give it to Dave – very early on, he realized the power of a demo. If Dave could get in front of someone, he’d make them a believer. He’d use the Peter Falk/Columbo technique of “let me show you one more thing.” It was very effective. It’s getting easier, but even today people sometimes have to see EM7 in action before they become believers.

Can you describe the early days of running a new business?

Dave: ScienceLogic is a classic case of entrepreneurship. For the first year we worked out of our basements. We kept the costs low in every conceivable way and spent the first year developing the product before we even made a sale.

Chris: We stayed at lots of odd places when we were on the road, took cheap flights with multiple layovers and purchased lots of our first test equipment on eBay. This was during the dot-com bust so there was lots of equipment for sale on eBay, really cheap!

Richard: The amount of equipment I had in my house was absolutely crazy. Back then, servers were huge – I had a Cisco 6509 Catalyst, a Compaq Proliant DL380, Brocade switch, IBM Netfinity 4500R, and tons of other machines.

Chris: I had to install a new circuit box at home because I was blowing breakers. I remember when that 6509 crashed, we revived it and it died again. The second death was final.

So you started in your houses – what was your first office space?

Dave: My friend, the CEO at Ernst & Young Technology had a few extra cubes and a data center in their office that they graciously allowed us to use. Their help was an important step in helping us really formalize the business. We started doing well and adding people, but ironically, their company was downsizing. Before long, many of their original YET people were gone and the ScienceLogic team kept growing in to the open cubes.

Our first leased space was converted warehouse space in Chantilly, VA that once housed an internet radio station. It was cool – it had a large salt water fish tank, a loft, a spiral staircase and a Star Trek door that retracted into the walls with the customary lights and “whooshing” sound.

We outgrew the Chantilly space, leading to our current office in Reston, VA.

Who was the first ScienceLogic customer?

Our first paying customer was Martins Point Health Care. We deployed there in July 2004 and are pleased to say they continue to be a ScienceLogic customer. Other early (and still) EM7 customers include Navy Knowledge Online and the Department of Transportation. Nearly all of our customers are still actively using EM7 and renewing their maintenance.

Where do you see the company in the next 5, 10 or 15 years?

Well, our revenue has doubled year-over-year in each of the last three years, so of course we’d like to continue to grow like that or even faster. In five years we’ve gone from three founders to the point where Dave does not know everyone’s fondest childhood memory. We’ll continue to scale our growth to cover the demands of our growing customer base.

Where do you see the industry going over the coming years?

Chris: IT is always moving and gaining in complexity, so network management is also becoming more complicated. There’s increasing diversity, new standards, virtualization and cloud computing. All of these are today’s technologies. Customers have a mix of the old and the new, so EM7 has to accommodate and support both.

Richard: Each generation of products has a new set of ways to monitor, but the “old” doesn’t go away. Even when a new, hot technology comes along, the old technologies still need to be supported. We work to ensure EM7 keeps up with both.

Dave: After five years we’re just hitting our stride and we’re just now reaching the tipping point in awareness of ScienceLogic and EM7. We’re all still passionate about the product and as Chris and Rich said, there’s still a lot do. We’ll continue disrupting the market with EM7. Our vision hasn’t changed, and with the increasing levels of automation that customers demand, the market needs are greater than ever. Our future is as bright, or brighter, than ever and we’ll continue to be looking for smart ways to automate traditionally manual IT Operations processes.

What’s your advice for someone interested in starting their own business?

Chris: Be passionate. That’s what has gotten me through the tough times. I didn’t really appreciate this thought when I heard others say it before. But it’s very true.

Richard: I agree. We met and talked with lots of people who told us, “That’s been done before.” But we kept going because we truly believed in what we were doing and we knew that while our approach was different, that it would be successful.

Richard: Be fearless. You can’t be too nervous and you need to be able to expect and handle the stress because it will be there. You have to learn to accept the stressful times as a necessary part of the process of starting out on your own.

Dave: Know your niche from the beginning and give potential customers a compelling reason to trust you and really benefit from your solution. You have to know the problem, see the gap and have a clear and consistent vision of how to solve the problem. Then you have to execute. If you don’t build your team with “doers” you won’t make it.

Chris: It helps to have friends. ScienceLogic was built on friendships and relationships, starting with the three of us. If you look at our team, most of our hires are referrals – people who developed and maintained great connections with other great people throughout their careers. Maintain your connections and keep in touch with your network of friends.

We are regularly hearing from our security clients about their difficulties finding people with the right skills – or when they do finally find them, these people are too costly to employ because their skills are in such demand.

Indeed, the “unavailability of people with the right skills” was cited as a top challenge for security groups in both our enterprise and SMB surveys.

In comparing need for talent across 25 different IT roles, Forrester analysts came to the conclusion that information security experts are among the hottest roles in IT, sharing the top spot with information/data architects.

The skills shortage is likely to get worse before it gets better. We’re unlikely to see a significant spike in security experts’ salaries to attract those we need to hire: large changes in compensation for senior security personnel would run against the current of economic belt-tightening. Another typical approach to offsetting the shortage would be to train up: foster the career development and advancement of existing security personnel on our payroll. However, with all the outsourcing that is going on – and which will increasingly occur – there is a shrinking pool from which to find people with “the right stuff” worth championing their advancement.

We could look outside of security to others in IT, or even to co-workers in other departments or business groups. But given how poor a job IT Security does of marketing its value proposition, I don’t hold much hope for attracting non-security people.

What do you think? Are we about to hit a very big wall when it comes to skills and staffing? Are you presently feeling the pain of a skills shortage? Do you see such a shortage looming? What measures are you taking to acquire and nurture talent? Which ones are successful and why?

I welcome your thoughts on the topic.