Economists have long understood the corollary concept of Coase's ceiling, a point above which organizations collapse under their own weight -- where hiring someone, however competent, means more work for everyone else than the new hire contributes. Software projects often bump their heads against Coase's ceiling: recall Frederick P. Brooks Jr.'s seminal study, The Mythical Man-Month (Addison-Wesley, 1975), which showed how adding another person onto a project can slow progress and increase errors.

What's new is something consultant and social technologist Clay Shirky calls "Coase's Floor," below which we find projects and activities that aren't worth their organizational costs -- things so esoteric, so frivolous, so nonsensical, or just so thoroughly unimportant that no organization, large or small, would ever bother with them. Things that you shake your head at when you see them and think, "That's ridiculous."

Sounds a lot like the Internet, doesn't it? And that's precisely Shirky's point. His new book, Here Comes Everybody: The Power of Organizing Without Organizations, explores a world where organizational costs are close to zero and where ad hoc, loosely connected groups of unpaid amateurs can create an encyclopedia larger than the Britannica and a computer operating system to challenge Microsoft's.

Shirky teaches at New York University's Interactive Telecommunications Program, but this is no academic book. Sacrificing rigor for readability, Here Comes Everybody is an entertaining as well as informative romp through some of the Internet's signal moments -- the Howard Dean phenomenon, Belarusian protests organized on LiveJournal, the lost cellphone of a woman named Ivanna, Meetup.com, flash mobs, Twitter, and more -- which Shirky uses to illustrate his points.

The book is filled with bits of insight and common sense, explaining why young people take better advantage of social tools, how the Internet affects social change, and how most Internet discourse falls somewhere between dinnertime conversation and publishing.

Shirky notes that "most user-generated content isn't 'content' at all, in the sense of being created for general consumption, any more than a phone call between you and a sibling is 'family-generated content.' Most of what gets created on any given day is just the ordinary stuff of life -- gossip, little updates, thinking out loud -- but now it's done in the same medium as professionally produced material. Unlike professionally produced material, however, Internet content can be organized after the fact."

No one coordinates Flickr's 6 million to 8 million users. Yet Flickr had the first photos from the 2005 London Transport bombings, beating the traditional news media. Why? People with cellphone cameras uploaded their photos to Flickr. They coordinated themselves using tools that Flickr provides. This is the sort of impromptu organization the Internet is ideally suited for. Shirky explains how these moments are harbingers of a future that can self-organize without formal hierarchies.

These nonorganizations allow for contributions from a wider group of people. A newspaper has to pay someone to take photos; it can't be bothered to hire someone to stand around London underground stations waiting for a major event. Similarly, Microsoft has to pay a programmer full time, and Encyclopedia Britannica has to pay someone to write articles. But Flickr can make use of a person with just one photo to contribute, Linux can harness the work of a programmer with little time, and Wikipedia benefits if someone corrects just a single typo. These aggregations of millions of actions that were previously below the Coasean floor have enormous potential.

But a flash mob is still a mob. In a world where the Coasean floor is at ground level, all sorts of organizations appear, including ones you might not like: violent political organizations, hate groups, Holocaust deniers, and so on. (Shirky's discussion of teen anorexia support groups makes for very disturbing reading.) This has considerable implications for security, both online and off.

We never realized how much our security could be attributed to distance and inconvenience -- how difficult it is to recruit, organize, coordinate, and communicate without formal organizations. That inadvertent measure of security is now gone. Bad guys, from hacker groups to terrorist groups, will use the same ad hoc organizational technologies that the rest of us do. And while there has been some success in closing down individual Web pages, discussion groups, and blogs, these are just stopgap measures.

In the end, a virtual community is still a community, and it needs to be treated as such. And just as the best way to keep a neighborhood safe is for a policeman to walk around it, the best way to keep a virtual community safe is to have a virtual police presence.

Crime isn't the only danger; there is also isolation. If people can segregate themselves in ever-increasingly specialized groups, then they're less likely to be exposed to alternative ideas. We see a mild form of this in the current political trend of rival political parties having their own news sources, their own narratives, and their own facts. Increased radicalization is another danger lurking below the Coasean floor.

There's no going back, though. We've all figured out that the Internet makes freedom of speech a much harder right to take away. As Shirky demonstrates, Web 2.0 is having the same effect on freedom of assembly. The consequences of this won't be fully seen for years.

Here Comes Everybody covers some of the same ground as Yochai Benkler's Wealth of Networks. But when I had to explain to one of my corporate attorneys how the Internet has changed the nature of public discourse, Shirky's book is the one I recommended.

This essay previously appeared in IEEE Spectrum.

I may be new to the team, but I’m (reasonably) old to the game. My name is Tyler Shields and I’m the latest addition to the Veracode research team. I started at Veracode in September 2008 as a Senior Security Researcher and have been immediately thrown into the fire. Working for a fast paced, highly energetic company like Veracode, keeps you busy and challenges you every day. I plan to blog on the most interesting pieces of my work with Veracode and hope that you find it enlightening or at the very least entertaining.

In the past I have worked as the security engineer at a .com startup, as an incident response and forensics specialist for the United States Postal Service (think HUGE network), and most recently as a security consultant for @stake and Symantec. I have consulted on engagements for Fortune 500 companies, most major financial institutions, and the highest levels of the United States government. As a consultant my focus was on anything related to application security including, application penetration assessments, product security assessments, secure development lifecycle consulting, and secure application architecture engagements. I lead the @stake/Symantec Application Security Center of Excellence that was used to help guide the knowledge of the global consulting team. I also spent time as the lead for the Symantec Vulnerability Research program in which a number of interesting vulnerabilities were discovered and publicly released. In my spare time I enjoy reverse engineering and malware research. I recently completed my graduate degree in Information Security/Computer Science from James Madison University in Virginia.

So… Here’s to a new job, a new blog poster, and of course lots of fun to come.

Here we talk to John Proctor, who started out as one of our first customers (and the first MSP customer). And he believed in it so much, he eventually became part of the ScienceLogic team. (Remember “I’m not only the President, I’m also a client” from the Hair Club for Men?)

John shares his perspectives about the service provider world and why he took a chance on a little-known product called EM7.

ScienceLogic: What is your background? How many years have you worked as a service provider and for what types of companies?

John Proctor: I have been working with Service providers for over twelve years. I worked at a major regional service provider for six years and before that I designed and built national and international networks for ISP’s and Fortune 500 companies as a consultant for PriceWaterhouseCoopers and WorldComm.

ScienceLogic: You were one of the first customers of EM7 – why did you choose it and how did you get over the hurdles associated with using a start-up company’s product?

John Proctor: We were actually customer number five. Back in 2004 when we evaluated and purchased EM7 we could see that EM7 provided about 80% of what we were looking for in one integrated solution right out of the box. One of the things that sold us on EM7 was that the ScienceLogic founders had all previously worked for a service provider, so we knew they understood our business and our challenges. But in the end, it comes down to features. Once we compared EM7 functionality to the alternatives, it was clearly a “no brainer.”

ScienceLogic: What other alternatives were being considered?

John Proctor: Well, we had started with a few point solutions, but as our business and product offerings matured, this resulted in a growing number of point solutions. What started with 3 or 4 ended up as 14 separate tools. They all had strengths but what they didn’t have was integration and because of this they could not scale. And, if the tools could not scale, our business could not grow.

So, naturally we started looking at framework solutions, but they are expensive to buy, expensive to implement, and expensive to maintain. At one point, we even considered some open source projects. There were several that showed promise, but we would still be stuck with tools that were not integrated. So then we considered hiring developers to cobble something together that would work for our business. The only problem with this alternative was that we felt it would take 6 to 8 months before we could have something viable to work with.

ScienceLogic: What products were you using before EM7? What were your goals?

John Proctor: Before we purchased EM7 we used 14 different point solutions to deliver our products and services to the marketplace. Tools like NetCool, Openview, Argent, Heat, What’s Up Gold as well as several other point solutions, vendor specific applications and manually updated spreadsheets. And, as I mentioned before, this does not scale. This also adds a great deal of complexity when you begin to consider business continuity and disaster recovery. All these tools were vital to the delivery of our products and services. Any service provider will tell you it is all about uptime. So if the product is uptime, the tools used to deliver it have to be available 24×7x365.

Our goals were simple: scale and redundancy. As it turns out, the solution was simple as well. EM7 provided a tool that could replace the functionality of almost half of the existing point solutions and the applications that could not be replaced were integrated with EM7 to provide our staff with a “single pane of glass” to see the status and performance of each area of the business from one application. We had visibility into everything from facility systems to applications using EM7.

ScienceLogic also delivers an extensible configuration that addressed uptime and redundancy. We deployed collectors throughout our network that reported back to a central pair of redundant database servers and with this configuration we were able to perform backups and add capacity without taking the system down.

ScienceLogic: Why are service providers different from enterprises? How are their needs different?

John Proctor: First and foremost, service providers face the same challenges that only the largest enterprises ever face and they also have many unique challenges that only service providers experience.

One challenge we faced was that we had multiple datacenters in different states. They were all interconnected with plenty of bandwidth between each site, but the tools were not designed to be used across the WAN. Our staff in our remote data center did not have the same access as our staff in the corporate office. Since EM7 is web-based, it immediately eliminated this problem.

Another challenge is that service providers must manage systems across multiple domains. Back in the early version of a specific tool we were using before EM7, the only way you could implement it across multiple domains was to put the same username and password on every computer that you monitored. Beyond the security concerns, maintenance was a nightmare. Anytime we had to change the password, we would get locked out of dozens upon dozens of systems. When the password was changed on the monitoring server, it would attempt to login to the remote machines and fail. Repeated attempts would result in the account getting locked. I think that vendor eventually addressed this issue, but service providers seldom find tools that were designed for their unique situations.

ScienceLogic: How is EM7 geared to service providers?

John Proctor: Enterprise IT is a trusted part of the business; they are one of the team. Service providers are outsiders that must earn trust by showing the customer exactly what they are doing.

EM7 provides a multi-tenant environment that allows service providers to manage systems across many different customers while at the same time providing the customer access to see the same information but only what’s relevant to them.

EM7 was built by service providers and even includes a few features just for them. Two of my favorites are bandwidth billing and the emergency notification system. Take bandwidth billing, for instance. EM7 provides a way to collect bandwidth utilization, store subscription information, and calculate a bill from any one of about 10 different methodologies. And at the end of the billing period, EM7 sends the completed report out to whomever you chose via email.

Another unique service provider feature is the emergency notification system. EM7 allows the provider to track what customers used their unique infrastructure components. If they have to perform maintenance on the infrastructure component or have a problem they can send an email to all of the impacted customers in a matter of minutes.

ScienceLogic: What trends do you see for service providers? What about big trends such as virtualization and cloud computing – how will they impact service providers?

John Proctor: Virtualization is really hot for service providers right now and for the same reasons as in the enterprise. Service providers run data centers and data centers must be powered and cooled. So, anytime they can use a virtual server instead of adding physical equipment it is a good thing. But then you add the complexity that multiple customers reside on the same host and you must track things like bandwidth utilizations by guest OS, and it all gets a little harder. Lucky for us this is not a problem for EM7.

I still think it’s early days for cloud computing. Depending on who you talk to, much of what service providers (especially the big ones) have already been doing with SAAS offerings and hosted applications could be described as cloud computing already. In which case, service providers are ahead of the game. But whatever the “final” definition, cloud computing actually shares many similarities with virtualization – in that service providers (or enterprises) will need to be able to manage far more “devices” in real-time with “zero downtime” expectations by customers. What this really means is that you’re going to see much more automation in provisioning and IT monitoring tools to handle the scale and speed with which things can change in the data center given vm migration and the talked-about switching between “clouds” that can be used for high availability.

It’s still predominantly made up of an army of skilled hackers focused on better ways to break systems apart and find new ways to exploit vulnerabilities than “security architects” who are designing secure components, protocols and ultimately secure systems. If you don’t believe me go have a conversation with a  so called application security  consultant about SAML or security issues in Enterprise Message Buses and you’ll almost definitely draw blank stares. Ask application security consultants if they know about the latest HTTP or HTML spec and they’ll likely say yes (and want to demonstrate the latest issues) but if you ask them about the latest WS-x spec you’ll likely draw more blank stares.  When was the last time you saw an attack drawn out as a UML sequence diagram? This is worrying and somewhat sad. I don’t think we are culturing, encouraging and nurturing people with the right skills to make a positive difference. 

On the other hand, it's probably a smart idea to screen screeners when they walk through airport security when they aren't working at that checkpoint at that time. The reason is the same reason you should screen everyone, including pilots who can crash their plane: you're not screening screeners (or pilots), you're screening people wearing screener (or pilot) uniforms and carrying screener (or pilot) IDs. You can either train your screeners to recognize authentic uniforms and IDs, or you can just screen everybody. The latter is just easier.