On April 23rd, 2015 I participated in CMSWire's SocBizChat on Twitter. Below is a summary of my responses to their questions.

What does the number of collaboration tools on the market say about the state of collaboration in organizations?

During the chat, I responded with: We don't need more tools, we need our tools to do more. We're already suffering from #InputOverload”. When asked what the “more” is, I responded "More connectivity. More integration. More intelligence. More workflow. More automation."

Still, this does not really answer the original question. The reason there are so many tools comes from a) organizations are not successfully collaborating with the tools they have today (i.e. there is a need) and b) no one (or small group of) vendors have yet to solve these needs. Compare that to the email, where just 3 vendors (Microsoft, Google and IBM) dominate the market.

The definition of the collaboration market is complex.  A few of the main categories are: 

• Suites from the enterprise software vendors that bring together multiple tools such as email, calendar, profiles, blogs, wikis, communities, social networks, chat, etc. Ex: Microsoft Office365/SharePoint, IBM Connections, Google Apps.
• Collaboration solutions built into business applications. ex: SAP Jam, Salesforce Chatter, Oracle Social Network, Infor Mingle
• Community-centric software. Ex: Jive, Bloomfire, Igloo, Huddle, HighQ, Jostle, ThoughtFarmer, Traction
• A growing popularity of persistent chat vendors such as: Atlasian HipChat, Cisco Spark, Convo, FlowDock, Glip, Hall, Intellinote, Slack, Unify Circuit 
• Project management-centric solutions such as Asana, Clarizen, LiquidPlanner, Mavenlink, Redbooth, Smartsheet, Trello, Workfront, Wrike and Wunderlist
• Mobile Messaging Apps like CoTap, Jive Chime, GroupMe, Talko, Mez, Lua
• Niche solutions for things like File-Sharing, Ideation, web-conferencing each with a dozen vendors
• A dozen others!

So to get back to the question, with no dominant vendors in this space, there is opportunity for many companies to go after customers. Heck, Facebook is even working on a business version of their platform, so clearly the opportunity still exists.

If collaboration occurs naturally, why do so many businesses have problems with it?

Collaboration fails when it is forced and does not fit into the way people work. If a tool meets a need, it will be used.
If a tool is implemented just because it's new and shinny, and not because it solves a need, then it will most likely fail.
Implementations struggle if you focus on the What? (tool), Who? (culture) but not the Why? (purpose)

For more on this topic, see my report on Purposeful Collaboration.

How can companies reach a balance between employee driven tool selection and enterprise security and governance concerns?

Security and governance need to be like special effects in movies... do their job so well you don't notice them.
What did I mean by this? Essentially, security should not be the responsibility of the end-user.  The platforms organizations use need to have built in features for identity, authentication, encryption, compliance, governance, etc.

Can companies succeed with small-scale collaboration efforts or is a larger social business strategy needed to scale

Small-scale and large scale collaboration do not compete, they serve different purposes.
Small scale collaboration could be entered around a specific department, project or geography. Not everyone needs to be involved in everything!
Look at the recent success of tools like Slack, which focus on small scale deployment, not company wide adoption.
Very few use-cases call for company-wide collaboration. Company-wide communication is great, but work gets done in teams.
Software vendors what to sell licenses company wide. Companies want to improve the way employees get their work done. ;-)
Silos are excellent for helping organize info, but they should not be barriers to access. "Open Silos = fences"

My main point here was, collaboration is great. I want to see people sharing information and expertise. But the “kumbaya” around sharing everything and working completely open and transparent is not realistic. Not everyone needs to have access to everything. That is not only insecure, but it also leads to information overload, which can more of a productivity burden than helper.  I’m not saying information should be locked away. There is a difference between enabling information to be discovered (searched for) vs. trying to push everything into the open all the time.

What will the collaboration tool landscape look like in five years?

The collaboration landscape will be just as cluttered in 5 years as it is today, it will just be a whole new set of tools.
The cycle has repeated for decades. Email wars. IM wars. Blog wars. Wiki wars. ESN wars. Chat room wars. The same will happen again in 5 years with a whole new set of tools.
Clearly there will be continued growth in mobile and wearables, but wearables will be very different. They won't be discrete new objects, they will be more embedded into existing items.
Most importantly, on the software side, products will have far more "intelligence" and help automate our decisions and our workflows.