The current thinking is that enterprise software companies need to open source part of their technology as an effort in low-cost marketing. The theory being that marketing $ that aren't productive for customers (e.g. advertising) are put towards development, which does help customers (and yet still feeds the marketing engine with branding, leads, etc.). This is certainly why most companies do OSS, but it's a tricky process.Having gone through this with Wildfire, we have seen the challenges and have a better idea of what it takes to make an OSS project successful. I think open-sourcing software can reshape the way dollars are applied to projects, but it requires a very focused effort.
IMHO, some elements that are important:**
Embrace professional services: Understand the service aspect of the model, and seek out the biggest users of your application to find opportunities for professional services. When these opportunities cease to be relevant to the majority of the community, find consultants to take on the work instead of core engineers -- this keeps the $ flowing efficiently.
Create something great: What you create really has to blow people away. Just making it free is useless unless it has the potential to pull an avid community of users.
Write a constitution & stick to it: Understand at the beginning of the process how the commercial offerings will differ from the free offerings philosophically. Communicate it and abide by it.
Make it a complete solution: Whatever you open source needs to provide a complete solution in and of itself. It may not have all the bells and whistles, but it needs to get the job done well.
Don't horde features: Arbitrarily pulling common features into a commercial edition will cause havoc with the community.
Have up-sell opportunities: Given #4 above, make sure your commercial applications can add significant value to customers without denying them features from the core project.
Be "open" with communication too: Stay in close contact with the community about the business as well: how you make money, partnerships, members of the team and more. Being tight-lipped about how you plan to make money doesn't foster trust and the community needs to understand how the project is funded.
Get SaaS-y: It's a buzz-acronym, but software as a service has some merit -- specifically, tying performance of the application to the payment stream. In other words, being paid for doing a good job as opposed to licensing your software once and leaving. Whether you are responsible for hosting the applications or not, seek out ways to build accountability for performance with your commercial customers.
Make support more appetizing: Support by itself is interesting to a certain percentage of customers, but finding ways to do proactive support (calling on customers periodically, sending updates that will affect them, helping them with performance, etc.) can expand the # of customers willing to purchase plans.
Build on passion: If your engineers aren't passionate about it, don't do it. It will become abundantly clear to everyone that it's just another form of marketing and there will be no energy. No energy = no community. No community = no marketshare, profit or value.
Disclaimer: We don't do all of these....yet :).
Summary: OSS can change the way companies allocate $ towards more productive activities, but it is by no means a path to success. Without a massive commitment to being a "true" open source initiative (and the passion to make that happen), most of these projects will fall flat and cause more financial harm than good.