From my experience, commitments to backwards/forwards compatibility are the gilded cage of the software engineering industry. I have particularly observed this to be the case for document file formats and programming languages/APIs. Customers and partners hate it when their existing data or code breaks; however, if you never break compatibility, you can seriously constrain your ability to innovate in the long run.
Are there solutions to this problem, other than gradual deprecation of old features? It seems like virtualization, as in Windows 7’s XP Mode, is one exciting possibility. Are there others?
Also, for those of us who want to design new systems that are as future-proof as possible, what lessons can we learn from past mistakes made in the industry?
Innovate by extending, not by rewriting the public APIs. Have consistent generic public interfaces to back-end functionality. You can rewrite private modules at any time as long as you supply the public API modules with the results that they expect.
Do your improvements in the back-end and leave the API consistent as much as possible. Create new modules and document them clearly when extending the public parts of your API, deprecation of the old ways will come naturally if you supply new and better ways to do things that come as additions to the old ways.
For document formats, always include version numbers and make sure you have ways to support all the existent versions. As with the APIs, add the new functionality by extending, not by rewriting.
When you want to bring fundamental changes to the overall architecture of your software, have the new version include the old one as a module – it will lead to a greater size but better support for older data and programs.