(This question is about the best way to temporarily and programatically keep new records of two databases in sync, having very different schemas, and ignoring the old -outdated and no longer required- records.)
I work on a company that provides TV programming information for guides, newspapers and websites.
I have an old system that has several limitations and is being replaced by a new one.
Different clients take the data in different formats (xml, sql, txt, even print-ready PDF) and in different ways (push, pull, partial dumps, simple exports, man assisted exports -like the PDF version-, etc). Some exports are generated once a month, others more than once a day.
The problem is that several clients have to rely on the data from the old system until the new one is fully developed and loaded, and the staff mantaining the data can’t have both databases in sync because it would require a lot of additional work, but switching systems overnight seems impossible given the size of the project.
We don’t want to make a full inport of the data from the old database to the new one, because most of it is already not necessary, and have a lot of garbage (ie, duplicate records with different detail level, old airings info that we only need as an archive).
We want the new records inserted in both databases, and the old ones being edited, copied to the new database as well.
We are about to start the development of the new system using Symfony with Doctrine, and I’ve decided we could design a set of ORM “proxy” classes that should have the same interface than a simple Doctrine ORM set of classes, but keep the sync going between two other set of classes (the ones that interface with the new system and the ones that interface with the old one). Eventually the old DB should be discarded along with the proxy classes, and the Doctrine ORM classes that connect directly to the new DB should take that place, as if the old system would never existed.
It´s a long shot, and I’m not completely confident with the approach.
Do anyone have experience with this kind of project?
Do you know of any common pitfall in this approach, or some other solution that could fit this situation?
I’m not sure if there is a method to accomplish this, I can only recommend that you should fully test the new db before you start migrating data, once your sure that it works as it should I suggest you to write some migrating applicacion that would query the requiered data to be exported from the old db, you don’t have to (but you should) debug the data from the old db, if the new db works, the duplicated data should be ignored by the restraints.
I used to work for a company that didn’t care for this kind of problems and at the end it was a complete mess, patch after patch, yelling and unecesary stress. I know… sad.