Question

We have different database IP addresses for our development and production environments. Our development environment is running locally on our developer machines and pointed to a single development database server on our local network. Our production environment uses a database hosted at RackSpace and is hosted on their local network. Somehow, it appears that our development IP address has been cached on production. Here’s what I have done so far:

• Verified that the IP address in app/etc/local.xml on production is correct.
• Deleted the contents of var/cache/* and var/full_page_cache/*
• Restarted our memcached servers to clear any strange caches there
• grepped our entire codebase for the dev IP address
• Dumped the mysql database and grepped the dump for the dev IP (we were desperate)
• Deleted the contents of /tmp
• Disabled custom modules

This has been working for weeks without a problem. It was when I disabled the config cache that problem started. I know what you’re thinking, that it just now finally picked up a config change that someone made since the last time the cache was cleared. That makes sense. What doesn’t make sense is that I’ve cleared every cache mentioned above, enabled the config cache using MageTool and everything works like a charm.

Answer 1

As it turns out, fixing this whole thing was a two step process.

Because our production and development environments require different IPs app/etc/local.xml is untracked and instead we track app/etc/local-example.xml so that all of our developers can quickly and easily copy it over to app/etc/local.xml and be up and running. This has become a bit of a company standard and we use it across all of our other projects. Thankfully, one of my coworkers discovered that Magento loads all the xml files in app/etc/.

So, no, our development IP wasn’t magically cached in some crazy obscure location, we were just inadvertently loading it. After renaming that file to app/etc/local.xml.example it stopped referencing our development IP. Yay!

Now, this isn’t directly related to the question, but because the solution introduced a new bug I want to mention it. Once we renamed the xml file and cleared all of the caches we started seeing a new error.

PHP Fatal error: Call to a member function setQueryHook() on a non-object in app/code/core/Mage/Core/Model/Resource/Setup.php on line 347

In our example file we were defining our database resource inside a single <default_setup /> node. For our production environment we actually have a triple-m setup with separate IPs for read and write queries so for production instead of a single <default_setup /> node we had <default_read /> and <default_write /> nodes. I have never been able to find documentation on exactly what is allowed and required in resources, but the read / write split was setup per the instructions in another StackOverflow post on the topic and, up until today, worked great.

On a hunch, I renamed the <default_write /> node to <default_setup /> and everything magically began working again. I’m not yet sure if the reads and write are correctly splitting, but I’ll update this answer once I confirm everything is working.