I am managing a website which has about 140.000 documents. This system had been running without a problem for more than two years using the same setup. We had to migrate our servers to a new infrastructure and since then there are some anomalies.
Every document has a status in the database. Status code (int) 5 means that the document is deleted/removed.
Every hour or so, the status code of some 10 documents changes to 5. We do not lose the actual document but it becomes unavailable to the end user. I am trying to figure out what’s causing the issue, but couldn’t find the source of it.
We have two queries which can change the status of the document. I have temporarily disabled one of them, and I have changed the status code of the deleted document to 6 in the codebase.
However, we still have the same problem and the status code changes to 5, not 6.
I have enabled the query log, and spent hours but there are no update queries to those deleted documents, except the view_count increase which is a very short and simple query. The document gets requested by someone at the same time it gets deleted. However, all the queries are select queries (except view_count) and there is no difference between other documents. And we have hundreds of page views every second. Only one of them gets deleted, randomly.
So, if I could see in the query log, I could try to find out that particular query in the code base. However, it even doesn’t show anything. So, I am clueless right now.
Do you have any suggestions? I have to solve this issue, so I can try anything.
New system is;
Debian 6.0.5
MySQL 5.5.24 – Percona Build
PHP-NGINX-SOLR box
Profile
UPDATEqueries with Jet Profiler or Percona Toolkit, analyse them and you shall find the answer.General Query Log may also help you finding that particular update query.