Question

We have a problem while rebuilding the Sphinx index for one of our MySQL databases. Rotating and rebuilding the index fails repeatedly, and the system continues to use the old index. The database size has doubled in the last months and it contains now tables with more than 10 million rows. During rebuilding of the Sphinx index with

indexer --config our_config.conf --rotate --all

(generated by the UltraSphinx plugin used here), we get the following error in the Sphinx daemon log file searchd.log

WARNING: rotating index 'main': preread failed: failed to open 
~/etc/sphinx//sphinx_index_main.new.spi: No such file or directory; 
using old index
WARNING: rotating index 'main': preread failure reported

The error messages occurred in increasing frequency as the database grew larger. Full indexing takes now about 1 1/2 hours, and the index files together are larger than one GB. We updated the Sphinx Version from 0.9.8.1 to 2.0.4 to solve this problem, but it still exists. The server has about 4 GB memory, and the configuration file begins with

indexer {
  mem_limit = 1024M
}

searchd {
  read_timeout = 5
  max_children = 300
  log = ~/etc/sphinx/log/searchd.log
  port = 3313
  max_matches = 100000
  query_log = ~/etc/sphinx/log/query.log
  seamless_rotate = 1
  pid_file = ~/etc/sphinx/log/searchd.pid
  address = 127.0.0.1
}
..

Has anyone an idea how the problem can be solved? I guess there seems to be some kind of index size limit which has to do with the 2 GB limit, wich is the maximal value for a signed 32 bit integer and maximal value for file sizes on certain file systems. The ext3 file system we use on our Ubuntu/Debian systems allows up to 16 GB for files, but certain libraries may limit the file size in practice to 2 GB (see here and here).

Answer 1

Check the output of indexer itself – it shows its progress during indexing, and might give clue as to the issue.