Question

Good day, friends. I’m running Fedora 13 on a 32b machine.

I have a huge table of 1.5B ip addresses (which will be split up when I have the resources, don’t worry ;)). When I run this query:

SELECT ip FROM ips ORDER BY RAND() LIMIT 500000;

Sometimes the table crashes (I get the error “MySQL Table is Marked as Crashed and Should be Repaired”), sometimes it doesn’t. My question is; what sorts of things cause MyISAM or InnoDB tables to crash? Does memory have an affect? Because my system monitor shows it only uses about 30%.

Here is my.cnf (the default, installing from meta-package):

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Also; I hear PostGreSQL is ‘more robust’ and doesn’t crash like MySQL does in these situations. Is this true or is it a wives tale?

Answer 1

Sorting 1500M of IP addresses is not the best idea.

If you want 500K random IP addresses, use this approach:

• Selecting random rows

It won’t be too efficient as well, though, since it needs two passes over the table.

You may put a rough estimate of the COUNT(*) instead of the subquery. This will speed the query up, at cost of a slight opportunity of missing some addresses or getting fewer results than needed.