I have a problem with ubuntu killing mySQL server due to out of memory when running Magento. I’ve ran the mySQLtuner script (which warned that possible memory allocation for mySQL was higher than system memory) and adjusted the mySQL /etc/mysql/my.cnf file accordingly.
According to the script the maximum allowed memory usage for mySQL is 218,9 MB which should not cause a problem. Free memory when the server is just started and everything is working is 277 / 590 MB. After clicking around for a wile in Magento frontend the mySQL server is killed, here is a dump from /var/log/syslog:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037361] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037367] mysqld cpuset=/ mems_allowed=0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037370] Pid: 948, comm: mysqld Not tainted 3.2.0-29-virtual #46-Ubuntu
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037372] Call Trace:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037382] [<ffffffff810bd5fd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037386] [<ffffffff81117ce1>] dump_header+0x91/0xe0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037388] [<ffffffff81118065>] oom_kill_process+0x85/0xb0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037391] [<ffffffff8111840a>] out_of_memory+0xfa/0x220
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037394] [<ffffffff8111de3a>] __alloc_pages_nodemask+0x7ea/0x800
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037398] [<ffffffff8165296e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037403] [<ffffffff81154b73>] alloc_pages_current+0xa3/0x110
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037406] [<ffffffff81114a4f>] __page_cache_alloc+0x8f/0xa0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037409] [<ffffffff81114d4e>] ? find_get_page+0x1e/0x90
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037412] [<ffffffff81116bd2>] filemap_fault+0x212/0x3c0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037415] [<ffffffff81136e62>] __do_fault+0x72/0x550
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037419] [<ffffffff8108ca60>] ? lock_hrtimer_base.isra.24+0x30/0x60
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037422] [<ffffffff8113a70a>] handle_pte_fault+0xfa/0x200
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037426] [<ffffffff8100648e>] ? xen_pmd_val+0xe/0x10
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037428] [<ffffffff810052e9>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037431] [<ffffffff8113abc8>] handle_mm_fault+0x1f8/0x350
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037434] [<ffffffff8165625b>] do_page_fault+0x14b/0x520
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037437] [<ffffffff8108d2a4>] ? hrtimer_start_range_ns+0x14/0x20
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037440] [<ffffffff8103cda5>] ? pvclock_clocksource_read+0x55/0xf0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037444] [<ffffffff8100a640>] ? xen_clocksource_read+0x20/0x30
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037447] [<ffffffff8100a729>] ? xen_clocksource_get_cycles+0x9/0x10
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037456] [<ffffffff81093177>] ? getnstimeofday+0x57/0xe0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037459] [<ffffffff81652eb5>] page_fault+0x25/0x30
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037460] Mem-Info:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037462] Node 0 DMA per-cpu:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037464] CPU 0: hi: 0, btch: 1 usd: 0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037466] Node 0 DMA32 per-cpu:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037468] CPU 0: hi: 186, btch: 31 usd: 123
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037471] active_anon:139853 inactive_anon:31 isolated_anon:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037472] active_file:38 inactive_file:169 isolated_file:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037473] unevictable:0 dirty:0 writeback:0 unstable:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037473] free:1375 slab_reclaimable:1934 slab_unreclaimable:2073
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037474] mapped:55 shmem:64 pagetables:2195 bounce:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037476] Node 0 DMA free:2464kB min:72kB low:88kB high:108kB active_anon:12232kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14524kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037484] lowmem_reserve[]: 0 597 597 597
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037487] Node 0 DMA32 free:3036kB min:3088kB low:3860kB high:4632kB active_anon:547180kB inactive_anon:124kB active_file:152kB inactive_file:664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:611856kB mlocked:0kB dirty:0kB writeback:0kB mapped:220kB shmem:256kB slab_reclaimable:7704kB slab_unreclaimable:8280kB kernel_stack:792kB pagetables:8756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1943 all_unreclaimable? yes
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037495] lowmem_reserve[]: 0 0 0 0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037498] Node 0 DMA: 8*4kB 6*8kB 5*16kB 2*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2464kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037506] Node 0 DMA32: 247*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3036kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037513] 279 total pagecache pages
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037514] 0 pages in swap cache
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037516] Swap cache stats: add 0, delete 0, find 0/0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037517] Free swap = 0kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037518] Total swap = 0kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041176] 159472 pages RAM
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041180] 8376 pages reserved
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041181] 16609 pages shared
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041182] 143523 pages non-shared
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041184] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041194] [ 246] 0 246 4306 49 0 0 0 upstart-udev-br
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041197] [ 250] 0 250 5365 116 0 -17 -1000 udevd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041200] [ 300] 0 300 5364 98 0 -17 -1000 udevd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041203] [ 301] 0 301 5364 97 0 -17 -1000 udevd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041206] [ 393] 0 393 3795 47 0 0 0 upstart-socket-
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041209] [ 444] 0 444 1814 123 0 0 0 dhclient3
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041212] [ 617] 0 617 12487 153 0 -17 -1000 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041215] [ 629] 101 629 63428 103 0 0 0 rsyslogd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041218] [ 636] 102 636 5952 74 0 0 0 dbus-daemon
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041221] [ 692] 0 692 3624 39 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041224] [ 699] 0 699 3624 43 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041227] [ 710] 0 710 3624 41 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041230] [ 711] 0 711 3624 41 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041236] [ 718] 0 718 3624 43 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041239] [ 725] 0 725 1080 36 0 0 0 acpid
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041242] [ 727] 0 727 4776 59 0 0 0 cron
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041245] [ 728] 0 728 4225 40 0 0 0 atd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041248] [ 753] 106 753 203063 19815 0 0 0 mysqld
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041251] [ 808] 103 808 46895 299 0 0 0 whoopsie
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041254] [ 857] 0 857 6274 78 0 0 0 master
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041257] [ 867] 107 867 6831 84 0 0 0 qmgr
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041260] [ 910] 0 910 76016 1435 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041263] [ 953] 33 953 84720 10145 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041266] [ 954] 33 954 90925 15531 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041272] [ 955] 33 955 76062 1474 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041275] [ 957] 33 957 81127 6426 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041278] [ 1065] 0 1065 22444 5138 0 0 0 /usr/share/webm
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041281] [ 1069] 0 1069 3624 41 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041284] [ 1243] 33 1243 90163 15653 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041287] [ 1246] 33 1246 79452 4952 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041290] [ 1247] 33 1247 93679 18462 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041292] [ 1248] 33 1248 93574 19110 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041295] [ 1249] 33 1249 92189 16935 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041298] [ 5116] 107 5116 6790 68 0 0 0 pickup
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041301] [ 5169] 0 5169 18338 200 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041304] [ 5287] 1000 5287 18338 200 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041307] [ 5288] 1000 5288 6248 1457 0 0 0 bash
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041310] [ 5562] 0 5562 22444 5098 0 0 0 miniserv.pl
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041313] [ 5597] 0 5597 18338 201 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041315] [ 5687] 1000 5687 18338 201 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041318] [ 5688] 1000 5688 6235 1446 0 0 0 bash
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041321] [ 5784] 1000 5784 10474 92 0 0 0 sudo
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041324] [ 5785] 0 5785 3337 171 0 0 0 nano
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041327] [ 5786] 33 5786 84584 10087 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041330] Out of memory: Kill process 753 (mysqld) score 131 or sacrifice child
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041396] Killed process 753 (mysqld) total-vm:812252kB, anon-rss:79260kB, file-rss:0kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.217049] init: mysql main process (753) killed by KILL signal
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.218251] init: mysql main process ended, respawning
Dec 28 11:11:53 ip-10-228-98-15 kernel: [37225.185699] type=1400 audit(1356693113.390:11): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=5794 comm="apparmor_parser"
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.152344] init: mysql main process (5798) terminated with status 1
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.152382] init: mysql main process ended, respawning
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.768092] init: mysql post-start process (5799) terminated with status 1
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.784475] type=1400 audit(1356693114.990:12): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=5822 comm="apparmor_parser"
Dec 28 11:11:55 ip-10-228-98-15 kernel: [37226.844954] init: mysql main process (5826) terminated with status 1
Dec 28 11:11:55 ip-10-228-98-15 kernel: [37226.844991] init: mysql respawning too fast, stopped
So basically mySQL is busted for hogging to much memory and then sentenced to death. The question is why? I thought my settings in my.cnf had limited the memory mySQL could use?
Another interesting question is why it won’t restart again? It has nothing to do with apparmor, I removed it but same result but without the apparmor reference in the syslog.
my.cnf:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 1
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 10
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
I do realize that these memory settings are way to low for any performance in Magento but I’ve tried all kinds of value here to the same result. Crasch!
The server is a EC2 micro instance running Ubuntu 12.04 x64 and i don’t by any means expect blazing speeds (no production use) but at least I should be able to configure it so it doesn’t crasch all the time. I’ve another EC2 micro instance running on 32bit Ubuntu and the mySQL server has never ever crashed on that one. Could it be related to to 32 vs 64 bit Ubuntu?
As you can see there are many questions but limited knowledge, grateful for any pointers. I’ll keep you posted.
In the end I went with the suggestion of Anton S above and simply accepted that the instance did not have enough memory. Launched it as an ec2 small instance instead and now it works like a charm.