Hello MySQL community!
I am about at the point of pulling my beard off trying to troubleshoot this issue with my production webserver. The machine is a CentOS 5.3 machine with Cpanel 11.40.1 Build 8 on it, running MySQL 5.5.34 CLI (which is the latest that you can get with cpanel). 2 GB ram.
I have had logging turned on and watching but can't seem to find the same query breaking the database twice in a row. I have tried repair table, myiasmchk, mysqlchk (the databases are primarily vbulletin forums, a few zend carts, a few magento carts, some wordpress - so it's not a strictly InnoDB environment) and virtually anything I could find on the subject of fixing a corrupt database. The last thing I did was grab a list of databases, mysql dump them with drop table and reimport them one by one (97 in all) - it crashed 12 times during this with the exact same error every time:
140105 20:30:38 InnoDB: Assertion failure in thread 2172095376 in file trx0purge.c line 840
InnoDB: Failing assertion: purge_sys->purge_trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:30:38 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=5
max_threads=100
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 283900 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0x8441f03]
/usr/sbin/mysqld(handle_fatal_signal+0x42b)[0x82e616b]
[0xf52420]
[0xf52402]
/lib/libc.so.6(gsignal+0x50)[0xa80e30]
/lib/libc.so.6(abort+0x101)[0xa82741]
/usr/sbin/mysqld[0x84b3a92]
/usr/sbin/mysqld[0x84b5c28]
/usr/sbin/mysqld[0x85a1983]
/usr/sbin/mysqld[0x8595fc2]
/usr/sbin/mysqld[0x84b30b1]
/usr/sbin/mysqld[0x84a4f8b]
/lib/libpthread.so.0[0x595912]
/lib/libc.so.6(clone+0x5e)[0xb2d7ce]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140105 20:30:38 mysqld_safe Number of processes running now: 0
140105 20:30:38 mysqld_safe mysqld restarted
140105 20:30:38 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
140105 20:30:38 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
140105 20:30:38 [Note] Plugin 'FEDERATED' is disabled.
140105 20:30:38 InnoDB: The InnoDB memory heap is disabled
140105 20:30:38 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
140105 20:30:38 InnoDB: Compressed tables use zlib 1.2.3
140105 20:30:38 InnoDB: Using Linux native AIO
140105 20:30:38 InnoDB: Initializing buffer pool, size = 500.0M
140105 20:30:38 InnoDB: Completed initialization of buffer pool
140105 20:30:38 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 12492634423
140105 20:30:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 12492635608
140105 20:30:38 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
140105 20:30:39 InnoDB: Waiting for the background threads to start
140105 20:30:40 InnoDB: 5.5.34 started; log sequence number 12492635608
140105 20:30:40 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
140105 20:30:40 [Note] - '0.0.0.0' resolves to '0.0.0.0';
140105 20:30:40 [Note] Server socket created on IP: '0.0.0.0'.
140105 20:30:40 [Note] Event Scheduler: Loaded 0 events
140105 20:30:40 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.34-cll' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
And for those who would like to see it, here is the my.cnf file:
root@kratos [~]# more /etc/my.cnf
[mysqld]
innodb_buffer_pool_instances=4
join_buffer_size=3M
table_open_cache=2000
table_definition_cache=1000
read_rnd_buffer_size=2M
log-error="/var/lib/mysql/error.log"
interactive_timeout=600
thread_cache_size=16K
tmp_table_size=24M
max_heap_table_size=24M
innodb_buffer_pool_size=500M
key_buffer_size=100M
key_buffer=64M
max_connections=100
log_warnings=2
log-slow-queries="/var/lib/mysql/slow.log"
long_query_time=5
query_cache_size=128M
query_cache_limit=148M
table_cache=1K
table_definition_cache=4k
wait_timeout=600
#log="/disk2/mysqllog/mysql.log"
max_heap_table_size=268435456
open_files_limit=3K
default-storage-engine=MyISAM
innodb_file_per_table=1
I would appreciate any help or guidance anybody can give me!
Thanks in advance,
Chris
I am about at the point of pulling my beard off trying to troubleshoot this issue with my production webserver. The machine is a CentOS 5.3 machine with Cpanel 11.40.1 Build 8 on it, running MySQL 5.5.34 CLI (which is the latest that you can get with cpanel). 2 GB ram.
I have had logging turned on and watching but can't seem to find the same query breaking the database twice in a row. I have tried repair table, myiasmchk, mysqlchk (the databases are primarily vbulletin forums, a few zend carts, a few magento carts, some wordpress - so it's not a strictly InnoDB environment) and virtually anything I could find on the subject of fixing a corrupt database. The last thing I did was grab a list of databases, mysql dump them with drop table and reimport them one by one (97 in all) - it crashed 12 times during this with the exact same error every time:
140105 20:30:38 InnoDB: Assertion failure in thread 2172095376 in file trx0purge.c line 840
InnoDB: Failing assertion: purge_sys->purge_trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:30:38 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=5
max_threads=100
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 283900 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0x8441f03]
/usr/sbin/mysqld(handle_fatal_signal+0x42b)[0x82e616b]
[0xf52420]
[0xf52402]
/lib/libc.so.6(gsignal+0x50)[0xa80e30]
/lib/libc.so.6(abort+0x101)[0xa82741]
/usr/sbin/mysqld[0x84b3a92]
/usr/sbin/mysqld[0x84b5c28]
/usr/sbin/mysqld[0x85a1983]
/usr/sbin/mysqld[0x8595fc2]
/usr/sbin/mysqld[0x84b30b1]
/usr/sbin/mysqld[0x84a4f8b]
/lib/libpthread.so.0[0x595912]
/lib/libc.so.6(clone+0x5e)[0xb2d7ce]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140105 20:30:38 mysqld_safe Number of processes running now: 0
140105 20:30:38 mysqld_safe mysqld restarted
140105 20:30:38 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
140105 20:30:38 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
140105 20:30:38 [Note] Plugin 'FEDERATED' is disabled.
140105 20:30:38 InnoDB: The InnoDB memory heap is disabled
140105 20:30:38 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
140105 20:30:38 InnoDB: Compressed tables use zlib 1.2.3
140105 20:30:38 InnoDB: Using Linux native AIO
140105 20:30:38 InnoDB: Initializing buffer pool, size = 500.0M
140105 20:30:38 InnoDB: Completed initialization of buffer pool
140105 20:30:38 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 12492634423
140105 20:30:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 12492635608
140105 20:30:38 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
140105 20:30:39 InnoDB: Waiting for the background threads to start
140105 20:30:40 InnoDB: 5.5.34 started; log sequence number 12492635608
140105 20:30:40 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
140105 20:30:40 [Note] - '0.0.0.0' resolves to '0.0.0.0';
140105 20:30:40 [Note] Server socket created on IP: '0.0.0.0'.
140105 20:30:40 [Note] Event Scheduler: Loaded 0 events
140105 20:30:40 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.34-cll' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
And for those who would like to see it, here is the my.cnf file:
root@kratos [~]# more /etc/my.cnf
[mysqld]
innodb_buffer_pool_instances=4
join_buffer_size=3M
table_open_cache=2000
table_definition_cache=1000
read_rnd_buffer_size=2M
log-error="/var/lib/mysql/error.log"
interactive_timeout=600
thread_cache_size=16K
tmp_table_size=24M
max_heap_table_size=24M
innodb_buffer_pool_size=500M
key_buffer_size=100M
key_buffer=64M
max_connections=100
log_warnings=2
log-slow-queries="/var/lib/mysql/slow.log"
long_query_time=5
query_cache_size=128M
query_cache_limit=148M
table_cache=1K
table_definition_cache=4k
wait_timeout=600
#log="/disk2/mysqllog/mysql.log"
max_heap_table_size=268435456
open_files_limit=3K
default-storage-engine=MyISAM
innodb_file_per_table=1
I would appreciate any help or guidance anybody can give me!
Thanks in advance,
Chris