Hi there
We had a problem today with innodb. Can anyone cast any light here? From error log:
--Thread 1171712320 has waited at btr0cur.c line 424 for 241.00 seconds the semaphore:
X-lock on RW-latch at 0x2aadc68dbd08 created in file buf0buf.c line 497
a writer (thread id 1171712320) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file ./../include/btr0btr.ic line 28
Last time write locked in file btr0cur.c line 424
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
=====================================
120822 10:49:50 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 35 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 316790, signal count 316625
--Thread 1171712320 has waited at btr0cur.c line 424 for 245.00 seconds the semaphore:
X-lock on RW-latch at 0x2aadc68dbd08 created in file buf0buf.c line 497
a writer (thread id 1171712320) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file ./../include/btr0btr.ic line 28
Last time write locked in file btr0cur.c line 424
Mutex spin waits 0, rounds 30206561, OS waits 186334
RW-shared spins 320330, OS waits 64394; RW-excl spins 1231046, OS waits 49177
------------
TRANSACTIONS
------------
Trx id counter 1 3435402732
Purge done for trx's n:o < 1 3435401929 undo n:o < 0 0
History list length 375
Total number of lock structs in row lock hash table 1
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 1 3435402731, ACTIVE 249 sec, process no 27961, OS thread id 1095047488 inserting, thread declared inside InnoDB 478
mysql tables in use 1, locked 1
2 lock struct(s), heap size 368, undo log entries 23
MySQL thread id 3083, query id 30880239 update
INSERT INTO db_name.table_name
(col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12, col13, col14)
VALUES
(5122, 5, 0, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX', 'XXX', 0, '2012-08-22 10:45:01'),(5122, 5, 32, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX', 'XXX', 0, '2012-08-22 10:45:01'),(5122, 5, 29, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX', 'XXX', 0, '2012-08-22 10:45:01'),(5122, 5, 27, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (write thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
292745 OS file reads, 17510133 OS file writes, 16812268 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 28, free list len 23952, seg size 23981,
271595 inserts, 336961 merged recs, 93788 merges
Hash table size 25499819, used cells 12268080, node heap has 24669 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 785 826604045
Log flushed up to 785 826604045
Last checkpoint at 785 528004551
0 pending log writes, 0 pending chkp writes
16732318 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 14283919898; in additional pool allocated 20971264
Buffer pool size 786432
Free buffers 357080
Database pages 404683
Modified db pages 54370
Pending reads 1
Pending writes: LRU 0, flush list 127, single page 0
Pages read 340104, created 71642, written 1487752
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
--------------
ROW OPERATIONS
--------------
1 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread process no. 27961, id 1171712320, state: doing insert buffer merge
Number of rows inserted 10388523, updated 8363305, deleted 11281397, read 136018350
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
This output repeats a number of times until:
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
120822 11:01:38InnoDB: Assertion failure in thread 1161222464 in file srv0srv.c line 2106
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Thread 1095047488 stopped in file ./../include/sync0sync.ic line 111
120822 11:01:38 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
After which the the database (innodb) goes into recovery mode.
Any thoughts?
We had a problem today with innodb. Can anyone cast any light here? From error log:
--Thread 1171712320 has waited at btr0cur.c line 424 for 241.00 seconds the semaphore:
X-lock on RW-latch at 0x2aadc68dbd08 created in file buf0buf.c line 497
a writer (thread id 1171712320) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file ./../include/btr0btr.ic line 28
Last time write locked in file btr0cur.c line 424
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
=====================================
120822 10:49:50 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 35 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 316790, signal count 316625
--Thread 1171712320 has waited at btr0cur.c line 424 for 245.00 seconds the semaphore:
X-lock on RW-latch at 0x2aadc68dbd08 created in file buf0buf.c line 497
a writer (thread id 1171712320) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file ./../include/btr0btr.ic line 28
Last time write locked in file btr0cur.c line 424
Mutex spin waits 0, rounds 30206561, OS waits 186334
RW-shared spins 320330, OS waits 64394; RW-excl spins 1231046, OS waits 49177
------------
TRANSACTIONS
------------
Trx id counter 1 3435402732
Purge done for trx's n:o < 1 3435401929 undo n:o < 0 0
History list length 375
Total number of lock structs in row lock hash table 1
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 1 3435402731, ACTIVE 249 sec, process no 27961, OS thread id 1095047488 inserting, thread declared inside InnoDB 478
mysql tables in use 1, locked 1
2 lock struct(s), heap size 368, undo log entries 23
MySQL thread id 3083, query id 30880239 update
INSERT INTO db_name.table_name
(col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12, col13, col14)
VALUES
(5122, 5, 0, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX', 'XXX', 0, '2012-08-22 10:45:01'),(5122, 5, 32, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX', 'XXX', 0, '2012-08-22 10:45:01'),(5122, 5, 29, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX', 'XXX', 0, '2012-08-22 10:45:01'),(5122, 5, 27, 1, 0, 6, 1, 1, 6, 'XXX', 'XXX
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (write thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
292745 OS file reads, 17510133 OS file writes, 16812268 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 28, free list len 23952, seg size 23981,
271595 inserts, 336961 merged recs, 93788 merges
Hash table size 25499819, used cells 12268080, node heap has 24669 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 785 826604045
Log flushed up to 785 826604045
Last checkpoint at 785 528004551
0 pending log writes, 0 pending chkp writes
16732318 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 14283919898; in additional pool allocated 20971264
Buffer pool size 786432
Free buffers 357080
Database pages 404683
Modified db pages 54370
Pending reads 1
Pending writes: LRU 0, flush list 127, single page 0
Pages read 340104, created 71642, written 1487752
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
--------------
ROW OPERATIONS
--------------
1 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread process no. 27961, id 1171712320, state: doing insert buffer merge
Number of rows inserted 10388523, updated 8363305, deleted 11281397, read 136018350
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
This output repeats a number of times until:
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
120822 11:01:38InnoDB: Assertion failure in thread 1161222464 in file srv0srv.c line 2106
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Thread 1095047488 stopped in file ./../include/sync0sync.ic line 111
120822 11:01:38 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
After which the the database (innodb) goes into recovery mode.
Any thoughts?