Hi,
MySQL 5.1.39 on Linux, InnoDB Engine
I was dropping a 500+ GB database and got an error which crashed MySQL. Almost all of the data was in one 2+ million row table. MySQL recovered, but the 2 million row table in the database is now corrupted and I can not drop the table or database.
How can I clean this up and drop the table and database?
before the database drop:
root@mysql-production-pro.jcvi.org clcdb_2> select count(*) from lazy_children ;
+----------+
| count(*) |
+----------+
| 2366753 |
+----------+
1 row in set (14 min 21.48 sec)
Command that failed:
root@mysql-production.jcvi.org (none)> drop database clcdb_2 ;
ERROR 2013 (HY000): Lost connection to MySQL server during query
root@mysql-production.jcvi.org (none)> show processlist ;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql-production.jcvi.org' (111)
ERROR:
Can't connect to the server
Fortunately, MySQL came back up after the crash, but the table/database is now corrupted. In errorlog when I try and drop the table now:
110811 11:20:56 [ERROR] Cannot find or open table clcdb_2/lazy_children from
the internal data dictionary of InnoDB though the .frm file for the
table exists. Maybe you have deleted and recreated InnoDB data
files but have forgotten to delete the corresponding .frm files
of InnoDB tables, or you have moved .frm files to another database?
or, the table contains indexes that this version of the engine
doesn't support.
See
http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting.html
how you can resolve the problem.
here is what is currently in the database directory on the server:
12 -rw-rw---- 1 mysql mysql 8630 Nov 24 2010 lazy_children.frm
12 -rw-rw---- 1 mysql mysql 8694 Nov 25 2010 #sql-b6d_3b908cb.frm
2052 -rw-rw---- 1 mysql mysql 2097152 Nov 25 2010 #sql-b6d_3b908cb.ibd
Errors from errorlog when mysql crashed during the drop database:
110811 11:03:52 InnoDB: Assertion failure in thread 1147169088 in file srv/srv0srv.c line 2137
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to
http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB:
http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110811 11:03:52 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=67108864
read_buffer_size=4194304
max_used_connections=251
max_threads=650
threads_connected=13
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 13384157 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/db/mysql/base/bin/mysqld(my_print_stacktrace+0x20) [0xa0a7a8]
/db/mysql/base/bin/mysqld(handle_segfault+0x368) [0x60354c]
/lib64/libpthread.so.0 [0x3b40c0de60]
/db/mysql/base/bin/mysqld(srv_error_monitor_thread+0x2f3) [0x938c3f]
/lib64/libpthread.so.0 [0x3b40c062e7]
/lib64/libc.so.6(clone+0x6d) [0x3b400ce3bd]
The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
110811 11:04:18 mysqld_safe Number of processes running now: 0
110811 11:04:18 mysqld_safe mysqld restarted
110811 11:04:18 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Log scan progressed past the checkpoint lsn 6017 2626484087
110811 11:04:21 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 8437 of name './misc_apis/#sql-5d87_133c21.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 8437 of name './misc_apis/@0023sql@002d5d87_133c21.ibd' already exists in the tablespace
InnoDB: memory cache!
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 6017 2631726592
InnoDB: Doing recovery: scanned up to log sequence number 6017 2636969472
InnoDB: Doing recovery: scanned up to log sequence number 6017 2642212352
InnoDB: Doing recovery: scanned up to log sequence number 6017 2647455232
InnoDB: Doing recovery: scanned up to log sequence number 6017 2652698112
InnoDB: Doing recovery: scanned up to log sequence number 6017 2657940992
InnoDB: Doing recovery: scanned up to log sequence number 6017 2663183872
InnoDB: Doing recovery: scanned up to log sequence number 6017 2668426752
InnoDB: Doing recovery: scanned up to log sequence number 6017 2673669632
InnoDB: Doing recovery: scanned up to log sequence number 6017 2678912512
InnoDB: Doing recovery: scanned up to log sequence number 6017 2684155392
InnoDB: Doing recovery: scanned up to log sequence number 6017 2689398272
InnoDB: Doing recovery: scanned up to log sequence number 6017 2694641152
InnoDB: Doing recovery: scanned up to log sequence number 6017 2699884032
InnoDB: Doing recovery: scanned up to log sequence number 6017 2705126912
InnoDB: Doing recovery: scanned up to log sequence number 6017 2710369792
InnoDB: Doing recovery: scanned up to log sequence number 6017 2715612672
InnoDB: Doing recovery: scanned up to log sequence number 6017 2720855552
InnoDB: Doing recovery: scanned up to log sequence number 6017 2726098432
InnoDB: Doing recovery: scanned up to log sequence number 6017 2731341312
InnoDB: Doing recovery: scanned up to log sequence number 6017 2736584192
InnoDB: Doing recovery: scanned up to log sequence number 6017 2741827072
InnoDB: Doing recovery: scanned up to log sequence number 6017 2747069952
InnoDB: Doing recovery: scanned up to log sequence number 6017 2749799344
110811 11:05:26 InnoDB: Error: table 'misc_apis/#sql-5d87_133c21'
InnoDB: in InnoDB data dictionary has tablespace id 8422,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB:
http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
110811 11:05:26 InnoDB: Error: table 'misc_apis/#sql-6dc9_15eaf1'
InnoDB: in InnoDB data dictionary has tablespace id 8393,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB:
http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
110811 11:05:26 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 238867296, file name ./mysql-bin.000017
110811 11:05:29 InnoDB: Started; log sequence number 6017 2749799344
110811 11:05:29 [ERROR] /db/mysql/base/bin/mysqld: Table './mysql/event' is marked as crashed and should be repaired
110811 11:05:29 [Warning] Checking table: './mysql/event'
110811 11:05:29 [ERROR] 1 client is using or hasn't closed the table properly
110811 11:05:29 [Note] Event Scheduler: Loaded 4 events
110811 11:05:29 [Note] /db/mysql/base/bin/mysqld: ready for connections.
Version: '5.1.39-log' socket: '/db/mysql/tmp/mysql.sock' port: 3306 MySQL Community Server (GPL)
110811 11:05:29 [Note] Event Scheduler: scheduler thread started with id 1
Thank you!
Angel Tirone