Hi guys,
Yesterday I experienced and unclean shutdown due a powe outage, today we found the MySQL service isn't starting and the InnoDB repair process stalls at "Reading tablespace information from the .ibd files"
I left it running for +5 hours without any luck, ib* files arent getting updated, top, iotop and cat /proc/$PID/io arent showing any work from mysqld so I believe the repair process isnt doing anything really. The init.d timeouts and fails after 5min.
Im running MySQL 5.1 on CentOS 6.5, my /var/lib/mysql is 230GB big, heres my my.cnf and mysqld error log file.
Can someone help? Any help is highly appreciated.
====
/etc/my.cnf:
[mysqld]
ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
innodb_file_per_table=1
# safe-show-database
skip-locking
skip-name-resolve
max_connections=700
key_buffer=512M
myisam_sort_buffer_size=64M
join_buffer=10M
record_buffer=1M
sort_buffer=2M
wait_timeout=165
connect_timeout=160
tmp_table_size=180M
max_heap_table_size=180M
innodb_buffer_pool_size = 2G
key_buffer_size=1G
query_cache_limit = 3M
query_cache_size = 128M
query_cache_type = 1
max_allowed_packet=1G
table_cache=8192
table_definition_cache=8192
table_open_cache=4096
interactive_timeout=50
open_files_limit=20000
join_buffer_size=16M
thread_cache_size=4
#log-queries-not-using-indexes
log-error=/var/log/mysqld.log
#log-slow-queries = /var/log/mysql-slow.log
#long_query_time = 55
server-id = 1
binlog-do-db=dbname1
binlog-do-db=dbname2
binlog-do-db=dbname3
binlog-do-db=dbname4
binlog-do-db=dbname5
binlog-do-db=dbname6
relay-log = /var/lib/mysql/mysql-relay-bin
relay-log-index = /var/lib/mysql/mysql-relay-bin.index
master-info-file = /var/lib/mysql/mysql-master.info
relay-log-info-file = /var/lib/mysql/mysql-relay-log.info
log-bin = /var/lib/mysql/mysql-bin
expire_logs_days = 10
====
====
/var/log/mysqld.log
150903 13:35:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150903 13:35:40 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
150903 13:35:40 [Warning] 'record_buffer' is deprecated and will be removed in a future release. Please use 'read_buffer_size' instead.
150903 13:35:40 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
150903 13:35:40 InnoDB: Initializing buffer pool, size = 2.0G
150903 13:35:40 InnoDB: Completed initialization of buffer pool
150903 13:35:40 InnoDB: highest supported file format is Barracuda.
InnoDB: ##########################################################
InnoDB: WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 44233, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 5594482121932 and 5594482121932.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150903 13:35:40 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
====
# service mysql start
Starting MySQL.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ERROR!
Yesterday I experienced and unclean shutdown due a powe outage, today we found the MySQL service isn't starting and the InnoDB repair process stalls at "Reading tablespace information from the .ibd files"
I left it running for +5 hours without any luck, ib* files arent getting updated, top, iotop and cat /proc/$PID/io arent showing any work from mysqld so I believe the repair process isnt doing anything really. The init.d timeouts and fails after 5min.
Im running MySQL 5.1 on CentOS 6.5, my /var/lib/mysql is 230GB big, heres my my.cnf and mysqld error log file.
Can someone help? Any help is highly appreciated.
====
/etc/my.cnf:
[mysqld]
ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
innodb_file_per_table=1
# safe-show-database
skip-locking
skip-name-resolve
max_connections=700
key_buffer=512M
myisam_sort_buffer_size=64M
join_buffer=10M
record_buffer=1M
sort_buffer=2M
wait_timeout=165
connect_timeout=160
tmp_table_size=180M
max_heap_table_size=180M
innodb_buffer_pool_size = 2G
key_buffer_size=1G
query_cache_limit = 3M
query_cache_size = 128M
query_cache_type = 1
max_allowed_packet=1G
table_cache=8192
table_definition_cache=8192
table_open_cache=4096
interactive_timeout=50
open_files_limit=20000
join_buffer_size=16M
thread_cache_size=4
#log-queries-not-using-indexes
log-error=/var/log/mysqld.log
#log-slow-queries = /var/log/mysql-slow.log
#long_query_time = 55
server-id = 1
binlog-do-db=dbname1
binlog-do-db=dbname2
binlog-do-db=dbname3
binlog-do-db=dbname4
binlog-do-db=dbname5
binlog-do-db=dbname6
relay-log = /var/lib/mysql/mysql-relay-bin
relay-log-index = /var/lib/mysql/mysql-relay-bin.index
master-info-file = /var/lib/mysql/mysql-master.info
relay-log-info-file = /var/lib/mysql/mysql-relay-log.info
log-bin = /var/lib/mysql/mysql-bin
expire_logs_days = 10
====
====
/var/log/mysqld.log
150903 13:35:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150903 13:35:40 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
150903 13:35:40 [Warning] 'record_buffer' is deprecated and will be removed in a future release. Please use 'read_buffer_size' instead.
150903 13:35:40 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
150903 13:35:40 InnoDB: Initializing buffer pool, size = 2.0G
150903 13:35:40 InnoDB: Completed initialization of buffer pool
150903 13:35:40 InnoDB: highest supported file format is Barracuda.
InnoDB: ##########################################################
InnoDB: WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 44233, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 5594482121932 and 5594482121932.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150903 13:35:40 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
====
# service mysql start
Starting MySQL.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ERROR!