Greetings
We use 5.6.15 common edition at a Linux Centos 6.4 box (at VMWare).
Since yesterday - we experience almost every hour a failure of MySQL processes,
with following errors :
….
2014-09-16 16:16:54 25931 [ERROR] InnoDB: The OS said file flush did not succeed
2014-09-16 16:16:54 7f65897fb700 InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2014-09-16 16:16:54 25931 [ERROR] InnoDB: File (unknown): 'flush' returned OS error 105. Cannot continue operation
pure virtual method called
pure virtual method called
140916 16:16:54 mysqld_safe Number of processes running now: 0
140916 16:16:54 mysqld_safe mysqld restarted
….
Our System/Storage/Linux/VMWare people are looking for some IO errors at OS level.
The first failures were accompinied by following messages in /var/log/messages:
...
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] Sense Key : Aborted Command [current]
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] Add. Sense: Internal target failure
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] CDB: Write(10): 2a 00 00 90 d9 5f 00 03 c0 00
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186340
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186341
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186342
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186343
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186344
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186345
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186346
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186347
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186348
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186349
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:55:01 vm-ngmmydbp1 kernel: __ratelimit: 110 callbacks suppressed
...
But in the latest few MySQL failures - there were no accompynied var-log-messages,
whereas the mysql errors left the same...
Any clue of how to resolve or workaround the situation ?
Best regards,
Avi
DBA
We use 5.6.15 common edition at a Linux Centos 6.4 box (at VMWare).
Since yesterday - we experience almost every hour a failure of MySQL processes,
with following errors :
….
2014-09-16 16:16:54 25931 [ERROR] InnoDB: The OS said file flush did not succeed
2014-09-16 16:16:54 7f65897fb700 InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2014-09-16 16:16:54 25931 [ERROR] InnoDB: File (unknown): 'flush' returned OS error 105. Cannot continue operation
pure virtual method called
pure virtual method called
140916 16:16:54 mysqld_safe Number of processes running now: 0
140916 16:16:54 mysqld_safe mysqld restarted
….
Our System/Storage/Linux/VMWare people are looking for some IO errors at OS level.
The first failures were accompinied by following messages in /var/log/messages:
...
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] Sense Key : Aborted Command [current]
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] Add. Sense: Internal target failure
Sep 15 18:54:28 vm-ngmmydbp1 kernel: sd 3:0:0:0: [sdi] CDB: Write(10): 2a 00 00 90 d9 5f 00 03 c0 00
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186340
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186341
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186342
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186343
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186344
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186345
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186346
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186347
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186348
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:54:28 vm-ngmmydbp1 kernel: Buffer I/O error on device dm-4, logical block 1186349
Sep 15 18:54:28 vm-ngmmydbp1 kernel: lost page write due to I/O error on dm-4
Sep 15 18:55:01 vm-ngmmydbp1 kernel: __ratelimit: 110 callbacks suppressed
...
But in the latest few MySQL failures - there were no accompynied var-log-messages,
whereas the mysql errors left the same...
Any clue of how to resolve or workaround the situation ?
Best regards,
Avi
DBA