Hello,
I am trying to import very large csv files into a MySQL database I have created. it is the TPC-H dbgen dataset with a scale of 1000. Basically I have a few tables, and the total data size 1.2TB. I am importing roughly 6 csv files, one is close to 500GB is size. With a 50GB dataset, I am able to import using the "load data local infile" command, however with this larger data i keep hitting the error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
I have tried adjust the wait_timeout and connect_timeout with no luck. Is there a size limit that MySQL can work with, or is there a setting i am missing?
This is the timeout settings I tried using:
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 28800 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 120 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| wait_timeout | 28800 |
+----------------------------+----------+
Thanks!
I am trying to import very large csv files into a MySQL database I have created. it is the TPC-H dbgen dataset with a scale of 1000. Basically I have a few tables, and the total data size 1.2TB. I am importing roughly 6 csv files, one is close to 500GB is size. With a 50GB dataset, I am able to import using the "load data local infile" command, however with this larger data i keep hitting the error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
I have tried adjust the wait_timeout and connect_timeout with no luck. Is there a size limit that MySQL can work with, or is there a setting i am missing?
This is the timeout settings I tried using:
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 28800 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 120 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| wait_timeout | 28800 |
+----------------------------+----------+
Thanks!