0

我必须将 90Mb 的数据插入 MySql 表中，并且我正在使用 INSERT IGNORE 命令来避免重复键的异常。性能是每秒 8 条记录，但似乎很慢。我能快点吗？

ps 我正在为每条记录插入记录，因为我从 sql 紧凑型数据库中读取数据

using (SqlCeConnection sqlConnection = new SqlCeConnection(connectionstrCe))
            {
                sqlConnection.Open();
                SqlCeCommand cmdCe = sqlConnection.CreateCommand();
                {
                    {

                        mySQLConnection.Open();

                        foreach (KeyValuePair<string, List<string>> t in tablesCeNames) //reading the tables property from the dictionary - column names and datatypes
                        {

                            string tableData = t.Key; 
                            List<string> columnData = t.Value;
//get the values from the table I want to transfer the data
                                cmdText = "SELECT * FROM " + tableData;
//compose the mysql command
                                cmdCe.CommandText = cmdText;

                                SqlCeDataReader dataReader = cmdCe.ExecuteReader(); //read
//InsertTable is a method that get the datareader and convert all the data from this table in a list array with the values to insert
                                inputValues = InsertTables(dataReader);


                                MySql.Data.MySqlClient.MySqlTransaction transakcija;
                                transakcija = mySQLConnection.BeginTransaction();

                                worker.ReportProgress(4, inputValues.Count);

                                foreach (string val in inputValues)//foreach row of values of the data table
                                {
                                    cmdSqlText = "INSERT IGNORE INTO " + tableData + "("; //compose the command for sql

                                    foreach (string cName in columnData)  //forach column in a table
                                    {
                                        string[] data = cName.Split(' ');
                                        if (!data[0].ToString().Equals("Id"))
                                        {
                                            cmdSqlText += data[0].ToString() + ","; //write the column names of the values that will be inserted  
                                        }
                                    }
                                    cmdSqlText = cmdSqlText.TrimEnd(',');
                                    cmdSqlText += ") VALUES (";
//val contains the values of this current record that i want to insert
                                    cmdSqlText += val;  //final command with insert ignore and the values of one record
                                    if (!val.Equals(""))
                                    {
                                        try
                                        {
                                            new MySql.Data.MySqlClient.MySqlCommand(cmdSqlText, mySQLConnection, transakcija).ExecuteNonQuery(); //execute insert on sql database
                                            WriteToTxt("uspješno upisano u mysql" + t.Key);
                                        }
                                        catch (MySql.Data.MySqlClient.MySqlException sqlEx)
                                        {
                                        }
                                    }
                                }

                                if (TablicaSveOK)
                                {
                                    transakcija.Commit();

                                }
                                else
                                {
                                    transakcija.Rollback();
                                }
                            }
                        }

                        if (mySQLConnection.State != System.Data.ConnectionState.Closed)
                        {
                            mySQLConnection.Close();
                        }
                    }
4

3 回答 3

1

将数据从 Sql 获取到文件并使用 LOAD DATA 怎么样？

http://dev.mysql.com/doc/refman/5.0/es/load-data.html

于 2013-08-01T09:06:01.240 回答
0

Rather then sending multiple calls you can send one call to insert all records. Insert it like this

insert into your table(col1, col2)
SELECT 'Name1', 'Location1' 
UNION ALL
SELECT 'Name2', 'Location2'
UNION ALL
SELECT 'Name3', 'Location3'

A part from this it is possible that your code is the bottle neck rather than the insert statement. So i would recommend you to first check where the problem lies and then go for the solution.

于 2013-08-01T09:08:04.007 回答
0

最新的 MySql.Connector 有一个名为MySqlBulkLoader的类，可用于以更面向 NET 的方式调用 MySql 的 LOAD DATA 语法。该类需要一个逗号分隔值文件来加载，并且速度非常快。

因此，您的工作是读取数据表中的所有 Sql Compact 记录，然后使用流写入器或专门的 CSV 写入器将所有内容写入文件中。

然后将数据加载到 MySql 表中的代码很简单，如下所示

string connStr = "server=localhost;user=root;database=........";
using(MySqlConnection conn = new MySqlConnection(connStr))
{
     MySqlBulkLoader bl = new MySqlBulkLoader(conn);
     bl.TableName = "yourdestinationtable";
     bl.FieldTerminator = "\t";
     bl.LineTerminator = "\n";
     bl.FileName = "path_to_your_comma_separated_value_file";
     try
     {
        conn.Open();
        int count = bl.Load();
     }
}
于 2013-08-01T09:14:38.857 回答