5

我在基本的 MySQL 上“还可以”,但这是“超出我的头脑”!

目标:

  • 导入数据库
  • 更新已更改的产品
  • 删除已删除的产品
  • 快速高效

数据库表很大,速度是个问题。

是不是一定要MyISAM 是inoDB 会更快?每个数据库将位于一个唯一的表中。

我被赋予了这个作为我正在尝试做的事情的起点:

CREATE TABLE `table` LIKE LiveTable
LOAD DATA INFILE..... INTO `table`
UPDATE `table`  SET delete=1; -- Set the delete field to true  because it will not have been updated
UPDATE `table` INNER JOIN`table`ON `LiveTable.ID`=`table.ID`
SET LiveTable.Col1=table.Col1, LiveTable.Col2=table.Col2….. delete=0
INSERT INTO LiveTable(ID,Col1,Col2,…  delete=0)
SELECT ID,Col1,Col2,...FROM `table`
LEFT JOIN LiveTable 
ON table.ID = LiveTable.ID
WHERE LiveTable.ID IS NULL
DELETE FROM LiveTableWHERE delete = 0
EMPTY TABLE `table`

> CREATE TABLE `product_table`   (
>      `programname` VARCHAR(100) NOT NULL,
>      `name`        VARCHAR(160) NOT NULL,
>      `keywords`    VARCHAR(300) NOT NULL,
>      `description` TEXT NOT NULL,
>      `sku`         VARCHAR(100) NOT NULL,
>      -- This is the only "unique identifier given, none will be duplicates"
>      `price`       DECIMAL(10, 2) NOT NULL,
>      `created`     TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
>      `updatedat`   TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00',
>      `delete`      TINYINT(4) NOT NULL DEFAULT '0',
>      PRIMARY KEY (`sku`)   ) ENGINE=myisam DEFAULT CHARSET=latin1;
> 
> CREATE TABLE IF NOT EXISTS `temptable` LIKE `product_table`;
> 
> TRUNCATE TABLE `temptable`; -- Remove data from temp table if for some
> reason it has data in it.
> 
> LOAD DATA LOW_PRIORITY LOCAL INFILE "catalog.csv" INTO TABLE
> `temptable`  FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """"
> LINES TERMINATED BY "\n"  IGNORE 1 LINES (`PROGRAMNAME`, `NAME`,
> `KEYWORDS`, `DESCRIPTION`, `SKU`, `PRICE`);
> 
> 
> UPDATE `temptable` SET    `delete` = 1; -- Set the delete field to
> true UPDATE `temptable` ttable
>        INNER JOIN `product_table` mtable
>          ON ( mtable.sku = ttable.sku ) SET    mtable.programname = ttable.programname,
>        mtable.name = ttable.name,
>        mtable.keywords = ttable.keywords,
>        mtable.description = ttable.description,
>        mtable.sku = ttable.sku,
>        mtable.price = ttable.price,
>        mtable.created = ttable.created,
>        mtable.updatedat = NOW(),-- Set Last Update
>        mtable.delete = 0; -- Set Delete to NO
> 
> -- Not sure what this is for...  I'm LOST at this part...   
> INSERT INTO `product_table` VALUES      (`programname`,
>              `name`,
>              `keywords`,
>              `description`,
>              `sku`,
>              `price`,
>              `created`,
>              `updatedat`,
>              `delete`);
> 
> -- This type of join requires alias as far as I know? 
> SELECT `programname`,
>        `name`,
>        `keywords`,
>        `description`,
>        `sku`,
>        `price`,
>        `created`,
>        `updatedat`,
>        `delete` FROM   `temptable` tmptable
>        LEFT JOIN `product_table` maintbl
>          ON tmptable.sku = maintbl.sku WHERE  maintbl.sku IS NULL;
> 
> DELETE FROM `product_table` WHERE  `delete` = 0;
> 
> TRUNCATE TABLE `temptable`; `` remove all the data from temporary
> table.
4

2 回答 2

4

我自己在这里回答了这个问题: https ://dba.stackexchange.com/questions/16197/innodb-update-slow-need-a-better-option/16283#16283

使用我从这里收到的信息、网络和几个互联网聊天室,我想出了。网络来源: http: //www.softwareprojects.com/resources/programming/t-how-to-use-mysql-fast-load-data-for-updates-1753.html

[DEMO][1] http://sqlfiddle.com/#!2/4ebe0/1

过程是:

Import into a new temp table.
Update The old table information with information in Temp table.
Insert new data into the table. (Real world I'm making a new CSV file and using LOAD INTO for the insert)
delete everything that is no longer in the data feed.
delete the temp table.

这似乎是迄今为止最快的过程。

让我知道你的意见是什么。

于 2012-04-10T00:04:43.203 回答
2

InnoDB 通常在表可用时比 MyISAM 好得多INSERTUPDATE并且DELETE正在发生,因为 InnoDB 使用行级锁定进行更新,而 MyISAM 使用表级锁定。

这是第一步。

第二步是在将数据加载到表之前禁用表上的所有索引 using ALTER TABLE .. DISABLE KEYS,然后在加载后重新启用它们 using ALTER TABLE .. ENABLE KEYS

以上两个显示了你的表现有很大的提高。

作为另一种优化,在进行大规模更新时,将它们分解成批次(可能基于主键),以便所有行不会同时锁定。

于 2012-03-28T18:14:15.873 回答