0

我一直在使用 WAMP 来摄取一些 csv 日志，并希望通过编写一些我需要采取的常规操作的脚本来转向更自动化的过程。

我在 PHPmyadmin 中使用直接 CSV 导入功能来处理 CSV 的方言和细节。

我已经用 Python 编写了一个上传器，使用解析日志文件的 MySQLdb，但是由于日志包含一些无用的字符，我发现我需要在我可能不想的地方进行大量的清理输入。 .

例如，日志是来自目录扫描器的一些数据，我无法控制人们使用的文件夹命名约定。我有这个文件夹：-

"C:\user\NZ Business Roundtable_Download_13Feb2013, 400 Access"

并且,char 被读取为新的字段标记（毕竟是 csv）。我真正想做的是忽略引号内的所有文本：-"......"

我看到了与'字符类似的问题，我相信还会有更多。

我发现了这个：- http://www.tech-recipes.com/rx/2345/import_csv_file_directly_into_mysql/显示了我如何编写 Python 脚本来像 PHPmyadmin 加载例程一样运行。主要使用这个片段：

load data local infile 'uniq.csv' into table tblUniq fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(uniqName, uniqCity, uniqComments)

然而，我想保护我已经编写脚本的表有一些深度处理和更改，所以想知道是否有一种方法可以“告诉”我想""用作文本封装的 MySQL。我要保护的主要处理是在创建新表时给它一个特定的表名，并在其余的处理过程中使用它。

我的表格制作脚本示例：-

def make_table(self):
    query ="DROP TABLE IF EXISTS `atl`.`{}`".format(self.table)
    self.cur.execute(query)
    query = "CREATE TABLE IF NOT EXISTS `atl`.`{}` (`PK` INT NOT NULL AUTO_INCREMENT PRIMARY KEY, `ID` varchar(10), `PARENT_ID` varchar(10), `URI` varchar(284), \
        `FILE_PATH` varchar(230), `NAME` varchar(125), `METHOD` varchar(9), `STATUS` varchar(4), `SIZE` varchar(9), \
        `TYPE` varchar(9), `EXT` varchar(11), `LAST_MODIFIED` varchar(19), `EXTENSION_MISMATCH` varchar(20), `MD5_HASH` varchar(32), \
        `FORMAT_COUNT` varchar(2), `PUID` varchar(9), `MIME_TYPE` varchar(71), `FORMAT_NAME` varchar(59), `FORMAT_VERSION` varchar(7), \
        `delete_flag` tinyint, `delete_reason` VARCHAR(80), `move_flag` TINYINT, `move_reason` VARCHAR(80), \
        `ext_change_flag` TINYINT, `ext_change_reason` VARCHAR(80), `ext_change_value` VARCHAR(4), `fname_change_flag` TINYINT, `fname_change_reason` VARCHAR(80),\
        `fname_change_value` VARCHAR(80))".format(self.table)
    self.cur.execute(query)
    self.mydb.commit()

我的摄取脚本示例：-

 def ingest_row(self, row):
    query = "insert"
    # Prepare SQL query to INSERT a record into the database.
    query = "INSERT INTO `atl`.`{0}` (`ID`, `PARENT_ID`, `URI`, `FILE_PATH`, `NAME`, `METHOD`, `STATUS`, `SIZE`, `TYPE`, `EXT`, \
        `EXTENSION_MISMATCH`, `LAST_MODIFIED`, `MD5_HASH`, `FORMAT_COUNT`, `PUID`, `MIME_TYPE`, `FORMAT_NAME`,  `FORMAT_VERSION`) \
        VALUES ('{1}','{2}','{3}','{4}','{5}','{6}','{7}','{8}','{9}','{10}','{11}','{12}','{13}','{14}','{15}','{16}','{17}','{18}')".format(self.table, row[0], row[1], row[2], row[3], row[4], \
         row[5], row[6], row[7], row[8], row[9], row[10], row[11], row[12], row[13], row[14], row[15], row[16], row[17])
    try:
        self.cur.execute(query)
        self.mydb.commit()
    except:
        print query
        quit()

日志示例：-

"ID","PARENT_ID","URI","FILE_PATH","NAME","METHOD","STATUS","SIZE","TYPE","EXT","LAST_MODIFIED","EXTENSION_MISMATCH","MD5_HASH","FORMAT_COUNT","PUID","MIME_TYPE","FORMAT_NAME","FORMAT_VERSION"
"1","","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access","NZ Business Roundtable_Download_13Feb2013, 400 Access",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
"2","1","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/1993/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access\1993","1993",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
4

2 回答 2

1

永远不要使用字符串格式、连接等来构建 sql 查询！

dbapi要求所有驱动程序支持参数化查询，参数应提供给execute游标的方法。对于MySQLdb支持格式样式参数化的 ，它看起来像：

cursor.execute('insert into sometable values (%s, %s)', ('spam', 'eggs'))

提供的参数已被库正确转义，因此您的字符串是否包含必须转义的字符并不重要。

在您的特殊情况下，唯一的例外是表名，因为转义会产生非法的 sql。

于 2013-07-14T20:19:28.580 回答
1

你应该使用SQL prepared statements. 混合数据和 sql 代码format打开了大门SQL injection（这几乎总是排名前 25 位的软件缺陷/安全问题的第一名）。

例如，这是您的数据：

>>> log = """\
... "ID","PARENT_ID","URI","FILE_PATH","NAME","METHOD","STATUS","SIZE","TYPE","EXT","LAST_MODIFIED","EXTENSION_MISMATCH","MD5_HASH","FORMAT_COUNT","PUID","MIME_TYPE","FORMAT_NAME","FORMAT_VERSION"
... "1","","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access","NZ Business Roundtable_Download_13Feb2013, 400 Access",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
... "2","1","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/1993/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access\1993","1993",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
... """

我没有文件，所以假设我有：

>>> import StringIO
>>> logfile = StringIO.StringIO(log)

然后让我们构建查询：

>>> import csv
>>> csvreader = csv.reader(logfile)
>>> fields = csvreader.next()
>>> 
>>> table = 'mytable'
>>> 
>>> fields_fmt = ', '.join([ '`%s`' % f for f in fields ])
>>> values_fmt = ', '.join(['%s'] * len(fields))
>>> query = "INSERT INTO `atl`.`{0}` ({1}) VALUES ({2})".format(
... #        self.table, fields_fmt, values_fmt)
...         table, fields_fmt, values_fmt)
>>> query
'INSERT INTO `atl`.`mytable` (`ID`, `PARENT_ID`, `URI`, `FILE_PATH`, `NAME`, `METHOD`, `STATUS`, `SIZE`, `TYPE`, `EXT`, `LAST_MODIFIED`, `EXTENSION_MISMATCH`, `MD5_HASH`, `FORMAT_COUNT`, `PUID`, `MIME_TYPE`, `FORMAT_NAME`, `FORMAT_VERSION`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'

那么如果你按摩ingest_row：

def ingest_row(self, row):
    try:
        self.cur.execute(query, row)
        self.mydb.commit()
    except:
        print query
        quit()

然后，您可以使用以下方法导入数据：

for row in csvreader:
    ingest_row(row)
于 2013-07-14T21:19:30.550 回答