8

我正在编写一个 SQL 脚本来将多个 .CSV 文件复制到 postgres 数据库中，如下所示：

COPY product(title, department) from 'ys.csv' CSV HEADER;

我有多个要复制的文件。我不想要：

COPY product(title, department) from 'ys1.csv' CSV HEADER;
COPY product(title, department) from 'ys2.csv' CSV HEADER;
COPY product(title, department) from 'ys3.csv' CSV HEADER;
COPY product(title, department) from 'ys4.csv' CSV HEADER;
COPY product(title, department) from 'ys5.csv' CSV HEADER;

我想为此使用 for 循环而不是多个复制命令。这可能吗？谢谢

4

6 回答 6

13

在 linux 管道中，将列出的文件的输出输出到 psql。copy使用标准输入：

cat /path_to/ys*.csv | psql -c 'COPY product(title, department) from stdin CSV HEADER'

在其他操作系统中寻找等价物

于 2013-08-30T16:31:25.677 回答
9

我尝试了上面的答案，但是在处理多个文件时出现错误。我认为在第二个文件中它没有切断标题。

这对我有用：

# get filenames
IMPFILES=(path/FileNamepart.csv)

# import the files
for i in ${IMPFILES[@]}
    do
        psql -U user -d database -c "\copy TABLE_NAME from '$i' DELIMITER ';' CSV HEADER"
        # move the imported file
        mv $i /FilePath
    done

就我而言，我在导入每个文件后移动它。如果发生错误，我知道去哪里找。如果该位置有新文件，我可以再次运行该脚本。

于 2015-02-04T20:02:42.350 回答
5

如果您想使用PROGRAM(Postgres > 9.3) 关键字，但您可以使用每个 csv 文件中的标题awk：

COPY product(title, department) FROM PROGRAM 'awk FNR-1 ys*.csv | cat' DELIMITER ',' CSV;
于 2019-12-05T15:07:10.113 回答
4

从 Postgres 9.3 开始，您可以使用 command 中的PROGRAM关键字运行 shellCOPY命令。

COPY product(title, department) from PROGRAM 'cat ys*.csv' FORMAT CSV HEADER
于 2018-10-29T21:07:29.400 回答
2

只有一个选项，使用 pg_ls_dir 和 format()。将 'E:\Online_Monitoring\Processed\' 文件夹中的所有文件插入 ONLMON_T_Online_Monitoring 表。

DO $$
DECLARE
  directory_path VARCHAR(500);
    rec RECORD;
BEGIN
  directory_path := 'E:\\Online_Monitoring\\Processed\\';
    FOR rec IN SELECT pg_ls_dir(directory_path) AS file_name
    LOOP
      EXECUTE format(
            '
                COPY ONLMON_T_Online_Monitoring
                    (
                        item
                    , storeCode
                    , data
                    )
                FROM %L
                WITH (FORMAT CSV, HEADER);
                ', directory_path || rec.file_name
        );
    END LOOP;
END; $$;
于 2019-02-27T09:04:15.873 回答
2

您可以使用 pg_ls_dir 遍历文件名。

DO $$

DECLARE file_path TEXT; -- Path where your CSV files are
DECLARE fn_i TEXT; -- Variable to hold name of current CSV file being inserted
DECLARE mytable TEXT; -- Variable to hold name of table to insert data into

BEGIN

    file_path := 'C:/Program Files/PostgreSQL/9.6/data/my_csvs/'; -- Declare the path to your CSV files. You probably need to put this in your PostgreSQL file path to avoid permission issues.
    mytable := 'product(title,department)'; -- Declare table to insert data into. You can give columns too since it's just going into an execute statement.

    CREATE TEMP TABLE files AS 
    SELECT file_path || pg_ls_dir AS fn -- get all of the files in the directory, prepending with file path
    FROM pg_ls_dir(file_path);

    LOOP    
        fn_i := (select fn from files limit 1); -- Pick the first file
        raise notice 'fn: %', fn_i;
        EXECUTE 'COPY ' || mytable || ' from ''' || fn_i || ''' with csv header';
        DELETE FROM files WHERE fn = fn_i; -- Delete the file just inserted from the queue
        EXIT  WHEN (SELECT COUNT(*) FROM files) = 0;
     END LOOP;

END $$;
于 2017-10-09T20:32:41.373 回答