bash - sed 语句更改/修改 CSV 分隔符和定界符

Question

我有一些包含逗号分隔值的 CSV 文件，并且某些列值可以包含如下字符,.<>!/\;&

我正在尝试将 CSV 转换为逗号分隔、引号括起来的 CSV

示例数据：

DateCreated,DateModified,SKU,Name,Category,Description,Url,OriginalUrl,Image,Image50,Image100,Image120,Image200,Image300,Image400,Price,Brand,ModelNumber
2012-10-19 10:52:50,2013-06-11 02:07:16,34,Austral Foldaway 45 Rotary Clothesline,Home & Garden > Household Supplies > Laundry Supplies > Drying Racks & Hangers,"Watch the Product Video            Plenty of Space to Hang a Family Wash  Austral's Foldaway 45 rotary clothesline is a folding head rotary clothes hoist beautifully finished in either Beige or Heritage Green.  Even though the Foldaway 45 is compact, you still get a large 45 metres of line space, big enough for a full family wash.  If you want the advantage of a rotary hoist, but dont want to lose your yard, then the Austral Foldaway 45 is the clothesline for you.   Installation Note: A core hole is only required when installing into existing concrete, e.g. a pathway. Not required in the ground(grass/soil).  To watch video on YouTube, click the following link: Austral Foldaway 45 Rotary Clothesline                   //           Customer Video Reviews            ",https://track.commissionfactory.com.au/p/10604/1718695,http://www.lifestyleclotheslines.com.au/austral-foldaway-45-rotary-clothesline/,http://content.commissionfactory.com.au/Products/7228/1718695.jpg,http://content.commissionfactory.com.au/Products/7228/1718695@50x50.jpg,http://content.commissionfactory.com.au/Products/7228/1718695@100x100.jpg,http://content.commissionfactory.com.au/Products/7228/1718695@120x120.jpg,http://content.commissionfactory.com.au/Products/7228/1718695@200x200.jpg,http://content.commissionfactory.com.au/Products/7228/1718695@300x300.jpg,http://content.commissionfactory.com.au/Products/7228/1718695@400x400.jpg,309.9000 AUD,Austral,FA45GR

我想要实现的输出是

"DateCreated","DateModified","SKU","Name","Category","Description","Url","OriginalUrl","Image","Image50","Image100","Image120","Image200","Image300","Image400","Price","Brand","ModelNumber"
"2012-10-19 10:52:50","2013-06-11 02:07:16","34","Austral Foldaway 45 Rotary Clothesline","Home & Garden > Household Supplies > Laundry Supplies > Drying Racks & Hangers","Watch the Product Video            Plenty of Space to Hang a Family Wash  Austral's Foldaway 45 rotary clothesline is a folding head rotary clothes hoist beautifully finished in either Beige or Heritage Green.  Even though the Foldaway 45 is compact, you still get a large 45 metres of line space, big enough for a full family wash.  If you want the advantage of a rotary hoist, but dont want to lose your yard, then the Austral Foldaway 45 is the clothesline for you.   Installation Note: A core hole is only required when installing into existing concrete, e.g. a pathway. Not required in the ground(grass/soil).  To watch video on YouTube, click the following link: Austral Foldaway 45 Rotary Clothesline                   //           Customer Video Reviews            ","https://track.commissionfactory.com.au/p/10604/1718695","http://www.lifestyleclotheslines.com.au/austral-foldaway-45-rotary-clothesline/","http://content.commissionfactory.com.au/Products/7228/1718695.jpg","http://content.commissionfactory.com.au/Products/7228/1718695@50x50.jpg","http://content.commissionfactory.com.au/Products/7228/1718695@100x100.jpg","http://content.commissionfactory.com.au/Products/7228/1718695@120x120.jpg","http://content.commissionfactory.com.au/Products/7228/1718695@200x200.jpg","http://content.commissionfactory.com.au/Products/7228/1718695@300x300.jpg","http://content.commissionfactory.com.au/Products/7228/1718695@400x400.jpg","309.9000 AUD","Austral","FA45GR"

非常感谢任何帮助。

Answer 1

首先，让我们尝试一个简单（而且“不够好”）的解决方案，它只是为每个字段添加一个双引号（包括那些已经有双引号的字段！这不是你想要的）

sed -r 's/([^,]*)/"\1"/g'

太好了，第一部分查找其中没有逗号的序列，第二部分在它们周围添加双引号，最后的“g”表示每行不止一次

这将转

abc,345, some words ,"some text","text,with,commas"

转换成 "abc","345"," some words ",""some text"",""text","with","commas""

需要注意的几点：

• 它正确地包围了“一些单词”，它们之间有空格，但也包围了初始和最终空格。我认为没关系，但如果不是，它可以修复

• 如果该字段已经有引号，它将再次被引用，这是不好的。需要修复

• 如果字段已经有引号并且内部文本有逗号（不应被视为字段分隔符），这些逗号也会被引用。这也需要修复

所以我们想匹配两个不同的正则表达式——要么是一个带引号的字符串，要么是一个没有逗号的字段：

sed -r 's/([^,"]*|"[^"]*")/"\1"/g'

结果现在是

"abc","345"," some words ",""some text"",""text,with,commas""

如您所见，我们在最初引用的文本上有一个双引号。我们将不得不使用第二个 sed 命令删除它：

sed -r 's/([^,"]*|"[^"]*")/"\1"/g' | sed 's/""/"/g'

这导致

"abc","345"," some words ","some text","text,with,commas"

耶！

Answer 2

听起来您希望文件中的每一行都以双引号开头和结尾。如果是这样，这应该工作：

sed -i.bak 's/^\(.*\)$/"\1"/' filename

Answer 3

试试这个perl解决方案。它优于我以前的awk，因为现在我使用了一个解析器来正确处理字段内的逗号。模块Text::CSV_XS必须工作：

#!/usr/bin/env perl

use strict;
use warnings;
use Text::CSV_XS;

die qq|Usage: perl $0 <csv-file>\n| unless @ARGV == 1;

open my $fh, '<', shift or die qq|ERROR: Could not open input file\n|;

my $csv = Text::CSV_XS->new( {
        always_quote => 1,
} );

while ( my $row = $csv->getline( $fh ) ) { 
        $csv->print( *STDOUT, $row );
        print "\n";
}
$csv->eof;
close $fh;