7

我正在读取如下所示的数据文件：

userId, fullName,email,password,activated,registrationDate,locale,notifyOnUpdates,lastSyncTime,plan_id,plan_period_months,plan_price,plan_exp_date,plan_is_trial,plan_is_trial_used,q_hear,q_occupation,pp_subid,pp_payments,pp_since,pp_cancelled,apikey
"2","John Smith,"john.smith@gmail.com","a","1","2004-07-23 14:19:32","en_US","1","2011-04-07 07:29:17","3",\N,\N,\N,"0","1",\N,\N,\N,\N,\N,\N,"d7734dce-4ae2-102a-8951-0040ca38ff83"

但实际文件大约有 20000 条记录。我使用以下 R 代码将其读入：

user = read.csv("~/Desktop/dbdump/users.txt", na.strings = "\\N", quote="")

我拥有的原因quote=""是因为没有它，导入会过早停止。我最终得到了 9569 个观察结果。为什么我不明白为什么确切地quote=""克服了这个问题，它似乎这样做了。

除了它引入了我必须“修复”的其他问题。我看到的第一个是日期最终是包含引号的字符串，当我使用它们时不想转换为实际日期to.Date()。

现在我可以修复字符串并破解我的方式。但更好地了解我在做什么。有人可以解释一下：

  1. 为什么quote=""修复“坏数据”
  2. 找出导致read.csv提前停止的原因的最佳实践技术是什么？（如果我只查看 +/- 指示行的输入数据，我看不出有任何问题）。

这是“问题”“附近”的行。我没看到你的伤害吗？

"16888","user1","user1@gmail.com","TeilS12","1","2008-01-19 08:47:45","en_US","0","2008-02-23 16:51:53","1",\N,\N,\N,"0","0","article","student",\N,\N,\N,\N,"ad949a8e-17ed-102b-9237-0040ca390025"
"16889","user2","user2@gmail.com","Gaspar","1","2008-01-19 10:34:11","en_US","1",\N,"1",\N,\N,\N,"0","0","email","journalist",\N,\N,\N,\N,"8b90f63a-17fc-102b-9237-0040ca390025"
"16890","user3","user3@gmail.com","boomblaadje","1","2008-01-19 14:36:54","en_US","0",\N,"1",\N,\N,\N,"0","0","article","student",\N,\N,\N,\N,"73f31f4a-181e-102b-9237-0040ca390025"
"16891","user4","user4@gmail.com","mytyty","1","2008-01-19 15:10:45","en_US","1","2008-01-19 15:16:45","1",\N,\N,\N,"0","0","google-ad","student",\N,\N,\N,\N,"2e48e308-1823-102b-9237-0040ca390025"
"16892","user5","user5@gmail.com","08091969","1","2008-01-19 15:12:50","en_US","1",\N,"1",\N,\N,\N,"0","0","dont","dont",\N,\N,\N,\N,"79051bc8-1823-102b-9237-0040ca390025"

* 更新 *

这更棘手。即使导入的总行数是 9569，如果我查看最后几行，它们对应于最后几行数据。因此，我推测在导入过程中发生了一些事情，导致很多行被跳过。事实上 15914 - 9569 = 6345 条记录。当我在那里有quote=""时，我得到15914。

所以我的问题可以修改：有没有办法让 read.csv 报告它决定不导入的行？

* 更新 2 *

@Dwin，我不得不删除 na.strings="\N" 因为 count.fields 函数不允许这样做。有了这个，我得到了这个看起来很有趣但我不明白的输出。

3     4    22    23    24 
1    83 15466   178     4

您的第二个命令会产生大量数据（并在达到 max.print 时停止。）但第一行是这样的：

[1]  2  4  2  3  5  3  3  3  5  3  3  3  2  3  4  2  3  2  2  3  2  2  4  2  4  3  5  4  3  4  3  3  3  3  3  2  4

我不明白输出是否应该显示每条输入记录中有多少字段。显然第一行都有超过 2,4,2 等字段......感觉我越来越近了，但仍然感到困惑！

4

2 回答 2

4

我发现的一个问题（感谢data.table）是 之后缺少引号（“）John Smith。这对于您拥有的其他行也可能是一个问题吗？

如果我在 之后添加“缺少的”引号John Smith，它读起来很好。

我将此数据保存到data.txt：

userId, fullName,email,password,activated,registrationDate,locale,notifyOnUpdates,lastSyncTime,plan_id,plan_period_months,plan_price,plan_exp_date,plan_is_trial,plan_is_trial_used,q_hear,q_occupation,pp_subid,pp_payments,pp_since,pp_cancelled,apikey
"2","John Smith","john.smith@gmail.com","a","1","2004-07-23 14:19:32","en_US","1","2011-04-07 07:29:17","3",\N,\N,\N,"0","1",\N,\N,\N,\N,\N,\N,"d7734dce-4ae2-102a-8951-0040ca38ff83"
"16888","user1","user1@gmail.com","TeilS12","1","2008-01-19 08:47:45","en_US","0","2008-02-23 16:51:53","1",\N,\N,\N,"0","0","article","student",\N,\N,\N,\N,"ad949a8e-17ed-102b-9237-0040ca390025"
"16889","user2","user2@gmail.com","Gaspar","1","2008-01-19 10:34:11","en_US","1",\N,"1",\N,\N,\N,"0","0","email","journalist",\N,\N,\N,\N,"8b90f63a-17fc-102b-9237-0040ca390025"
"16890","user3","user3@gmail.com","boomblaadje","1","2008-01-19 14:36:54","en_US","0",\N,"1",\N,\N,\N,"0","0","article","student",\N,\N,\N,\N,"73f31f4a-181e-102b-9237-0040ca390025"
"16891","user4","user4@gmail.com","mytyty","1","2008-01-19 15:10:45","en_US","1","2008-01-19 15:16:45","1",\N,\N,\N,"0","0","google-ad","student",\N,\N,\N,\N,"2e48e308-1823-102b-9237-0040ca390025"
"16892","user5","user5@gmail.com","08091969","1","2008-01-19 15:12:50","en_US","1",\N,"1",\N,\N,\N,"0","0","dont","dont",\N,\N,\N,\N,"79051bc8-1823-102b-9237-0040ca390025"

这是一个代码。两者都fread工作read.csv正常。

require(data.table)

dat1 <- fread("data.txt", header = T, na.strings = "\\N")
dat1

dat2 <- read.csv("data.txt", header = T, na.strings = "\\N")
dat2
于 2013-04-02T19:21:20.913 回答
4

该count.fields函数在确定在哪里查找格式错误的数据时非常有用。

这给出了每行忽略引用的字段列表，如果有嵌入的逗号，可能会出现问题：

table( count.fields("~/Desktop/dbdump/users.txt", quote="", sep=",") )

这给出了一个忽略引号和 "#"(octothorpe) 作为注释字符的列表：

table( count.fields("~/Desktop/dbdump/users.txt",  quote="", comment.char="") )

Atfer 查看您为第一个表格报告的内容......其中大部分都符合要求......您可以获得具有非 22 值的行位置列表（使用逗号和非引号设置）：

which( count.fields("~/Desktop/dbdump/users.txt", quote="", sep=",") != 22)

fill=TRUE有时，如果唯一的困难是在行尾缺少逗号，则可以解决问题。

于 2013-04-02T20:14:55.740 回答