2

一位同事向我发送了一个 Elasticsearch 查询结果（100000 条记录，数百个属性），如下所示：

pets_json <- paste0('[{"animal":"cat","attributes":{"intelligence":"medium","noises":[{"noise":"meow","code":4},{"noise":"hiss","code":2}]}},',
                     '{"animal":"dog","attributes":{"intelligence":"high","noises":{"noise":"bark","code":1}}},',
                     '{"animal":"snake","attributes":{"intelligence":"low","noises":{"noise":"hiss","code":2}}}]')

有一个多余的密钥，code我不需要捕获。

我想生成一个类似于以下内容的 data.frame：

animal  intelligence  noises.bark noises.hiss noises.meow   
   cat        medium            0           1           1 
   dog          high            1           0           0
 snake           low            0           1           0

我可以在 json 中阅读，但flatten=TRUE不能完全变平：

library(jsonlite)
str(df <- fromJSON(txt=pets_json, flatten=TRUE))
# 'data.frame': 3 obs. of  3 variables:
#   $ animal                 : chr  "cat" "dog" "snake"
# $ attributes.intelligence: chr  "medium" "high" "low"
# $ attributes.noises      :List of 3
# ..$ :'data.frame':    2 obs. of  2 variables: \
#   .. ..$ noise   : chr  "meow" "hiss"         \
# .. ..$ code: int  4 2                          |
# ..$ :List of 2                                 |
# .. ..$ noise   : chr "bark"                    |- need to remove code and flatten    
# .. ..$ code: int 1                             |
# ..$ :List of 2                                 |
# .. ..$ noise   : chr "hiss"                   /
# .. ..$ code: int 2                           /

因为展平不完整，我可以在调用另一个之前使用这个中间阶段来摆脱code不需要的键flatten()，但我知道摆脱键的唯一方法真的很慢：

for( l in which(sapply(df, is.list)) ){
  for( l2 in which(sapply(df[[l]], is.list))){
    df[[l]][[l2]]['code'] <- NULL
  }
}
( df <- data.frame(flatten(df)) )
#   animal attributes.intelligence attributes.noises
# 1    cat                  medium        meow, hiss
# 2    dog                    high              bark
# 3  snake                     low              hiss

然后在那之后……？我知道使用tidyr::separate我可能会想出一种spread将噪声值放入列并设置标志的hacky方法。但这一次只适用于一个属性，我可能有数百个。我事先并不知道所有可能的属性值。

如何有效地生成所需的 data.frame？谢谢你的时间！

4

2 回答 2

2

我认为没有一种超级简单的方法可以以正确的格式获取它，但这里有一个尝试：

out <- fromJSON(pets_json)

# drop the "code" data and do some initial cleaning
out$noises <- lapply(
  out$attributes$noises, 
  function(x) unlist(x[-match("code",names(x))]) 
)

# extract the key part of the intelligence variable
out$intelligence <- out$attributes$intelligence

# set up a vector of all possible noises
unq_noises <- unique(unlist(out$noises)) 

# make the new separate noise variables
out[unq_noises] <- t(vapply(
  out$noises, 
  function(x) unq_noises %in% x,
  FUN.VALUE=logical(length(out$noises)))
)

# clean up no longer needed variables
out[c("attributes","noises")] <- list(NULL)

out

#  animal intelligence  meow  hiss  bark
#1    cat       medium  TRUE  TRUE FALSE
#2    dog         high FALSE FALSE  TRUE
#3  snake          low FALSE  TRUE FALSE
于 2015-10-24T04:47:33.877 回答
2

magrittr 和 data.table 的基本情况

这是另一个结合magrittr和data.table额外的时代精神布朗尼点的建议：

# Do not simplify to data.frame
str(df <- fromJSON(txt=pets_json, simplifyDataFrame=F))

# The %<>% operator create a pipe and assigns back to the variable
df %<>% 
  lapply(. %>%
    data.table(animal = .$animal, 
               intelligence = .$attributes$intelligence, 
               noises = unlist(.$attributes$noises)) %>% # Create a data.table
    .[!noises %in% as.character(0:9)] ) %>% # Remove numeric values
  rbindlist %>% # Combine into a single data.table
  dcast(animal + intelligence ~ paste0("noises.", noises), # Cast the noises variables
        value.var = "noises", 
        fill = 0, # Put 0 instead of NA
        fun.aggregate = function(x) 1) # Put 1 instead of noise

最终格式符合您的要求：

df
#    animal intelligence noises.bark noises.hiss noises.meow
# 1:    cat       medium           0           1           1
# 2:    dog         high           1           0           0
# 3:  snake          low           0           1           0

对于多个属性

现在，您似乎想要对多个属性进行概括。假设您的数据也有一个colors属性，例如：

pets_json <- paste0('[{"animal":"cat","attributes":{"intelligence":"medium","noises":[{"noise":"meow","code":4},{"noise":"hiss","code":2}],"colors":[{"color":"brown","code":4},{"color":"white","code":2}]}},',
                    '{"animal":"dog","attributes":{"intelligence":"high","noises":{"noise":"bark","code":1},"colors":{"color":"brown","code":4}}},',
                    '{"animal":"snake","attributes":{"intelligence":"low","noises":{"noise":"hiss","code":2},"colors":[{"color":"green","code":4},{"color":"brown","code":4}]}}]')

然后你可以按照这个通用的代码，它相当难看，但应该可以正常工作：

# Do not simplify to data.frame
str(df <- fromJSON(txt=pets_json, simplifyDataFrame=F))

# Set up the attributes names
attr.names <- c("noises", "colors")

# The %<>% operator create a pipe and assigns back to the variable
df %<>% 
  lapply(function(.)
    eval(parse(text=paste0(
      "data.table(animal = .$animal, ",
      "intelligence = .$attributes$intelligence, ", 
      paste0(attr.names, " = unlist(.$attributes$", attr.names, ")", collapse=", "), 
      ")")))
    %>%
      .[eval(parse(text=paste("!", attr.names, "%in% as.character(0:9)", collapse = " & ")))] ) %>%
  rbindlist 

# Cast each variable and merge together
df <- dcast(melt(df, measure.vars=c(attr.names)), 
        animal + intelligence ~ variable + value, sep=".")

#    animal intelligence noises.bark noises.hiss noises.meow colors.brown
# 1:    cat       medium           0           1           1            1
# 2:    dog         high           1           0           0            1
# 3:  snake          low           0           1           0            1
#    colors.green colors.white
# 1:            0            1
# 2:            0            0
# 3:            1            0

此解决方案也适用于单个属性，例如attr.names <- c("noises").

于 2015-10-24T06:22:47.493 回答