I am working with a large (millions of rows) data table with a list column containing deeply nested lists, which do not have uniform structure, size or order of elements (list(x=1,y=2) and list(y=2,x=1) may both be present and should be treated as identical). I need to repeatedly perform arbitrary groupings that include some columns from the data table as well as a subset of the data in the list column. Not all rows have values that will match the subset.
The approach I've come up with feels overly complicated. Here are the key points:
Identifying values in a nested list structure. My approach is to use
ul <- unlist(list_col), which "flattens" nested data structures and builds hierarchical names for direct access to each element, e.g.,address.country.code.Ensuring that permutations of the same unlisted data are considered equal from a grouping standpoint. My approach is to order the unlisted vectors by the names of their values via
ul[order(names(ul))]and assign the result as a new character vector column by reference.Performing grouping on subsets of the flattened values. I was not able to get
by=to work in any way with a column whose values are lists or vectors. Therefore, I had to find a way to map unique character vectors to simple values. I did this withdigest.
Here are the two workhorse functions:
# Flatten list column in a data.table
flatten_list_col <- function(dt, col_name, flattened_col_name='props') {
flatten_props <- function(d) {
if (length(d) > 0) {
ul <- unlist(d)
names <- names(ul)
if (length(names) > 0) {
ul[order(names)]
} else {
NA
}
} else {
NA
}
}
flattened <- lapply(dt[[col_name]], flatten_props)
dt[, as.character(flattened_col_name) := list(flattened), with=F]
}
# Group by properties in a flattened list column
group_props <- function(prop_group, prop_col_name='props') {
substitute({
l <- lapply(eval(as.name(prop_col_name)), function(x) x[names(x) %in% prop_group])
as.character(lapply(l, digest))
}, list(prop_group=prop_group, prop_col_name=prop_col_name))
}
Here is a reproducible example:
library(data.table)
dt <- data.table(
id=c(1,1,1,2,2,2),
count=c(1,1,2,2,3,3),
d=list(
list(x=1, y=2),
list(y=2, x=1),
list(x=1, y=2, z=3),
list(y=5, abc=list(a=1, b=2, c=3)),
NA,
NULL
)
)
flatten_list_col(dt, 'd')
dt[, list(total=sum(count)), by=list(id, eval(group_props(c('x', 'y'))))]
The output is:
> flatten_list_col(dt, 'd')
id count d props
1: 1 1 <list> 1,2
2: 1 1 <list> 1,2
3: 1 2 <list> 1,2,3
4: 2 2 <list> 1,2,3,5
5: 2 3 NA NA
6: 2 3 NA
> dt[, list(total=sum(count)), by=list(id, eval(group_props(c('x', 'y'))))]
id group_props total
1: 1 325c6bbb2c33456d0301cf3909dd1572 4
2: 2 7aa1e567cd0d6920848d331d3e49fb7e 2
3: 2 ee7aa3b9ffe6bffdee83b6ecda90faac 6
This approach works but is pretty inefficient because of the need to flatten & order the lists and because of the need to calculate digests. I'm wondering about the following:
Can this be done without having to create a flattened column by instead retrieving values directly from the list column? This will probably require specifying selected properties as expressions as opposed to simple names.
Is there a way to get around the need for
digest?