r - R中Windows 10的并行包

Question

我有这个数据集，我试图在 R 中解析。来自HMDB的数据和数据集名称是Serum Metabolites（以 xml 文件的格式）。xml 文件包含大约 25K 代谢物节点，每个我想解析到子节点

我有一个将 XML 文件解析为 R 中的列表对象的代码。由于 XML 文件非常大，并且每个代谢物都有大约 12 个我想要的子节点，因此解析文件需要很长时间。约 3 小时至 1,000 种代谢物。我正在尝试使用该软件包parallel，但收到并出错。

包裹：

library("XML")
library("xml2")
library( "magrittr" )  #for pipe operator %>%
library("pbapply") # to track on progress  
library("parallel")

功能：

# The function receives an XML file (its location) and returns a list of nodes
 
Short_Parser_HMDB <- function(xml.file_location){
  start.time<- Sys.time()
  # Read as xml file
  doc <- read_xml( xml.file_location )
  #get metabolite nodes (only first three used in this sample)
  
  met.nodes <- xml_find_all( doc, ".//d1:metabolite" )  [1:1000] # [(i*1000+1):(1000*i+1000)]  # [1:3]  
  #list of data.frame
  xpath_child.v <- c( "./d1:accession",
                      "./d1:name"  ,
                      "./d1:description",
                      "./d1:synonyms/d1:synonym"  ,
                      "./d1:chemical_formula"   ,
                      "./d1:smiles" ,
                      "./d1:inchikey"    ,
                      "./d1:biological_properties/d1:pathways/d1:pathway/d1:name"   ,
                      "./d1:diseases/d1:disease/d1:name"   ,
                      "./d1:diseases/d1:disease/d1:references",
                      
                      "./d1:kegg_id"   ,                
                      "./d1:meta_cyc_id"
  )
  
  child.names.v <- c( "accession",
                      "name" ,  
                      "description" ,
                      "synonyms"  ,
                      "chemical_formula" , 
                      "smiles" ,
                      "inchikey"  , 
                      "pathways_names" ,
                      "diseases_name",
                      "references",
                      
                      "kegg_id" , 
                      "meta_cyc_id"
  ) 
  #first, loop over the met.nodes
  L.sec_acc <- parLapply(cl, met.nodes, function(x) {   # pblapply to track progress or lapply but slows down dramticlly the function  and parLapply fo parallel 
    #second, loop over the xpath desired child-nodes
    temp <-  parLapply(cl, xpath_child.v, function(y) { 
      xml_find_all(x, y ) %>% xml_text(trim = T) %>% data.frame( value = .)
    })
    #set their names
    names(temp) = child.names.v
    return(temp)
  }) 
  end.time<- Sys.time()
  total.time<- end.time-start.time
  print(total.time)
  return(L.sec_acc )
    
}

现在创建环境：

# select the location where the XML file is 
location= "D:/path/to/file//HMDB/DataSets/serum_metabolites/serum_metabolites.xml"


cl <-makeCluster(detectCores(), type="PSOCK")
clusterExport(cl, c("Short_Parser_HMDB", "cl"))
clusterEvalQ(cl,{library("parallel") 
                library("magrittr")
                library("XML")
                library("xml2")
  })

并执行：

Short_outp<-Short_Parser_HMDB(location)
stopCluster(cl)

收到的错误：

> Short_outp<-Short_Parser_HMDB(location)
Error in checkForRemoteErrors(val) : 
  one node produced an error: invalid connection

基于这些链接，尝试实现并行：

但找不到invalid connection 错误

我正在使用 Windows 10 最新的 R 版本 4.0.2（不确定信息是否足够）

任何提示或想法将不胜感激

r - R中Windows 10的并行包

0 回答 0

Related

Reference