I'm trying to transition from Pandas
into Xarray
for N-Dimensional DataArrays
to expand my repertoire.
Realistically, I'm going to have a bunch of different pd.DataFrames
(in this case row=month, col=attribute) along a particular axis (patients in the mock example below) that I would like to merge (w/o using panels or multindex :), thank you). I want to convert them to xr.DataArrays
so I can build dimensions upon them. I made a mock dataset to give a gist of what I'm talking about.
For this dataset I made up, imagine 100 patients, 12 months, 10000 attributes, 3 replicates (per attribute)
which would be a typical 4D dataset. Basically, I'm condensing the 3 replicates per attribute
by the mean
so I end up with a 2D pd.DataFrame
(row=months, col=attributes) this DataFrame is the value in my dictionary and the patient it came from is the key (i.e. (patient_x : DataFrame_X) )
I'm also going to include a round about way I did it with np.ndarray
placeholder but it would be really convenient if I could generate a N-dimensional DataArray from a dictionary whose key was patient_x and the value was a DataFrame_X
How can I create a N-Dimensional DataArray
using Xarray
from a dictionary of Pandas DataFrames
?
import xarray as xr
import numpy as np
import pandas as pd
np.random.seed(1618033)
#Set dimensions
a,b,c,d = 100,12,10000,3 #100 patients, 12 months, 10000 attributes, 3 replicates
#Create labels
patients = ["patient_%d" % i for i in range(a)]
months = [j for j in range(b)]
attributes = ["attr_%d" % k for k in range(c)]
replicates = [l for l in range(d)]
coords = [patients,months,attributes]
dims = ["Patients","Months","Attributes"]
#Dict of DataFrames
D_patient_DF = dict()
for i, patient in enumerate(patients):
A_placeholder = np.zeros((b,c))
for j, month in enumerate(months):
#Attribute x Replicates
A_attrReplicates = np.random.random((c,d))
#Collapse into 1D Vector
V_attrExp = A_attrReplicates.mean(axis=1)
#Fill array with row
A_placeholder[j,:] = V_attrExp
#Assign dataframe for every patient
DF_data = pd.DataFrame(A_placeholder, index = months, columns = attributes)
D_patient_DF[patient] = DF_data
xr.DataArray(D_patient_DF).dims
#() its empty
D_patient_DF
#{'patient_0': attr_0 attr_1 attr_2 attr_3 attr_4 attr_5 attr_6 \
# 0 0.445446 0.422018 0.343454 0.140700 0.567435 0.362194 0.563799
# 1 0.440010 0.548535 0.810903 0.482867 0.469542 0.591939 0.579344
# 2 0.645719 0.450773 0.386939 0.418496 0.508290 0.431033 0.622270
# 3 0.555855 0.633393 0.555197 0.556342 0.489865 0.204200 0.823043
# 4 0.916768 0.590534 0.597989 0.592359 0.484624 0.478347 0.507789
# 5 0.847069 0.634923 0.591008 0.249107 0.655182 0.394640 0.579700
# 6 0.700385 0.505331 0.377745 0.651936 0.334216 0.489728 0.282544
# 7 0.777810 0.423889 0.414316 0.389318 0.565144 0.394320 0.511034
# 8 0.440633 0.069643 0.675037 0.365963 0.647660 0.520047 0.539253
# 9 0.333213 0.328315 0.662203 0.594030 0.790758 0.754032 0.602375
# 10 0.470330 0.419496 0.171292 0.677439 0.683759 0.646363 0.465788
# 11 0.758556 0.674664 0.801860 0.612087 0.567770 0.801514 0.179939