python - Pandas 过滤并将日期转换为 datetime64ns

Question

我试图找出一个问题，但到目前为止我找不到任何解决方案，希望您能提供帮助。我有一个 DataFrame，我想转换str为，datatime但有一些我想过滤掉的无效行。这里有两个例子：

Out[6]:
  #  name    date
  0  aa      2012-11-30T14:00:00+01:00
  1  bb      2012-12-01T08:16:00+01:00
  2  cc      2012-12-01T10:14:00+01:00
  3  ee      2012-12-01T11:05:00+01:00
  4  gg      2012-12-01T11:05:00+01:00

In [7]: df2
Out[7]:
  #  name    date
  0  aa      2012-11-30T14:00:00+01:00
  1  bb      2012-12-01T08:16:00+01:00
  2  cc      2012-12-01T10:14:00+01:00
  3  ee      2012-12-01T11:05:00+01:00
  4  ff      fsadfi2 2ih3ro
  5  gg      2012-12-01T11:05:00+01:00

In [11]: df.dtypes
Out[11]:
name    <class 'str'>
date    <class 'str'>
dtype: object

In [12]: df2.dtypes
Out[12]:
name    <class 'str'>
date    <class 'str'>
dtype: object

df我很好，它只有date列中的有效日期。但是df2有一些无效的行。让我们df首先看一下我可以转换为的以下行datetime：

df['pdate']=df.date.values.astype('datetime64[ns]')

效果很好：


In [16]: df
Out[16]:
  #  name    date                       pdate
  0  aa      2012-11-30T14:00:00+01:00  2012-11-30 13:00:00.000000000
  1  bb      2012-12-01T08:16:00+01:00  2012-12-01 07:16:00.000000000
  2  cc      2012-12-01T10:14:00+01:00  2012-12-01 09:14:00.000000000
  3  ee      2012-12-01T11:05:00+01:00  2012-12-01 10:05:00.000000000
  4  gg      2012-12-01T11:05:00+01:00  2012-12-01 10:05:00.000000000

In [17]: df.dtypes
Out[17]:
name      <class 'str'>
date      <class 'str'>
pdate    datetime64[ns]
dtype: object

现在我尝试用一个非常简单的str.contains::

In [18]: df2_filtered=df2[df2['date'].str.contains(':00')]

In [19]: df2_filtered
Out[19]:
  #  name    date
  0  aa      2012-11-30T14:00:00+01:00
  1  bb      2012-12-01T08:16:00+01:00
  2  cc      2012-12-01T10:14:00+01:00
  3  ee      2012-12-01T11:05:00+01:00
  4  gg      2012-12-01T11:05:00+01:00

In [20]: df2_filtered.dtypes
Out[20]:
name    <class 'str'>
date    <class 'str'>
dtype: object

它只有5 Rows. 现在我尝试转换并收到一条很好的错误消息：

In [21]: df2_filtered['pdate']=df2_filtered.date.values.astype('datetime64[ns]')
    ...:
/usr/local/bin/ipython:1: DeprecationWarning: parsing timezone aware datetimes is deprecated; this will raise an error in the future
  #!/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-563087d6f949> in <module>
----> 1 df2_filtered['pdate']=df2_filtered.date.values.astype('datetime64[ns]')

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py in __setitem__(self, name, value)
   4370         if isinstance(name, six.string_types):
   4371             if isinstance(value, (np.ndarray, Column)):
-> 4372                 self.add_column(name, value)
   4373             else:
   4374                 self.add_virtual_column(name, value)

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py in add_column(self, name, data, dtype)
   5743         #     self._length_original = len(data)
   5744         #     self._index_end = self._length_unfiltered
-> 5745         super(DataFrameArrays, self).add_column(name, data, dtype=dtype)
   5746         self._length_unfiltered = int(round(self._length_original * self._active_fraction))
   5747         # self.set_active_fraction(self._active_fraction)

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py in add_column(self, name, f_or_array, dtype)
   2872                     # give a better warning to avoid confusion
   2873                     if len(self) == len(ar):
-> 2874                         raise ValueError("Array is of length %s, while the length of the DataFrame is %s due to the filtering, the (unfiltered) length is %s." % (len(ar), len(self), self.length_unfiltered()))
   2875                 raise ValueError("array is of length %s, while the length of the DataFrame is %s" % (len(ar), self.length_original()))
   2876             # assert self.length_unfiltered() == len(data), "columns should be of equal length, length should be %d, while it is %d" % ( self.length_unfiltered(), len(data))

ValueError: Array is of length 5, while the length of the DataFrame is 5 due to the filtering, the (unfiltered) length is 6.

说：ValueError：数组的长度是5，而DataFrame的长度是5，由于过滤，（未过滤的）长度是6。

但据我了解，df2_filtered我只有 5 行。我不知道为什么df2.

基本上我的问题是如何过滤掉不必要的数据并将列转换为日期时间？

更新

基于Maarten Breddels我尝试使用：

df2_filtered['pdate']=df2_filtered.date.astype('datetime64[ns]')

这似乎有效，但是当我尝试使用时，df2_filtered我得到以下信息。

In [57]: df2_filtered
Out[57]: ERROR:MainThread:vaex:error evaluating: pdate at rows 0-5
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
    result = self[expression]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 141, in __getitem__
    raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: 'Unknown variables or column: "astype(date, \'datetime64[ns]\')"'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 3467, in table_part
    values[name] = df.evaluate(name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5038, in evaluate
    dtype = dtypes[expression] = self.dtype(expression, internal=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 2005, in dtype
    data = self.evaluate(expression, 0, 1, filtered=False, internal=True, parallel=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5143, in evaluate
    value = scope.evaluate(expression)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
    result = self[expression]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 136, in __getitem__
    self.values[variable] = self.evaluate(expression)  # , out=self.buffers[variable])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 100, in evaluate
    result = eval(expression, expression_namespace, self)
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/functions.py", line 2106, in _astype
    return x.astype(dtype)
AttributeError: 'ColumnStringArrow' object has no attribute 'astype'
ERROR:MainThread:vaex:error evaluating: pdate at rows 0-5
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
    result = self[expression]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 141, in __getitem__
    raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: 'Unknown variables or column: "astype(date, \'datetime64[ns]\')"'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 3467, in table_part
    values[name] = df.evaluate(name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5038, in evaluate
    dtype = dtypes[expression] = self.dtype(expression, internal=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 2005, in dtype
    data = self.evaluate(expression, 0, 1, filtered=False, internal=True, parallel=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5143, in evaluate
    value = scope.evaluate(expression)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
    result = self[expression]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 136, in __getitem__
    self.values[variable] = self.evaluate(expression)  # , out=self.buffers[variable])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 100, in evaluate
    result = eval(expression, expression_namespace, self)
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/functions.py", line 2106, in _astype
    return x.astype(dtype)
AttributeError: 'ColumnStringArrow' object has no attribute 'astype'

  #  name    date                       pdate
  0  aa      2012-11-30T14:00:00+01:00  error
  1  bb      2012-12-01T08:16:00+01:00  error
  2  cc      2012-12-01T10:14:00+01:00  error
  3  ee      2012-12-01T11:05:00+01:00  error
  4  gg      2012-12-01T11:05:00+01:00  error

score 1 · Accepted Answer

vaex 主要作者在这里。好问题，什么 halumpago 是正确的，df2_filtered 在内部仍然是 6 行长。您尝试执行以下操作：

# Adds a numpy arrays to the dataframe
df2_filtered['pdate'] = df2_filtered.date.values.astype('datetime64[ns]')

是将长度为 5 的数组（这就是.values返回值）添加到内部有一堆长度为 6 的数组的 DataFrame 中。这就是错误消息试图传达的内容。如果错误不清楚，或者其他原因，请通过https://github.com/vaexio/vaex/issues告诉我们。理论上我们可以支持这一点，但我们必须创建一个长度为 6 的数组，并将数据复制到其中。当您使用 10 亿行时，这并不理想。

在 vaex 中，您最好不要使用底层数组（好吧，有时您需要它们，这就是我们支持.values和朋友的原因）。相反，表达式系统试图尽可能接近地模仿 Pandas 系列/numpy 数组。如果删除.values，您将向 DataFrame 添加一个新的虚拟列而不是数组：

# Adds a virtual column (backed by an expression) to the dataframe
# at zero memory cost
df2_filtered['pdate']=df2_filtered.date.astype('datetime64[ns]')

Vaex 将愉快地存储该表达式，并仅针对您尚未过滤掉的行对其进行评估。请注意，如果您在此新列之前和之后打印出 DataFrame 的大小（以字节为单位），则它的内存使用量是相同的：

print(df2.nbytes)
648

注意：我们忽略了虚拟列簿记的内存使用情况，这当然是几个字节，但是当您使用 100 GB 的数据时可以忽略不计）。

只是为了好玩，为了打破 vaex DataFrame 与 Pandas 数据帧相同的错觉，您实际上可以删除您的过滤器：

print(df2_filtered.drop_filter())
  #  name    date                       pdate
  0  aa      2012-11-30T14:00:00+01:00  2012-11-30 13:00:00.000000000
  1  bb      2012-12-01T08:16:00+01:00  2012-12-01 07:16:00.000000000
  2  cc      2012-12-01T10:14:00+01:00  2012-12-01 09:14:00.000000000
  3  ee      2012-12-01T11:05:00+01:00  2012-12-01 10:05:00.000000000
  4  ff      fsadfi2 2ih3ro             NaT
  5  gg      2012-12-01T11:05:00+01:00  2012-12-01 10:05:00.000000000

所以数据实际上从未真正消失过，我们只是对你隐藏它:)。

这允许您使用非常大的 Vaex DataFrames 并添加许多新列，进行大量过滤，并且仍然没有 MemoryError。我们一直引用原始数据，只保留过滤器的掩码和计算的表达式。

score 0 · Accepted Answer

不幸的是，我没有完整的答案，但我可能对您问题的这一部分有所了解：

我不知道为什么 df2 中有多少行很重要。

这很重要，因为据我了解，vaex通过存储定义该列的操作来构造新列（看起来他们称它们为“虚拟列”）。相比之下，Pandas 通过计算、存储和复制新列的实际值来构造新列。

Pandas对内存的要求非常高，但在移动数据时为您提供了很大的灵活性。虚拟列不会有这种灵活性，但您的程序可能会更好地使用内存。

看看您正在执行过滤的线路：

df2_filtered=df2[df2['date'].str.contains(':00')]

与 Pandas 不同，df2_filtered它不是内存中的实际“事物”。相反，它是对原始数据框的引用df2加上一些额外的逻辑，告诉 vaex 忽略不以“：00”结尾的任何内容。

因此，当您运行时：

df2_filtered['pdate']=df2_filtered.date.values.astype('datetime64[ns]')

您实际上是在要求vaex在其中创建一个新列，df2因为df2_filtered实际上只是对df2. vaex不知道如何处理被过滤掉的行df2，所以它会抛出你看到的错误。

因此，要进行转换，您需要一些方法来填充df2. 不幸的是，我还不够熟悉，vaex无法对此提供帮助。我尝试了@datanovice 的方法，但vaex抱怨它不知道如何处理NaT值。

score 0 · Accepted Answer

IIUC，pd.to_datetime它允许您使用某些关键字参数将列转换为 DateTime。在这种情况下，您需要errors='coerce'

print(df)

   name                       date
0   aa  2012-11-30T14:00:00+01:00
1   bb  2012-12-01T08:16:00+01:00
2   cc  2012-12-01T10:14:00+01:00
3   ee  2012-12-01T11:05:00+01:00
4   ff              fsadfi22ih3ro
5   gg  2012-12-01T11:05:00+01:00

df['date'] = pd.to_datetime(df['date'],errors='coerce')
print(df)
      name                      date
0   aa 2012-11-30 14:00:00+01:00
1   bb 2012-12-01 08:16:00+01:00
2   cc 2012-12-01 10:14:00+01:00
3   ee 2012-12-01 11:05:00+01:00
4   ff                       NaT
5   gg 2012-12-01 11:05:00+01:00

现在只需删除带有.dropna()while 子集日期列的行。

df.dropna(subset=['date'])
print(df)
    name                      date
0   aa 2012-11-30 14:00:00+01:00
1   bb 2012-12-01 08:16:00+01:00
2   cc 2012-12-01 10:14:00+01:00
3   ee 2012-12-01 11:05:00+01:00
5   gg 2012-12-01 11:05:00+01:00

print(df.dtypes)
name                                  object
date    datetime64[ns, pytz.FixedOffset(60)]
dtype: object

python - Pandas 过滤并将日期转换为 datetime64ns

3 回答 3

Related

Reference