我有以下示例 Spark 数据框
import pandas as pd
import pyspark
import pyspark.sql.functions as fn
from pyspark.sql.window import Window
raw_df = pd.DataFrame([
(1115, dt.datetime(2019,8,5,18,20), dt.datetime(2019,8,5,18,40)),
(484, dt.datetime(2019,8,5,18,30), dt.datetime(2019,8,9,18,40)),
(484, dt.datetime(2019,8,4,18,30), dt.datetime(2019,8,6,18,40)),
(484, dt.datetime(2019,8,2,18,30), dt.datetime(2019,8,3,18,40)),
(484, dt.datetime(2019,8,7,18,50), dt.datetime(2019,8,9,18,50)),
(1115, dt.datetime(2019,8,6,18,20), dt.datetime(2019,8,6,18,40)),
], columns=['server_id', 'start_time', 'end_time'])
df = spark.createDataFrame(raw_df)
这导致
+---------+-------------------+-------------------+
|server_id| start_time| end_time|
+---------+-------------------+-------------------+
| 1115|2019-08-05 18:20:00|2019-08-05 18:40:00|
| 484|2019-08-05 18:30:00|2019-08-09 18:40:00|
| 484|2019-08-04 18:30:00|2019-08-06 18:40:00|
| 484|2019-08-02 18:30:00|2019-08-03 18:40:00|
| 484|2019-08-07 18:50:00|2019-08-09 18:50:00|
| 1115|2019-08-06 18:20:00|2019-08-06 18:40:00|
+---------+-------------------+-------------------+
这表示每个服务器的使用日期范围。我想将其转换为不重叠日期的时间序列。
我想在不使用 UDF的情况下实现这一点。
这就是我现在正在做的,这是错误的
w = Window().orderBy(fn.lit('A'))
# Separate start/end date of usage into rows
df = (df.withColumn('start_end_time', fn.array('start_time', 'end_time'))
.withColumn('event_dt', fn.explode('start_end_time'))
.withColumn('row_num', fn.row_number().over(w)))
# Indicate start/end date of the usage (start date will always be on odd rows)
df = (df.withColumn('is_start', fn.when(fn.col('row_num')%2 == 0, 0).otherwise(1))
.select('server_id', 'event_dt', 'is_start'))
这使
+---------+-------------------+--------+
|server_id| event_dt|is_start|
+---------+-------------------+--------+
| 1115|2019-08-05 18:20:00| 1|
| 1115|2019-08-05 18:40:00| 0|
| 484|2019-08-05 18:30:00| 1|
| 484|2019-08-09 18:40:00| 0|
| 484|2019-08-04 18:30:00| 1|
| 484|2019-08-06 18:40:00| 0|
| 484|2019-08-02 18:30:00| 1|
| 484|2019-08-03 18:40:00| 0|
| 484|2019-08-07 18:50:00| 1|
| 484|2019-08-09 18:50:00| 0|
| 1115|2019-08-06 18:20:00| 1|
| 1115|2019-08-06 18:40:00| 0|
+---------+-------------------+--------+
但我想达到的最终结果如下:
+---------+-------------------+--------+
|server_id| event_dt|is_start|
+---------+-------------------+--------+
| 1115|2019-08-05 18:20:00| 1|
| 1115|2019-08-05 18:40:00| 0|
| 1115|2019-08-06 18:20:00| 1|
| 1115|2019-08-06 18:40:00| 0|
| 484|2019-08-02 18:30:00| 1|
| 484|2019-08-03 18:40:00| 0|
| 484|2019-08-04 18:30:00| 1|
| 484|2019-08-09 18:50:00| 0|
+---------+-------------------+--------+
所以对于server_id
484,我有实际的开始和结束日期,中间没有任何噪音。
您对如何在不使用 UDF 的情况下实现这一目标有任何建议吗?
谢谢