1

我的 DataFrame 在 Continent_Name 列中有空值，我希望将其替换为同一列的模式。

+-----------------+-----------------------+-------------------------+-----------------------+-------------------------------+--------------+
|     Country_Name|Number_of_Beer_Servings|Number_of_Spirit_Servings|Number_of_Wine_servings|Pure_alcohol_Consumption_litres|Continent_Name|
+-----------------+-----------------------+-------------------------+-----------------------+-------------------------------+--------------+
|      Afghanistan|                      0|                        0|                      0|                            0.0|            AS|
|          Albania|                     89|                      132|                     54|                            4.9|            EU|
|          Algeria|                     25|                        0|                     14|                            0.7|            AF|
|          Andorra|                    245|                      138|                    312|                           12.4|            EU|
|           Angola|                    217|                       57|                     45|                            5.9|            AF|
|Antigua & Barbuda|                    102|                      128|                     45|                            4.9|          null|
|        Argentina|                    193|                       25|                    221|                            8.3|            SA|
|          Armenia|                     21|                      179|                     11|                            3.8|            EU|
|        Australia|                    261|                       72|                    212|                           10.4|            OC|
|          Austria|                    279|                       75|                    191|                            9.7|            EU|
|       Azerbaijan|                     21|                       46|                      5|                            1.3|            EU|
|          Bahamas|                    122|                      176|                     51|                            6.3|          null|
|          Bahrain|                     42|                       63|                      7|                            2.0|            AS|
|       Bangladesh|                      0|                        0|                      0|                            0.0|            AS|
|         Barbados|                    143|                      173|                     36|                            6.3|          null|
|          Belarus|                    142|                      373|                     42|                           14.4|            EU|
|          Belgium|                    295|                       84|                    212|                           10.5|            EU|
|           Belize|                    263|                      114|                      8|                            6.8|          null|
|            Benin|                     34|                        4|                     13|                            1.1|            AF|
|           Bhutan|                     23|                        0|                      0|                            0.4|            AS|
+-----------------+-----------------------+-------------------------+-----------------------+-------------------------------+--------------+

仅显示前 20 行

我尝试了以下方式：

for column in df_copy['Continent_Name']:
    df_copy['Continent_Name'].fillna(df_copy['Continent_Name'].mode()[0], inplace=True)

出现的错误：

TypeError: Column is not iterable
4

1 回答 1

0

创建DataFrame以下。

df = spark.createDataFrame([('Afghanistan',0,0,0,0.0,'AS'),('Albania',89,132,54,4.9,'EU'),
                            ('Algeria',25,0,14,0.7,'AF'),('Andorra',245,138,312,12.4,'EU'),
                            ('Angola',217,57,45,5.9,'AF'),('Antigua&Barbuda',102,128,45,4.9,None),
                            ('Argentina',193,25,221,8.3,'SA'),('Armenia',21,179,11,3.8,'EU'),
                            ('Australia',261,72,212,10.4,'OC'),('Austria',279,75,191,9.7,'EU'),
                            ('Azerbaijan',21,46,5,1.3,'EU'),('Bahamas',122,176,51,6.3,None),
                            ('Bahrain',42,63,7,2.0,'AS'),('Bangladesh',0,0,0,0.0,'AS'),
                            ('Barbados',143,173,36,6.3,None),('Belarus',142,373,42,14.4,'EU'),
                            ('Belgium',295,84,212,10.5,'EU'),('Belize',263,114,8,6.8,None),
                            ('Benin',34,4,13,1.1,'AF'),('Bhutan',23,0,0,0.4,'AS')],
                            ['Country_Name','Number_of_Beer_Servings','Number_of_Spirit_Servings',
                             'Number_of_Wine_servings','Pure_alcohol_Consumption_litres',
                             'Continent_Name'])

由于我们打算找到Mode，我们需要寻找Continent_Name最常出现的值。

df1 = df.where(col('Continent_Name').isNotNull())

Resistering我们的 DataFrame 作为视图，并在其上应用 SQL 命令group by，然后 count Continent_Name。

df1.registerTempTable('table')
df2=spark.sql(
    'SELECT Continent_Name, COUNT(Continent_Name) AS count FROM table GROUP BY Continent_Name ORDER BY count desc'
)
df2.show()
+--------------+-----+
|Continent_Name|count|
+--------------+-----+
|            EU|    7|
|            AS|    4|
|            AF|    3|
|            SA|    1|
|            OC|    1|
+--------------+-----+

最后，返回first组的元素。

mode_value = df2.first()['Continent_Name']
print(mode_value)
     EU

mode_value获取后，填写使用.fillna()函数即可。

df = df.fillna({'Continent_Name':mode_value})
df.show()
+---------------+-----------------------+-------------------------+-----------------------+-------------------------------+--------------+
|   Country_Name|Number_of_Beer_Servings|Number_of_Spirit_Servings|Number_of_Wine_servings|Pure_alcohol_Consumption_litres|Continent_Name|
+---------------+-----------------------+-------------------------+-----------------------+-------------------------------+--------------+
|    Afghanistan|                      0|                        0|                      0|                            0.0|            AS|
|        Albania|                     89|                      132|                     54|                            4.9|            EU|
|        Algeria|                     25|                        0|                     14|                            0.7|            AF|
|        Andorra|                    245|                      138|                    312|                           12.4|            EU|
|         Angola|                    217|                       57|                     45|                            5.9|            AF|
|Antigua&Barbuda|                    102|                      128|                     45|                            4.9|            EU|
|      Argentina|                    193|                       25|                    221|                            8.3|            SA|
|        Armenia|                     21|                      179|                     11|                            3.8|            EU|
|      Australia|                    261|                       72|                    212|                           10.4|            OC|
|        Austria|                    279|                       75|                    191|                            9.7|            EU|
|     Azerbaijan|                     21|                       46|                      5|                            1.3|            EU|
|        Bahamas|                    122|                      176|                     51|                            6.3|            EU|
|        Bahrain|                     42|                       63|                      7|                            2.0|            AS|
|     Bangladesh|                      0|                        0|                      0|                            0.0|            AS|
|       Barbados|                    143|                      173|                     36|                            6.3|            EU|
|        Belarus|                    142|                      373|                     42|                           14.4|            EU|
|        Belgium|                    295|                       84|                    212|                           10.5|            EU|
|         Belize|                    263|                      114|                      8|                            6.8|            EU|
|          Benin|                     34|                        4|                     13|                            1.1|            AF|
|         Bhutan|                     23|                        0|                      0|                            0.4|            AS|
+---------------+-----------------------+-------------------------+-----------------------+-------------------------------+--------------+
于 2019-09-17T11:16:03.307 回答