-1

我想将下面的 JSON 数据转换为 avro 格式，我使用下面的代码片段将 JSON 数据写入 avro 格式但收到错误。如果有人可以提供帮助，那就太好了。

from fastavro import writer, reader, schema
from rec_avro import to_rec_avro_destructive, from_rec_avro_destructive, rec_avro_schema

def getweatherdata():
    url = 'https://api.openweathermap.org/data/2.5/onecall?lat=33.441792&lon=-94.037689&exclude=hourly,daily&appid=' + apikey
    response = requests.get(url)
    data = response.text
    return data
 
def turntoavro():
    avro_objects = (to_rec_avro_destructive(rec) for rec in getweatherdata())
    with open('json_in_avro.avro', 'wb') as f_out:
        writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)



turntoavro()


    Error details:
    
      File "fastavro/_write.pyx", line 269, in fastavro._write.write_record
    TypeError: Expected dict, got str
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "datalake.py", line 30, in <module>
        turntoavro()
      File "datalake.py", line 26, in turntoavro
        writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)
      File "fastavro/_write.pyx", line 652, in fastavro._write.writer
      File "fastavro/_write.pyx", line 605, in fastavro._write.Writer.write
      File "fastavro/_write.pyx", line 341, in fastavro._write.write_data
      File "fastavro/_write.pyx", line 278, in fastavro._write.write_record
    AttributeError: 'str' object has no attribute 'get'

样本数据：

    {
      "lat": 33.44,
      "lon": -94.04,
      "timezone": "America/Chicago",
      "timezone_offset": -18000

   }
4

2 回答 2

0

要检索对您发出的请求的响应，您使用response.text它以字符串而不是 JSON 格式返回响应。您必须改为使用response.json()JSON 格式：

import json    
def getweatherdata():
    url = 'https://api.openweathermap.org/data/2.5/onecall?lat=33.441792&lon=-94.037689&exclude=hourly,daily&appid=' + apikey
    response = requests.get(url)
    data = response.json()
    return data
     
def turntoavro():
    avro_objects = (to_rec_avro_destructive(rec) for rec in getweatherdata())
    with open('json_in_avro.avro', 'wb') as f_out:
        writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)
    
    
    
turntoavro()
于 2020-08-08T01:09:20.063 回答
0

正如其中一个答案中提到的，您可能想要使用response.json()而不是response.text让您返回一个实际的 JSON 字典。

然而，另一个问题是它getweatherdata()返回一个字典，所以当你avro_objects = (to_rec_avro_destructive(rec) for rec in getweatherdata())迭代该字典中的键时。相反，你应该做avro_objects = [to_rec_avro_destructive(getweatherdata())]

我相信这段代码应该适合你：

from fastavro import writer, reader, schema
from rec_avro import to_rec_avro_destructive, from_rec_avro_destructive, rec_avro_schema

def getweatherdata():
    url = 'https://api.openweathermap.org/data/2.5/onecall?lat=33.441792&lon=-94.037689&exclude=hourly,daily&appid=' + apikey
    response = requests.get(url)
    data = response.json()
    return data
 
def turntoavro():
    avro_objects = [to_rec_avro_destructive(getweatherdata())]
    with open('json_in_avro.avro', 'wb') as f_out:
        writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)

turntoavro()
于 2020-08-10T14:46:32.923 回答