3

我正在尝试检查我的 appengine 备份文件，以确定何时发生数据损坏。我使用 gsutil 找到并下载文件：

gsutil ls -l gs://my_backup/ > my_backup.txt
gsutil cp gs://my_backup/LongAlphaString.Mymodel.backup_info file://1.backup_info

然后我创建了一个小型 python 程序，尝试读取文件并使用 appengine 库对其进行解析。

#!/usr/bin/python

APPENGINE_PATH='/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/'
ADDITIONAL_LIBS = [
'lib/yaml/lib'
]
import sys
sys.path.append(APPENGINE_PATH)
for l in ADDITIONAL_LIBS:
  sys.path.append(APPENGINE_PATH+l)

import logging
from google.appengine.api.files import records
import cStringIO

def parse_backup_info_file(content):
  """Returns entities iterator from a backup_info file content."""
  reader = records.RecordsReader(cStringIO.StringIO(content))
  version = reader.read()
  if version != '1':
    raise IOError('Unsupported version')
  return (datastore.Entity.FromPb(record) for record in reader)


INPUT_FILE_NAME='1.backup_info'

f=open(INPUT_FILE_NAME, 'rb')
f.seek(0)
content=f.read()
records = parse_backup_info_file(content)
for r in records:
  logging.info(r)

f.close()

的代码parse_backup_info_file是从 backup_handler.py复制的

当我运行程序时，我得到以下输出：

./view_record.py 
Traceback (most recent call last):
  File "./view_record.py", line 30, in <module>
    records = parse_backup_info_file(content)
  File "./view_record.py", line 19, in parse_backup_info_file
    version = reader.read()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/records.py", line 335, in read
    (chunk, record_type) = self.__try_read_record()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/records.py", line 307, in __try_read_record
    (length, len(data)))
EOFError: Not enough data read. Expected: 24898 but got 2112

我尝试了六个不同的 backup_info 文件，它们都显示相同的错误（具有不同的数字。） 我注意到它们都有相同的预期长度： 我在制作时正在查看同一模型的不同版本那个观察，当我查看其他模块的备份文件时，这不是真的。

EOFError: Not enough data read. Expected: 24932 but got 911
EOFError: Not enough data read. Expected: 25409 but got 2220

我的方法有什么明显的问题吗？

我猜另一个选项是appengine 备份实用程序没有创建有效的备份文件。您可以提出的任何其他建议都将非常受欢迎。提前致谢

4

1 回答 1

2

运行 AppEngine 数据存储区备份时会创建多个元数据文件：

LongAlphaString.backup_info创建一次。这包含有关在数据存储备份中创建的所有实体类型和备份文件的元数据。

LongAlphaString.[EntityType].backup_info为每个实体类型创建一次。这包含有关为 [EntityType] 创建的特定备份文件的元数据以及 [EntityType] 的架构信息。

您的代码用于查询 LongAlphaString.backup_info 的文件内容，但您似乎正在尝试查询 LongAlphaString.[EntityType].backup_info 的文件内容。这是一个脚本，它将以人类可读的格式打印每种文件类型的内容：

import cStringIO
import os
import sys

sys.path.append('/usr/local/google_appengine')
from google.appengine.api import datastore
from google.appengine.api.files import records
from google.appengine.ext.datastore_admin import backup_pb2

ALL_BACKUP_INFO = 'long_string.backup_info'
ENTITY_KINDS = ['long_string.entity_kind.backup_info']


def parse_backup_info_file(content):
    """Returns entities iterator from a backup_info file content."""
    reader = records.RecordsReader(cStringIO.StringIO(content))
    version = reader.read()
    if version != '1':
        raise IOError('Unsupported version')
    return (datastore.Entity.FromPb(record) for record in reader)


print "*****" + ALL_BACKUP_INFO + "*****"
with open(ALL_BACKUP_INFO, 'r') as myfile:
    parsed = parse_backup_info_file(myfile.read())
    for record in parsed:
        print record

for entity_kind in ENTITY_KINDS:
    print os.linesep + "*****" + entity_kind + "*****"
    with open(entity_kind, 'r') as myfile:
        backup = backup_pb2.Backup()
        backup.ParseFromString(myfile.read())
        print backup
于 2014-10-06T21:19:50.823 回答