python - 抓取时如何检查“未更改”文件？

Question

我正在尝试构建一个索引 .deb 文件并解析内部结构并提取一些元信息的爬虫。

现在，我想尽可能便宜地做到这一点 - 但也尽可能频繁（可能每 5-6 小时一次）。所以我想确保我不下载未更改的文件。

也许我在这里遗漏了一些东西，但是如何在不下载文件的情况下检查文件是否未更改？

Answer 1

Download the meta data (directory listing) which will give you the timestamp, file size etc. Use that to determine if the file has changed. You can also check MD5 hash if available. Once you are sure the file has changed, you can mark it for download.

A similar strategy is used by web browsers to determine if an asset has changed. Even email clients use similar strategy, especially on hand held devices. They first download only the mail header. If user wants to read the mail, then body is downloaded.