sqlite - 对 SQLite 的许多小写入

Question

我有一个应用程序，它一直在运行并接收一些消息（它们的速率从每秒几条到每小时一条不等）。每条消息都应该放入 SQLite 数据库中。最好的方法是什么？

在每条消息上打开和关闭数据库听起来并不好：如果每秒有几十个，它会非常慢。

另一方面，如果进程意外终止，打开数据库一次并仅写入它可能会导致数据丢失。

Answer 1

It sounds like whatever you do, you'll have to make a trade-off.

If safety is your top-most concern, then update the database on each message and take the speed hit.

If you want a compromise, then update the database write every so many messages. For instance, maintain a buffer and every 100th message, issue an update, wrapped in a transaction.

The transaction wrapping is important for two reasons. First, it maximizes speed. Second, it can help you recover from errors if you employ logging.

If you do the batch update above, you can add an additional level of safety by logging each message as it comes to a file. You will reset this log every time a database update is successfully issues. That way, if an update fails, you know it failed on the entire block (since you are using transactions) and your log will have the information that did not update. This will allow you to re-issue the update, or even see if there was a problem with the data that caused the failure. This of course assumes that keeping a log is cheaper than updating the database, which can be the case depending on how you are connecting.

Answer 2

如果您的最高速率是“每秒几次”，那么我认为打开和关闭数据库没有真正的问题。如果在服务器发生故障时立即记录数据至关重要，则尤其如此。

我们在报告产品中使用 SQLite，我们能够获得的最佳性能是一次以数千个块的形式记录行。我们的默认设置是 50k。这意味着我们的应用程序会等待，直到收集到 50k 行数据，然后将其作为一个事务提交。

Answer 3

有一个简单的算法可以根据消息速率调整应用程序的行为：

• 当你刚刚写了一条消息时，检查是否有任何新消息。
  • 如果是，也写下那条信息，然后重复。
• 只有当您用完立即可用的消息时，才提交事务并关闭数据库。

以这种方式，每条消息都会立即保存，除非消息率变得太高。

注意：关闭数据库不会增加数据的持久性（这就是事务提交的目的），它只会释放一点内存。