0

I'm going to use AWS Database Migration Service (DMS) with AWS MSK(Kafka).

I'd like to send all changes within the same transaction into the same partition of Kafka topic - in order to guarantee correct message order(reference integrity)

For this purpose I'm going to enable the following property:

IncludeTransactionDetails – Provides detailed transaction information from the source database. This information includes a commit timestamp, a log position, and values for transaction_id, previous_transaction_id, and transaction_record_id (the record offset within a transaction). The default is false. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kafka.html

Also, as I may see from the same documentation:

AWS DMS supports the following two forms for partition keys:

1. SchemaName.TableName: A combination of the schema and table name.

2. ${AttributeName}: The value of one of the fields in the JSON, or the primary key of the table in the source database.

I have a question - in case of 'IncludeTransactionDetails = true', will I be able to use 'transaction_id' from event JSON as partition key for MSK(Kafka) migration topic?

4

1 回答 1

0

文档说，您可以定义分区键来对数据进行分组“您还可以为每个表定义一个分区键，Apache Kafka 使用它来将数据分组到其分区中”

于 2020-08-23T08:24:34.330 回答