1

我使用下面的脚本创建连接器，但在 S3 中，我看到 /year=2015/month=12/day=07/hour=15/ 的分区格式。有没有办法实现 'dt'=YYYY-MM-dd/'hour'=HH/ 格式的分区？

curl -X POST \
    -H "Content-Type: application/json" \
    --data '{
    "name": "content.logging.test",
    "config": {
        "topics": "content.logging",
        "connector.class": "io.confluent.connect.s3.S3SinkConnector",
        "storage.class": "io.confluent.connect.s3.storage.S3Storage",
        "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
        "s3.region": "ap-northeast-1",
        "s3.bucket.name": "kafka-connect-test",
        "locale": "en-US",
        "timezone": "UTC",
        "tasks.max": 1,
        "flush.size": 10,
        "partitioner.class": "io.confluent.connect.storage.partitioner.HourlyPartitioner",
        "partition.duration.ms": 3600000,
        "path.format": "'dt'=YYYY-MM-dd/'hour'=HH/"
        }
    }' http://$CONNECT_REST_ADVERTISED_HOST_NAME:8083/connectors
4

1 回答 1

2

如果要使用格式，则应使用 TimeBasedPartitioner

https://docs.confluent.io/kafka-connect-s3-sink/current/index.html#partitioning-records-into-s3-objects

于 2021-01-05T15:37:51.757 回答