我不确定这是否是实际的正确答案，但我现在已经采用了这个解决方案。
AzCopy 更快一些，但由于它是可执行的，我无法在自动化中使用它。
我编写了自己的运行手册（可以修改为工作流），它实现了以下 AzCopy 命令
AzCopy /Source:$sourceUri /Dest:$destUri /SourceKey:$sourceStorageKey /DestKey:$destStorageAccountKey /S /XO /Y
- 查看列表 blob，我们只能通过 blob 前缀来拟合 blob。所以我不能拉出按修改日期过滤的 blob。这让我不得不拉出整个 blob 列表。
- 我每次使用 ContinuationToken 从源和目标 Get-AzureStorageBlob 提取 20,000 个 blob
- 循环提取 20,000 个源 blob，查看它们是否在目标中不存在或在源中已被修改
- 如果 2 为真，那么我将这些 blob 写入目标
- 通过 700 万个 blob 大约需要 3-4 小时。任务将根据要写入目标的 blob 数量而延长。
一段代码
#loop throught the source container blobs,
# and copy the blob to destination that are not already there
$MaxReturn = 20000
$Total = 0
$Token = $null
$FilesTransferred = 0;
$FilesTransferSuccess = 0;
$FilesTransferFail = 0;
$sw = [Diagnostics.Stopwatch]::StartNew();
DO
{
$SrcBlobs = Get-AzureStorageBlob -Context $sourceContext -Container $container -MaxCount $MaxReturn -ContinuationToken $Token |
Select-Object -Property Name, LastModified, ContinuationToken
$DestBlobsHash = @{}
Get-AzureStorageBlob -Context $destContext -Container $container -MaxCount $MaxReturn -ContinuationToken $Token |
Select-Object -Property Name, LastModified, ContinuationToken |
ForEach { $DestBlobsHash[$_.Name] = $_.LastModified.UtcDateTime }
$Total += $SrcBlobs.Count
if($SrcBlobs.Length -le 0) {
Break;
}
$Token = $SrcBlobs[$SrcBlobs.Count -1].ContinuationToken;
ForEach ($SrcBlob in $SrcBlobs){
# search in destination blobs for the source blob and unmodified, if found copy it
$CopyThisBlob = $false
if(!$DestBlobsHash.count -ne 0){
$CopyThisBlob = $true
} elseif(!$DestBlobsHash.ContainsKey($SrcBlob.Name)){
$CopyThisBlob = $true
} elseif($SrcBlob.LastModified.UtcDateTime -gt $DestBlobsHash.Item($SrcBlob.Name)){
$CopyThisBlob = $true
}
if($CopyThisBlob){
#Start copying the blobs to container
$blobToCopy = $SrcBlob.Name
"Copying blob: $blobToCopy to destination"
$FilesTransferred++
try {
$c = Start-AzureStorageBlobCopy -SrcContainer $container -SrcBlob $blobToCopy -DestContainer $container -DestBlob $blobToCopy -SrcContext $sourceContext -DestContext $destContext -Force
$FilesTransferSuccess++
} catch {
Write-Error "$blobToCopy transfer failed"
$FilesTransferFail++
}
}
}
}
While ($Token -ne $Null)
$sw.Stop()
"Total blobs in container $container : $Total"
"Total files transferred: $FilesTransferred"
"Transfer successfully: $FilesTransferSuccess"
"Transfer failed: $FilesTransferFail"
"Elapsed time: $($sw.Elapsed) `n"