azure - Azure 文件共享同步

Question

我正在寻找在 Azure 存储帐户中跨两个文件共享同步文件的最佳方式。到目前为止，看起来 AZCopy 是最好的选择，但在特定情况下或过滤器的情况下它将不起作用。

例如：

• 我想复制在某个日期（上周、一个月等）之后添加的文件。
• 跳过一些文件夹/子文件夹

使用 Powershell 也是一种选择，但同步大量文件可能会很慢。

您是否在 AZCopy 之上有一些脚本或包装器可以帮助覆盖按文件夹和文件类型进行过滤？

谢谢伊霍尔

Answer 1

据我所知，Azure 本身无法保持两个 Azure 文件共享同步。

作为一种解决方法，您需要在 Azure 上构建一个 VM 并将这两个共享挂载在上面。然后你只需要找到一个可以保持两个文件夹同步的解决方案。在 Windows 和 Linux 上都有几种解决方案。

Answer 2

这是同步一些文件夹的脚本：

function Copy-AzureFileStorageWithPS{
<#
.SYNOPSIS
Copy a file share from a from one storage account to another using PowerShell

.DESCRIPTION
This function will copy a file share or specific folder inside of file share from one storage account to another

.PARAMETER SourceStorageAccountName

.PARAMETER DestStorageAccountName

.PARAMETER SourceResourceGroupName

.PARAMETER DestResourceGroupName

.PARAMETER FoldersToSkip

.PARAMETER FoldersToCopy

.EXAMPLE
. .\Copy-AzureFileStorageWithPS -SourceStorageAccountName  "some" -DestStorageAccountName  "other" -SourceResourceGroupName  "RG" -DestResourceGroupName  "OtherRg" 
#>

[CmdletBinding()]
param(
    [Parameter (Mandatory = $true)]
    [string] $SourceStorageAccountName,

    [Parameter (Mandatory = $true)]
    [string] $DestStorageAccountName,

    [Parameter (Mandatory = $true)]
    [string] $SourceResourceGroupName,

    [Parameter (Mandatory = $true)]
    [string] $DestResourceGroupName,

    [Parameter (Mandatory = $true)]
    [string] $FoldersToSkip = "",

    [Parameter (Mandatory = $true)]
    [string] $FoldersToCopy = ""
)

$exclusions = $FoldersToSkip.ToLower().Split(',').Trim()
$copyList = $FoldersToCopy.ToLower().Split(',').Trim()

$SourceStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $SourceStorageAccountName -ResourceGroupName $SourceResourceGroupName)[0].Value
$DestStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $DestStorageAccountName -ResourceGroupName $DestResourceGroupName)[0].Value

$SourceContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccountName -StorageAccountKey $SourceStorageAccountKey
$DestContext = New-AzureStorageContext -StorageAccountName $DestStorageAccountName -StorageAccountKey $DestStorageAccountKey

#Get all shares from Source SA
$shares  = Get-AzureStorageshare -Context $SourceContext

foreach($share in $shares){
    $destShare = Get-AzureStorageshare -Context $DestContext -Name $share.Name -ErrorAction SilentlyContinue

    if(-Not $destShare)
    {
        $destShare = New-AzureStorageShare $share.Name -Context $DestContext            
        Write-Host "Successfully created share $($share.Name) in Account $($DestContext.StorageAccountName)"
    }

    $shareContent = Get-AzureStorageFile -Share $share     

    foreach($content in $shareContent)
    {
         if(-Not $exclusions.Contains($content.Name.ToLower())){
            if($copyList.Length -ne 0){
                if($copyList.Contains($content.Name.ToLower())){
                    Write-Output "Starting copy of $($content.Name) from $($content.Uri.AbsoluteUri) to $($destShare.Uri.AbsoluteUri+"/"+$content.Name)"
                    Copy-AonAzureFolders -SourceShare $share -SourceFolder $content -DestShare $destShare -SourceContext $SourceContext -DestContext $DestContext
                }
            }
            else
            {
                Write-Output "Starting copy of $($content.Name) from $($content.Uri.AbsoluteUri) to $($destShare.Uri.AbsoluteUri+"/"+$content.Name)"
                Copy-AonAzureFolders -SourceShare $share -SourceFolder $content -DestShare $destShare -SourceContext $SourceContext -DestContext $DestContext
            }
        }
        else{
                Write-Output "Excluding folder $($content.Name) from copying to $($destShare.Name)"
        }
    }
}
}


function Copy-AonAzureFolders
{
[CmdletBinding()]
param(
    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Storage.File.CloudFileShare] $SourceShare,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Storage.File.CloudFileDirectory] $SourceFolder,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Storage.File.CloudFileShare] $DestShare,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Commands.Common.Storage.AzureStorageContext] $SourceContext,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Commands.Common.Storage.AzureStorageContext] $DestContext
)
$pathWithoutShare = $($SourceFolder.Uri.LocalPath).Replace("/$($SourceShare.Name)","")

$shareFolders = Get-AzureStorageFile -Share $SourceShare -Path $pathWithoutShare | Get-AzureStorageFile | Where-Object {$_.GetType().Name -eq "CloudFileDirectory"}

$folderExist = Get-AzureStorageFile -Share $DestShare -Path $pathWithoutShare -ErrorAction SilentlyContinue
if(-Not $folderExist){
    $newFolder = New-AzureStorageDirectory -Share $DestShare -Path $pathWithoutShare
}

foreach($folder in $shareFolders)
{
    #Recursive call    
    Copy-AonAzureFolders -SourceShare $SourceShare -SourceFolder $folder -DestShare $DestShare -SourceContext $SourceContext -DestContext $DestContext       
}

$shareFiles = Get-AzureStorageFile -Share $SourceShare -Path $($pathWithoutShare+"/") | Get-AzureStorageFile | Where-Object {$_.GetType().Name -eq "CloudFile"}
foreach($file in $shareFiles)
{
    $fullFilePath = $pathWithoutShare + "/" + $($file.Name)

    Write-Host "-SrcShareName $($SourceShare.Name) -SrcFilePath $fullFilePath -DestShareName $($DestShare.Name) -DestFilePath $fullFilePath"
    # copy a file to the new directory
    $copyfile = Start-AzureStorageFileCopy -SrcShareName $($SourceShare.Name) -SrcFilePath $fullFilePath `
                                           -DestShareName $($DestShare.Name) -DestFilePath $fullFilePath -Context $SourceContext -DestContext $DestContext -Force

    $status = $copyfile | Get-AzureStorageFileCopyState
    while ($status.Status -eq "Pending")
    {
        $status = $copyfile | Get-AzureStorageFileCopyState
        Start-Sleep -Milliseconds 150
    }
    $copyfile
}    
}

希望对您有所帮助，但速度很慢

Answer 3

一个用 Python 编写的实用程序，但可以使用 Docker 启动：

https://github.com/dokkur/azure-fileshare-sync

跑：

AZURE_STORAGE_ACCOUNT=account\
AZURE_STORAGE_ACCESS_KEY=key\
python -m src.azure_sync ./local/path sharename[/remote/path]