powershell - 大文件复制期间的进度（复制项目和写入进度？）

Question

有没有办法在 PowerShell 中复制一个非常大的文件（从一台服务器到另一台服务器）并显示其进度？

有一些解决方案可以将 Write-Progress 与循环结合使用来复制许多文件并显示进度。但是我似乎找不到任何可以显示单个文件进度的东西。

有什么想法吗？

Answer 1

仅使用 BitsTransfer 似乎是一个更好的解决方案，它似乎在大多数具有 PowerShell 2.0 或更高版本的 Windows 机器上都提供了 OOTB。

Import-Module BitsTransfer
Start-BitsTransfer -Source $Source -Destination $Destination -Description "Backup" -DisplayName "Backup"

Answer 2

I haven't heard about progress with Copy-Item. If you don't want to use any external tool, you can experiment with streams. The size of buffer varies, you may try different values (from 2kb to 64kb).

function Copy-File {
    param( [string]$from, [string]$to)
    $ffile = [io.file]::OpenRead($from)
    $tofile = [io.file]::OpenWrite($to)
    Write-Progress -Activity "Copying file" -status "$from -> $to" -PercentComplete 0
    try {
        [byte[]]$buff = new-object byte[] 4096
        [long]$total = [int]$count = 0
        do {
            $count = $ffile.Read($buff, 0, $buff.Length)
            $tofile.Write($buff, 0, $count)
            $total += $count
            if ($total % 1mb -eq 0) {
                Write-Progress -Activity "Copying file" -status "$from -> $to" `
                   -PercentComplete ([long]($total * 100 / $ffile.Length))
            }
        } while ($count -gt 0)
    }
    finally {
        $ffile.Dispose()
        $tofile.Dispose()
        Write-Progress -Activity "Copying file" -Status "Ready" -Completed
    }
}

Answer 3

或者，此选项使用本机 Windows 进度条...

$FOF_CREATEPROGRESSDLG = "&H0&"

$objShell = New-Object -ComObject "Shell.Application"

$objFolder = $objShell.NameSpace($DestLocation) 

$objFolder.CopyHere($srcFile, $FOF_CREATEPROGRESSDLG)

Answer 4

cmd /c copy /z src dest

不是纯 PowerShell，而是在 PowerShell 中可执行，它以百分比显示进度

Answer 5

我修改了 stej 中的代码（这很棒，正是我需要的！）以使用更大的缓冲区，[long] 用于更大的文件，并使用 System.Diagnostics.Stopwatch 类来跟踪经过的时间并估计剩余时间。

还增加了传输期间传输速率的报告，并输出总经过时间和总传输速率。

使用 4MB（4096*1024 字节）缓冲区，通过 wifi 从 NAS 复制到笔记本电脑上的 USB 记忆棒，比 Win7 本机吞吐量更好。

在待办事项清单上：

• 添加错误处理（catch）
• 处理 get-childitem 文件列表作为输入
• 复制多个文件时的嵌套进度条（文件 x of y，如果复制了总数据，则为 % 等）
• 缓冲区大小的输入参数

随意使用/改进:-)

function Copy-File {
param( [string]$from, [string]$to)
$ffile = [io.file]::OpenRead($from)
$tofile = [io.file]::OpenWrite($to)
Write-Progress `
    -Activity "Copying file" `
    -status ($from.Split("\")|select -last 1) `
    -PercentComplete 0
try {
    $sw = [System.Diagnostics.Stopwatch]::StartNew();
    [byte[]]$buff = new-object byte[] (4096*1024)
    [long]$total = [long]$count = 0
    do {
        $count = $ffile.Read($buff, 0, $buff.Length)
        $tofile.Write($buff, 0, $count)
        $total += $count
        [int]$pctcomp = ([int]($total/$ffile.Length* 100));
        [int]$secselapsed = [int]($sw.elapsedmilliseconds.ToString())/1000;
        if ( $secselapsed -ne 0 ) {
            [single]$xferrate = (($total/$secselapsed)/1mb);
        } else {
            [single]$xferrate = 0.0
        }
        if ($total % 1mb -eq 0) {
            if($pctcomp -gt 0)`
                {[int]$secsleft = ((($secselapsed/$pctcomp)* 100)-$secselapsed);
                } else {
                [int]$secsleft = 0};
            Write-Progress `
                -Activity ($pctcomp.ToString() + "% Copying file @ " + "{0:n2}" -f $xferrate + " MB/s")`
                -status ($from.Split("\")|select -last 1) `
                -PercentComplete $pctcomp `
                -SecondsRemaining $secsleft;
        }
    } while ($count -gt 0)
$sw.Stop();
$sw.Reset();
}
finally {
    write-host (($from.Split("\")|select -last 1) + `
     " copied in " + $secselapsed + " seconds at " + `
     "{0:n2}" -f [int](($ffile.length/$secselapsed)/1mb) + " MB/s.");
     $ffile.Close();
     $tofile.Close();
    }
}

Answer 6

不是我知道的。无论如何，我不建议为此使用 copy-item 。我不认为它被设计为像 robocopy.exe 那样健壮，以支持重试，您希望通过网络复制非常大的文件。

Answer 7

我发现上面的例子都不符合我的需要，我想复制一个带有子目录的目录，问题是我的源目录有太多文件所以我很快就达到了 BITS 文件限制（我有 > 1500 个文件）还有总目录尺寸相当大。

我在https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress/找到了一个使用 robocopy 的函数，这是一个很好的起点，但是我发现它不够健壮，无法处理尾部斜杠，优雅的空格，并且在脚本停止时不会停止复制。

这是我的改进版：

function Copy-ItemWithProgress
{
    <#
    .SYNOPSIS
    RoboCopy with PowerShell progress.

    .DESCRIPTION
    Performs file copy with RoboCopy. Output from RoboCopy is captured,
    parsed, and returned as Powershell native status and progress.

    .PARAMETER Source
    Directory to copy files from, this should not contain trailing slashes

    .PARAMETER Destination
    DIrectory to copy files to, this should not contain trailing slahes

    .PARAMETER FilesToCopy
    A wildcard expresion of which files to copy, defaults to *.*

    .PARAMETER RobocopyArgs
    List of arguments passed directly to Robocopy.
    Must not conflict with defaults: /ndl /TEE /Bytes /NC /nfl /Log

    .PARAMETER ProgressID
    When specified (>=0) will use this identifier for the progress bar

    .PARAMETER ParentProgressID
    When specified (>= 0) will use this identifier as the parent ID for progress bars
    so that they appear nested which allows for usage in more complex scripts.

    .OUTPUTS
    Returns an object with the status of final copy.
    REMINDER: Any error level below 8 can be considered a success by RoboCopy.

    .EXAMPLE
    C:\PS> .\Copy-ItemWithProgress c:\Src d:\Dest

    Copy the contents of the c:\Src directory to a directory d:\Dest
    Without the /e or /mir switch, only files from the root of c:\src are copied.

    .EXAMPLE
    C:\PS> .\Copy-ItemWithProgress '"c:\Src Files"' d:\Dest /mir /xf *.log -Verbose

    Copy the contents of the 'c:\Name with Space' directory to a directory d:\Dest
    /mir and /XF parameters are passed to robocopy, and script is run verbose

    .LINK
    https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress

    .NOTES
    By Keith S. Garner (KeithGa@KeithGa.com) - 6/23/2014
    With inspiration by Trevor Sullivan @pcgeek86
    Tweaked by Justin Marshall - 02/20/2020

    #>

    [CmdletBinding()]
    param(
        [Parameter(Mandatory=$true)]
        [string]$Source,
        [Parameter(Mandatory=$true)]
        [string]$Destination,
        [Parameter(Mandatory=$false)]
        [string]$FilesToCopy="*.*",
        [Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)] 
        [string[]] $RobocopyArgs,
        [int]$ParentProgressID=-1,
        [int]$ProgressID=-1
    )

    #handle spaces and trailing slashes
    $SourceDir = '"{0}"' -f ($Source -replace "\\+$","")
    $TargetDir = '"{0}"' -f ($Destination -replace "\\+$","")


    $ScanLog  = [IO.Path]::GetTempFileName()
    $RoboLog  = [IO.Path]::GetTempFileName()
    $ScanArgs = @($SourceDir,$TargetDir,$FilesToCopy) + $RobocopyArgs + "/ndl /TEE /bytes /Log:$ScanLog /nfl /L".Split(" ")
    $RoboArgs = @($SourceDir,$TargetDir,$FilesToCopy) + $RobocopyArgs + "/ndl /TEE /bytes /Log:$RoboLog /NC".Split(" ")

    # Launch Robocopy Processes
    write-verbose ("Robocopy Scan:`n" + ($ScanArgs -join " "))
    write-verbose ("Robocopy Full:`n" + ($RoboArgs -join " "))
    $ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs
    try
    {
        $RoboRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $RoboArgs
        try
        {
            # Parse Robocopy "Scan" pass
            $ScanRun.WaitForExit()
            $LogData = get-content $ScanLog
            if ($ScanRun.ExitCode -ge 8)
            {
                $LogData|out-string|Write-Error
                throw "Robocopy $($ScanRun.ExitCode)"
            }
            $FileSize = [regex]::Match($LogData[-4],".+:\s+(\d+)\s+(\d+)").Groups[2].Value
            write-verbose ("Robocopy Bytes: $FileSize `n" +($LogData -join "`n"))
            #determine progress parameters
            $ProgressParms=@{}
            if ($ParentProgressID -ge 0) {
                $ProgressParms['ParentID']=$ParentProgressID
            }
            if ($ProgressID -ge 0) {
                $ProgressParms['ID']=$ProgressID
            } else {
                $ProgressParms['ID']=$RoboRun.Id
            }
            # Monitor Full RoboCopy
            while (!$RoboRun.HasExited)
            {
                $LogData = get-content $RoboLog
                $Files = $LogData -match "^\s*(\d+)\s+(\S+)"
                if ($null -ne $Files )
                {
                    $copied = ($Files[0..($Files.Length-2)] | ForEach-Object {$_.Split("`t")[-2]} | Measure-Object -sum).Sum
                    if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
                    {
                        write-progress Copy -ParentID $ProgressParms['ID'] -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
                        $Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
                    }
                    else
                    {
                        write-progress Copy -ParentID $ProgressParms['ID'] -Complete
                    }
                    write-progress ROBOCOPY  -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1] @ProgressParms
                }
            }
        } finally {
            if (!$RoboRun.HasExited) {Write-Warning "Terminating copy process with ID $($RoboRun.Id)..."; $RoboRun.Kill() ; }
            $RoboRun.WaitForExit()
            # Parse full RoboCopy pass results, and cleanup
            (get-content $RoboLog)[-11..-2] | out-string | Write-Verbose
            remove-item $RoboLog
            write-output ([PSCustomObject]@{ ExitCode = $RoboRun.ExitCode })

        }
    } finally {
        if (!$ScanRun.HasExited) {Write-Warning "Terminating scan process with ID $($ScanRun.Id)..."; $ScanRun.Kill() }
        $ScanRun.WaitForExit()

        remove-item $ScanLog
    }
}

Answer 8

讨厌成为一个碰到老话题的人，但我发现这篇文章非常有用。在stej对片段进行性能测试并对其进行改进Graham Gold以及Nacht的 BITS 建议之后，我得出结论：

1. 我真的很喜欢格雷厄姆的时间估计和速度读数命令。
2. 我也非常喜欢使用 BITS 作为我的传输方法显着提高的速度。

面对两者之间的抉择……我发现 Start-BitsTransfer 支持异步模式。所以这是我合并两者的结果。

function Copy-File {
    # ref: https://stackoverflow.com/a/55527732/3626361
    param([string]$From, [string]$To)

    try {
        $job = Start-BitsTransfer -Source $From -Destination $To `
            -Description "Moving: $From => $To" `
            -DisplayName "Backup" -Asynchronous

        # Start stopwatch
        $sw = [System.Diagnostics.Stopwatch]::StartNew()
        Write-Progress -Activity "Connecting..."

        while ($job.JobState.ToString() -ne "Transferred") {
            switch ($job.JobState.ToString()) {
                "Connecting" {
                    break
                }
                "Transferring" {
                    $pctcomp = ($job.BytesTransferred / $job.BytesTotal) * 100
                    $elapsed = ($sw.elapsedmilliseconds.ToString()) / 1000

                    if ($elapsed -eq 0) {
                        $xferrate = 0.0
                    }
                    else {
                        $xferrate = (($job.BytesTransferred / $elapsed) / 1mb);
                    }

                    if ($job.BytesTransferred % 1mb -eq 0) {
                        if ($pctcomp -gt 0) {
                            $secsleft = ((($elapsed / $pctcomp) * 100) - $elapsed)
                        }
                        else {
                            $secsleft = 0
                        }

                        Write-Progress -Activity ("Copying file '" + ($From.Split("\") | Select-Object -last 1) + "' @ " + "{0:n2}" -f $xferrate + "MB/s") `
                            -PercentComplete $pctcomp `
                            -SecondsRemaining $secsleft
                    }
                    break
                }
                "Transferred" {
                    break
                }
                Default {
                    throw $job.JobState.ToString() + " unexpected BITS state."
                }
            }
        }

        $sw.Stop()
        $sw.Reset()
    }
    finally {
        Complete-BitsTransfer -BitsJob $job
        Write-Progress -Activity "Completed" -Completed
    }
}

Answer 9

此递归函数将文件和目录从源路径递归复制到目标路径

如果目标路径上已存在文件，则仅使用较新的文件复制它们。

Function Copy-FilesBitsTransfer(
        [Parameter(Mandatory=$true)][String]$sourcePath, 
        [Parameter(Mandatory=$true)][String]$destinationPath, 
        [Parameter(Mandatory=$false)][bool]$createRootDirectory = $true)
{
    $item = Get-Item $sourcePath
    $itemName = Split-Path $sourcePath -leaf
    if (!$item.PSIsContainer){ #Item Is a file

        $clientFileTime = Get-Item $sourcePath | select LastWriteTime -ExpandProperty LastWriteTime

        if (!(Test-Path -Path $destinationPath\$itemName)){
            Start-BitsTransfer -Source $sourcePath -Destination $destinationPath -Description "$sourcePath >> $destinationPath" -DisplayName "Copy Template file" -Confirm:$false
            if (!$?){
                return $false
            }
        }
        else{
            $serverFileTime = Get-Item $destinationPath\$itemName | select LastWriteTime -ExpandProperty LastWriteTime

            if ($serverFileTime -lt $clientFileTime)
            {
                Start-BitsTransfer -Source $sourcePath -Destination $destinationPath -Description "$sourcePath >> $destinationPath" -DisplayName "Copy Template file" -Confirm:$false
                if (!$?){
                    return $false
                }
            }
        }
    }
    else{ #Item Is a directory
        if ($createRootDirectory){
            $destinationPath = "$destinationPath\$itemName"
            if (!(Test-Path -Path $destinationPath -PathType Container)){
                if (Test-Path -Path $destinationPath -PathType Leaf){ #In case item is a file, delete it.
                    Remove-Item -Path $destinationPath
                }

                New-Item -ItemType Directory $destinationPath | Out-Null
                if (!$?){
                    return $false
                }

            }
        }
        Foreach ($fileOrDirectory in (Get-Item -Path "$sourcePath\*"))
        {
            $status = Copy-FilesBitsTransfer $fileOrDirectory $destinationPath $true
            if (!$status){
                return $false
            }
        }
    }

    return $true
}

Answer 10

来自嘿，脚本专家的 Sean Kearney ！博客有一个我发现效果很好的解决方案。

Function Copy-WithProgress
{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory=$true,
            ValueFromPipelineByPropertyName=$true,
            Position=0)]
        $Source,
        [Parameter(Mandatory=$true,
            ValueFromPipelineByPropertyName=$true,
            Position=0)]
        $Destination
    )

    $Source=$Source.tolower()
    $Filelist=Get-Childitem "$Source" –Recurse
    $Total=$Filelist.count
    $Position=0

    foreach ($File in $Filelist)
    {
        $Filename=$File.Fullname.tolower().replace($Source,'')
        $DestinationFile=($Destination+$Filename)
        Write-Progress -Activity "Copying data from '$source' to '$Destination'" -Status "Copying File $Filename" -PercentComplete (($Position/$total)*100)
        Copy-Item $File.FullName -Destination $DestinationFile
        $Position++
    }
}

然后使用它：

Copy-WithProgress -Source $src -Destination $dest

Answer 11

Trevor Sullivan 有一篇关于如何在 Robocopy 上将名为Copy-ItemWithProgress的命令添加到 PowerShell 的文章。

Answer 12

在复制单个大文件时获取更新进度

在复制文件时获取进度条的一种方法是在单独的进程中启动复制操作，然后从父进程轮询目标文件的大小，直到子进程完成。

function Copy-LargeFileWithProgress {
    param(
        [string] $SourcePath,
        [string] $DestPath
    )
    
    $source = Get-Item $SourcePath
    $name   = $source.Name
    $size   = $source.Size
    $argv   = @('-Command', "Copy-Item '$SourcePath' '$DestPath'")

    $proc   = Start-Process 'pwsh' $argv -PassThru 
 
    while (-not $proc.HasExited) {
        Start-Sleep -Seconds 2
        if (Test-Path $DestPath) {
            $destSize = (Get-Item $DestPath).Size
            $status   = '{0:N0}/{1:N0} bytes' -f $destSize, $size
            $complete = [Math]::Max($destSize / $size * 100, 100)
    
            Write-Progress -Activity        "Copying $name" `
                           -Status          $status `
                           -PercentComplete $complete
        }
    }
}

上面的代码可以在 Linux 上运行，但还没有在 Windows 上尝试过。在 Windows 上，-WindowStyle Minimized可能需要该选项Start-Process。

上面的实现是稀疏的，以简洁地演示该方法。如果有人想使用它，它可以使用一些润色，例如以更好的格式显示状态消息（即，1.51G/210.00G而不是原始字节数）。此外，最初报告的大小(Get-Item $DestPath).Size可能在几次迭代中不正确（因此使用了 max 操作）。这可以更优雅地解释。