0

我正在从网络服务器进行每日图像的 BITS 传输，并且在传输过程中我不断得到随机丢弃。

当它循环下载时，我偶尔会收到“连接过早关闭”或“安全通道支持中发生错误”。每个文件夹中大约有 180 张图像，其中可能有 5-10% 发生这种情况。对于那些未完成的，我需要重试下载。

我这样做的代码如下 - 我不完美的解决方法是运行循环两次，但我希望找到更好的解决方案。

# Set the URL where the images are located
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'

# Set the local path where the images will be stored
$path = 'C:\images\Wind_Waves\latest\'

# Create a list of all assets returned from $url
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | Where-Object{ $_.tagName -eq 'A' -and $_.href.ToLower().EndsWith("jpg") }

# Create a list of all href items from the table & call it $images
$images = $table.href 

# Enumerate all of the images - for troubleshooting purposes - can be removed
$images

# Check to make sure there are images available for download - arbitrarily picked more than 2 $images
if($images.count -gt 2){

    # Delete all of the files in the "latest" folder
    Remove-Item ($path + "*.*") -Force
# For loop to check to see if we already have the image and, if not, download it
ForEach ($image in $images)
{
if(![System.IO.File]::Exists($path + $image)){
    Write-Output "Downloading: " $image
    Start-BitsTransfer -Source ($url + $image) -Destination $path -TransferType Download -RetryInterval 60
    Start-Sleep 2
    }
}
Get-BitsTransfer | Where-Object {$_.JobState -eq "Transferred"} | Complete-BitsTransfer
} else {
Write-Output "No images to download"}
4

3 回答 3

1

这是上述代码的一些修改版本。发生错误时，BITS 传输作业对象似乎消失了，因此尝试查找/恢复该作业是没有用的。相反，我将整个 Try-Catch 块包装在一个 while 循环中，并在下载文件时退出。

$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
$MaxRetries = 3     # Initialize the maximum number of retry attempts.

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object {
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)

<#
Check to make sure there are images available for download - arbitrarily 
picked more than 2 $images
#>
if ($images.count -gt 2) {
    Remove-Item ($path + '*.*') -Force
    ForEach ($image in $images) {
        # Due to occasional failures to transfer, wrap the BITS transfer in a while loop
        # re-initialize the exit counter for each new image
        $retryCount = 0
        while ($retryCount -le $MaxRetries){
            Try {
                Write-Verbose -Message "Downloading: $image" -Verbose

                if (![System.IO.File]::Exists($path + $image)) {
                    $StartBitsTransferSplat = @{
                        Source        = ($url + $image) 
                        Destination   = $path 
                        RetryInterval = 60
                    }
                    Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                    Start-Sleep 2
                }

                # To get here, the transfer must have finished, so set the counter
                # greater than the max value to exit the loop
                $retryCount = $MaxRetries + 1
            } # End Try block

            Catch {
                $PSItem.Exception.Message
                $retryCount += 1
                Write-Warning -Message "Download of $image not complete or failed. Attempting retry #: $retryCount" -Verbose
            } # End Catch Block
            
        } # End While loop for retries

    } # End of loop over images

} # End of test for new images
else {
    Write-Warning -Message 'No images to download'
    $PSItem.Exception.Message
} # End of result for no new images
于 2020-09-18T15:23:47.647 回答
1

我在您的代码中看不到任何错误处理以在失败时恢复/重试/重新启动。

  1. 这意味着为什么循环或 Get 中没有 try/catch？
  2. 如果 Get 在循环中的每个下载作业上，为什么它在循环之外？

下载是TransferType的默认值，所以不需要指定，如果你这样做通常会产生错误。

所以，像这样的东西。我确实对此进行了测试，但从未失败过。然而，我有一个非常高速的互联网连接。如果您在企业内部执行此操作，边缘设备（过滤器、代理，也可能会减慢速度，可能会导致超时。）

$url  = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object{
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)

<#
Check to make sure there are images available for download - arbitrarily 
picked more than 2 $images
#>
if($images.count -gt 2)
{
    Remove-Item ($path + '*.*') -Force
    ForEach ($image in $images)
    {
        Try
        {
            Write-Verbose -Message "Downloading: $image" -Verbose

            if(![System.IO.File]::Exists($path + $image))
            {
                $StartBitsTransferSplat = @{
                    Source              = ($url + $image) 
                    Destination         = $path 
                    RetryInterval       = 60
                }
                Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                Start-Sleep 2
            }

            Get-BitsTransfer | 
            Where-Object {$PSItem.JobState -eq 'Transferred'} | 
            Complete-BitsTransfer
        }
        Catch
        {
            $PSItem.Exception.Message
            Write-Warning -Message "Download of $image not complete or failed. Attempting a resume/retry" -Verbose
            Get-BitsTransfer -Name $image | Resume-BitsTransfer

        }
    }
} 
else 
{
    Write-Warning -Message 'No images to download'
    $PSItem.Exception.Message
}

查看帮助文件

Resume-BitsTransfer 模块：bitstransfer 恢复 BITS 传输作业。

# Example 1: Resume all BITS transfer jobs owned by the current user
Get-BitsTransfer | Resume-BitsTransfer

# Example 2: Resume a new BITS transfer job that was initially suspended
$Bits = Start-BitsTransfer -DisplayName "MyJob" -Suspended
Add-BitsTransfer -BitsJob $Bits  -ClientFileName C:\myFile -ServerFileName http://www.SomeSiteName.com/file1
Resume-BitsTransfer -BitsJob $Bits -Asynchronous

# Example 3: Resume the BITS transfer by the specified display name
Get-BitsTransfer -Name "TestJob01" | Resume-BitsTransfer
于 2020-09-03T03:02:06.663 回答
0

这是 postanote 提供的代码和 Do-While 循环的组合，如果抛出错误，则重试下载最多 5 次。

$url  = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object{
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>

($images = $table.href)

 <# Check to make sure there are images available for download - arbitrarily 
    picked more than 2 $images #>

    if($images.count -gt 2)
    {
        Remove-Item ($path + '*.*') -Force
        ForEach ($image in $images)
        {
            # Create a Do-While loop to retry downloads up to 5 times if they fail
            $Stoploop = $false
            [int]$Retrycount = "0"
            do{
            Try
            {
                Write-Verbose -Message "Downloading: $image" -Verbose
    
                if(![System.IO.File]::Exists($path + $image))
                {
                    $StartBitsTransferSplat = @{
                        Source              = ($url + $image) 
                        Destination         = $path 
                        RetryInterval       = 60
                    }
                    Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                    Start-Sleep 10
                    $Stoploop = $true
                }
    
                Get-BitsTransfer | 
                Where-Object {$PSItem.JobState -eq 'Transferred'} | 
                Complete-BitsTransfer
            }
            Catch
            {
                if ($Retrycount -gt 5){
                $PSItem.Exception.Message
                Write-Warning -Message "Download of $image not complete or failed." -Verbose
                $Stoploop = $true
                }
                else {
                Write-Host "Could not download the image, retrying..."
                Start-Sleep 10
                $Retrycount = $Retrycount + 1
                }
            }
        }
        While ($Stoploop -eq $false)
        }
    }
    else 
    {
        Write-Warning -Message 'No images to download'
        $PSItem.Exception.Message
    }
于 2020-09-04T14:35:27.863 回答