0

作为计划作业的一部分，我需要从许多服务器并行下载一些 web 内容，但我找不到并行/异步运行下载的正确方法。如何才能做到这一点？

没有任何并行性，我可以这样做，但速度很慢：

$web = [System.Net.WebClient]::new()
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# $srvList is a list of servers of viariable length
$allData = ""
foreach ($srv in $srvList) {
    $url = "https:\\$srv\MyWebPage"
    $data = $web.DownloadString($url)
    $allData += $data
}

但是如何通过“$web.DownloadStringAsync”并行执行此操作？我找到了这个片段，但我没有看到如何获取每个调用的结果以及如何连接它：

$job = Register-ObjectEvent -InputObject $web -EventName DownloadStringCompleted -Action {
    Write-Host 'Download completed'
    write-host $EventArgs.Result
}
$web.DownloadString($url)

有人知道如何以一种简短而聪明的方式解决这个问题吗？

4

2 回答 2

1

最好和最快的方法是使用运行空间：

Add-Type -AssemblyName System.Collections

$GH = [hashtable]::Synchronized(@{})

[System.Collections.Generic.List[PSObject]]$GH.results = [System.Collections.Generic.List[string]]::new()
[System.Collections.Generic.List[string]]$GH.servers = @('server1','server2');
[System.Collections.Generic.List[string]]$GH.functions = @('Download-Content');

[System.Collections.Generic.List[PSObject]]$jobs = @()


#-----------------------------------------------------------------
function Download-Content {
#-----------------------------------------------------------------

   # a function which runs parallel

   param(
    [string]$server
   )

   $web = [System.Net.WebClient]::new()
   [Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
   [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

   $url = "https:\\$server\MyWebPage"
   $data = $web.DownloadString($url)
   $GH.results.Add( $data )
}

#-----------------------------------------------------------------
function Create-InitialSessionState {
#-----------------------------------------------------------------

    param(
        [System.Collections.Generic.List[string]]$functionNameList
    )

    # Setting up an initial session state object
    $initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()

    foreach( $functionName in $functionNameList ) {

        # Getting the function definition for the functions to add
        $functionDefinition = Get-Content function:\$functionName
        $functionEntry = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName, $functionDefinition

        # And add it to the iss object
        [void]$initialSessionState.Commands.Add($functionEntry)
    }

    return $initialSessionState
}

#-----------------------------------------------------------------
function Create-RunspacePool {
#-----------------------------------------------------------------

    param(
        [InitialSessionState]$initialSessionState
    )

    $runspacePool = [RunspaceFactory]::CreateRunspacePool(1, ([int]$env:NUMBER_OF_PROCESSORS + 1), $initialSessionState, $Host)
    $runspacePool.ApartmentState = 'MTA'
    $runspacePool.ThreadOptions  = "ReuseThread"
    [void]$runspacePool.Open()

    return $runspacePool
}


#-----------------------------------------------------------------
function Release-Runspaces {
#-----------------------------------------------------------------

    $runspaces = Get-Runspace | Where { $_.Id -gt 1 }

    foreach( $runspace in $runspaces ) {
        try{
            [void]$runspace.Close()
            [void]$runspace.Dispose()
            }
        catch {
        }
    }
}


$initialSessionState = Create-InitialSessionState -functionNameList $GH.functions
$runspacePool        = Create-RunspacePool -initialSessionState $initialSessionState


foreach ($server in $GH.servers)
{
    Write-Host $server

    $job = [System.Management.Automation.PowerShell]::Create($initialSessionState)
    $job.RunspacePool = $runspacePool

    $scriptBlock = { param ( [hashtable]$GH, [string]$server ); Download-Content -server $server }

    [void]$job.AddScript( $scriptBlock ).AddArgument( $GH ).AddArgument( $server )
    $jobs += New-Object PSObject -Property @{
                                    RunNum = $jobCounter++
                                    JobObj = $job
                                    Result = $job.BeginInvoke() }

    do {
        Sleep -Seconds 1 
    } while( $runspacePool.GetAvailableRunspaces() -lt 1 )

}

Do {
    Sleep -Seconds 1
} While( $jobs.Result.IsCompleted -contains $false)


$GH.results

Release-Runspaces | Out-Null

[void]$runspacePool.Close()
[void]$runspacePool.Dispose()
于 2019-12-11T16:58:17.070 回答
0

最后我通过事件找到了一个简单的解决方案。这是我的代码片段：

cls
Remove-Variable * -ea 0

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$srvList = @('srv1','srv2','srv3')
$webObjList = [System.Collections.ArrayList]::new()
$eventList  = [System.Collections.ArrayList]::new()
$resultList = [System.Collections.ArrayList]::new()

$i=0
foreach ($srv in $srvList) {
    $null  = $webObjList.add([System.Net.WebClient]::new())
    $null  = $eventList.add($(Register-ObjectEvent -InputObject $webObjList[$i] -EventName DownloadStringCompleted -SourceIdentifier $srv))
    $null  = $resultList.add($webObjList[$i].DownloadStringTaskAsync("https://$srv/MyWebPage"))
    $i++
}
do {sleep -Milliseconds 10} until ($resultList.IsCompleted -notcontains $false)
foreach ($srv in $srvList) {Unregister-Event $srv}

# show all Results:
$resultList.result
于 2019-12-14T10:44:56.583 回答