2

我正在制作一个必须尽快处理大约 5000 个字符串的程序。其中大约 2000 个字符串必须通过 web 请求翻译到 mymemory.translated.net。(参见下面的代码,JSON 部分已删除,因为此处不需要)

Try

          url = "http://api.mymemory.translated.net/get?q=" & Firstpart & "!&langpair=de|it&de=somemail@christmas.com"

          request = DirectCast(WebRequest.Create(url), HttpWebRequest)
          response = DirectCast(request.GetResponse(), HttpWebResponse)
          myreader = New StreamReader(response.GetResponseStream())

          Dim rawresp As String
          rawresp = myreader.ReadToEnd()
          Debug.WriteLine("Raw:" & rawresp)


          Catch ex As Exception
              MessageBox.Show(ex.ToString)

          End Try

代码本身工作正常,问题是它是一个阻塞代码,每个字符串需要大约 1 秒。我所有的琴弦都超过半小时。我需要将此代码转换为非阻塞代码并同时进行多个调用。有人可以告诉我我该怎么做吗?我在考虑一个后台工作者,但这不会加快速度..它只会在不同的线程上执行代码......

谢谢!

4

3 回答 3

2

如果要发送 10 个并行请求,则必须创建 10 个 BackgroundWorker。或者手动创建10个线程。然后迭代,每当一个工作者/线程完成时,给它一个新任务。

我不建议触发 5000 个并行线程/工作者,你必须小心:这样的负载可能被解释为垃圾邮件或服务器的攻击。不要过度,也许可以与translated.net 交谈并询问他们接受的工作量。还要考虑你的机器和上游的互联网可以处理什么。

于 2013-06-07T06:29:59.293 回答
2

问题是您不仅受到最大并发操作数的阻碍。HttpWebRequests本质上会受到限制(我相信默认策略在任何给定时间只允许 2 个),因此您也必须覆盖该行为。请参考下面的代码。

Imports System.Diagnostics
Imports System.IO
Imports System.Net
Imports System.Threading
Imports System.Threading.Tasks

Public Class Form1

  ''' <summary>
  ''' Test entry point.
  ''' </summary>
  Private Sub Form1_Load() Handles MyBase.Load
    ' Generate enough words for us to test thoroughput.
    Dim words = Enumerable.Range(1, 100) _
      .Select(Function(i) "Word" + i.ToString()) _
      .ToArray()

    ' Maximum theoretical number of concurrent requests.
    Dim maxDegreeOfParallelism = 24
    Dim sw = Stopwatch.StartNew()

    ' Capture information regarding current SynchronizationContext
    ' so that we can perform thread marshalling later on.
    Dim uiScheduler = TaskScheduler.FromCurrentSynchronizationContext()
    Dim uiFactory = New TaskFactory(uiScheduler)

    Dim transformTask = Task.Factory.StartNew(
      Sub()
        ' Apply the transformation in parallel.
        ' Parallel.ForEach implements clever load
        ' balancing, so, since each request won't
        ' be doing much CPU work, it will spawn
        ' many parallel streams - likely more than
        ' the number of CPUs available.
        Parallel.ForEach(words, New ParallelOptions With {.MaxDegreeOfParallelism = maxDegreeOfParallelism},
          Sub(word)
            ' We are running on a thread pool thread now.
            ' Be careful not to access any UI until we hit
            ' uiFactory.StartNew(...)

            ' Perform transformation.
            Dim url = "http://api.mymemory.translated.net/get?q=" & word & "!&langpair=de|it&de=somemail@christmas.com"
            Dim request = DirectCast(WebRequest.Create(url), HttpWebRequest)

            ' Note that unless you specify this explicitly,
            ' the framework will use the default and you
            ' will be limited to 2 parallel requests
            ' regardless of how many threads you spawn.
            request.ServicePoint.ConnectionLimit = maxDegreeOfParallelism

            Using response = DirectCast(request.GetResponse(), HttpWebResponse)
              Using myreader As New StreamReader(response.GetResponseStream())
                Dim rawresp = myreader.ReadToEnd()

                Debug.WriteLine("Raw:" & rawresp)

                ' Transform the raw response here.
                Dim processed = rawresp

                uiFactory.StartNew(
                  Sub()
                    ' This is running on the UI thread,
                    ' so we can access the controls,
                    ' i.e. add the processed result
                    ' to the data grid.
                    Me.Text = processed
                  End Sub, TaskCreationOptions.PreferFairness)
              End Using
            End Using
          End Sub)
      End Sub)

    transformTask.ContinueWith(
      Sub(t As Task)
        ' Always stop the stopwatch.
        sw.Stop()

        ' Again, we are back on the UI thread, so we
        ' could access UI controls if we needed to.
        If t.Status = TaskStatus.Faulted Then
          Debug.Print("The transformation errored: {0}", t.Exception)
        Else
          Debug.Print("Operation completed in {0} s.", sw.ElapsedMilliseconds / 1000)
        End If
      End Sub,
      uiScheduler)
  End Sub

End Class
于 2013-06-07T06:35:19.440 回答
1

我会为每个请求创建一个任务,因此您可以使用以下命令为每个调用创建一个回调ContinueWith

  For Each InputString As String In myCollectionString


            Tasks.Task(Of String).Factory.StartNew(Function(inputString)

                    Dim request As HttpWebRequest
                    Dim myreader As StreamReader
                    Dim response As HttpWebResponse
                    Dim rawResp As String = String.Empty

                    Try

                      Dim url As String = "http://api.mymemory.translated.net/get?q=" & inputString & "!&langpair=de|it&de=somemail@christmas.com"

                      request = DirectCast(WebRequest.Create(url), HttpWebRequest)
                      response = DirectCast(request.GetResponse(), HttpWebResponse)
                      myreader = New StreamReader(response.GetResponseStream())

                      rawResp = myreader.ReadToEnd()
                      Debug.WriteLine("Raw:" & rawResp)


                    Catch ex As Exception
                      MessageBox.Show(ex.ToString)

                     End Try

                     Return rawResp

              End Function, CancellationToken.None, _ 
              Tasks.TaskCreationOptions.None).ContinueWith _
              (Sub(task As Tasks.Task(Of String))                                                                                                 
                'Dom something with result                                                                                                                          
                 Console.WriteLine(task.Result)                                                                                                                     
              End Sub)    

        Next
于 2013-06-07T06:38:15.903 回答