这是代码:
for (int x = 0; x < imagesSatelliteUrls.Count; x++)
{
if (!imagesSatelliteUrls[x].StartsWith("http://"))
{
imagesSatelliteUrls[x] = stringForSatelliteMapUrls + imagesSatelliteUrls[x];
}
using (WebClient client = new WebClient())
{
if (!imagesSatelliteUrls[x].Contains("href"))
{
client.DownloadFile(imagesSatelliteUrls[x],
UrlsDir + "SatelliteImage" + counter.ToString("D6"));
}
}
counter++;
}
它将逐个文件下载。List imagesSatelliteUrls 包含按组排序的 260 个文件链接。
例如:
index[0] "Group 1"
index[1] some link ....
index[2] some link ....
.
.
.
index[34] "Group 2"
index[35] some link ....
index[36] some link ....
.
.
.
.
index[71] "Group 3"
依此类推,有 7 个组。我希望它从每个组下载第一个文件,这意味着下载并行 7 个文件。Group 1 的第一个文件 2 3 4 5 6 7 然后,如果其中一个文件在任何组中完成,它将开始从该组下载下一个文件。
所以我会看到每秒有 7 个文件下载,每个文件来自另一个组。一个文件在某个组中完成下载,它应该移动到同一组中的下一个并开始下载。
我该怎么做 ?由于我现在使用的这个 client.DownloadFile 只会逐个文件下载文件。
尝试下载并行:
这是代码:
Parallel.For(0, imagesSatelliteUrls.Count, /*new ParallelOptions { MaxDegreeOfParallelism = 20 },*/ x =>
{
if (!imagesSatelliteUrls[x].StartsWith("http://"))
{
imagesSatelliteUrls[x] = stringForSatelliteMapUrls + imagesSatelliteUrls[x];
}
using (WebClient client = new WebClient())
{
if (!imagesSatelliteUrls[x].Contains("href"))
{
client.DownloadFile(imagesSatelliteUrls[x],
UrlsDir + "SatelliteImage" + counter.ToString("D6"));
}
}
counter++;
}); // end of Paralle
例外是:
System.Net.WebException was unhandled by user code
HResult=-2146233079
Message=An exception occurred during a WebClient request.
Source=System
StackTrace:
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
at System.Net.WebClient.DownloadFile(String address, String fileName)
at WeatherMaps.ExtractImages.<>c__DisplayClass2.<.ctor>b__0(Int32 x) in d:\C-Sharp\WeatherMaps\WeatherMaps\WeatherMaps\ExtractImages.cs:line 145
at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
InnerException: System.IO.IOException
HResult=-2147024864
Message=The process cannot access the file 'd:\localpath\Urls\SatelliteImage000000' because it is being used by another process.
Source=mscorlib
StackTrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
InnerException:
我也试过这段代码:
for (int i = 0; i < 7; i++)
{
Task.Factory.StartNew(() =>
{
// Here you can easily implement your checking algo as you see fit
while (counter < imagesSatelliteUrls.Count)
{
if (!imagesSatelliteUrls[count].StartsWith("http://"))
{
imagesSatelliteUrls[count] = stringForSatelliteMapUrls + imagesSatelliteUrls[count];
}
using (WebClient client = new WebClient())
{
if (!imagesSatelliteUrls[count].Contains("href"))
{
client.DownloadFile(imagesSatelliteUrls[count], UrlsDir + "SatelliteImage" + counter.ToString("D6"));
}
}
lock (this)
{
count++;
counter++;
}
}
});
}
System.Net.WebException was unhandled by user code
HResult=-2146233079
Message=An exception occurred during a WebClient request.
Source=System
StackTrace:
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
at System.Net.WebClient.DownloadFile(String address, String fileName)
at WeatherMaps.ExtractImages.<>c__DisplayClass4.<.ctor>b__2() in d:\C-Sharp\WeatherMaps\WeatherMaps\WeatherMaps\ExtractImages.cs:line 122
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
InnerException: System.IO.IOException
HResult=-2147024864
Message=The process cannot access the file 'd:\localpath\Urls\SatelliteImage000000' because it is being used by another process.
Source=mscorlib
StackTrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
InnerException: