1

我的数据处理有点问题。

public void ParseDetails()
{            
    for (int i = 0; i < mListAppInfo.Count; ++i)
    {
        ParseOneDetail(i);
    }
}

对于 300 条记录，通常需要大约 13-15 分钟。

我试图通过使用来改进，Parallel.For()但它总是在某个时候停止。

public void ParseDetails()
{
    Parallel.For(0, mListAppInfo.Count, i => ParseOneDetail(i));
}

在方法ParseOneDetail(int index)中，我设置了一个输出日志来跟踪正在处理的记录 ID。总是在某个时候挂掉，我不知道为什么..

ParseOneDetail(): 89 ...
ParseOneDetail(): 90 ...
ParseOneDetail(): 243 ...
ParseOneDetail(): 92 ...
ParseOneDetail(): 244 ...
ParseOneDetail(): 93 ...
ParseOneDetail(): 245 ...
ParseOneDetail(): 247 ...
ParseOneDetail(): 94 ...
ParseOneDetail(): 248 ...
ParseOneDetail(): 95 ...
ParseOneDetail(): 99 ...
ParseOneDetail(): 249 ...
ParseOneDetail(): 100 ...
_ <hang at this point>

感谢您的帮助和建议，以改善这一点。谢谢！

Edit 1：更新方法：

private void ParseOneDetail(int index)
{
    Console.WriteLine("ParseOneDetail(): " + index + " ... ");
    ApplicationInfo appInfo = mListAppInfo[index];

    var htmlWeb = new HtmlWeb();
    var document = htmlWeb.Load(appInfo.AppAnnieURL);

    // get first one only
    HtmlNode nodeStoreURL = document.DocumentNode.SelectSingleNode(Constants.XPATH_FIRST);
    appInfo.StoreURL = nodeStoreURL.Attributes[Constants.HREF].Value;
}

Edit 2Enigmativity：这是按照建议运行一段时间后的错误输出，

ParseOneDetail(): 234 ...
ParseOneDetail(): 87 ...
ParseOneDetail(): 235 ...
ParseOneDetail(): 236 ...
ParseOneDetail(): 88 ...
ParseOneDetail(): 238 ...
ParseOneDetail(): 89 ...
ParseOneDetail(): 90 ...
ParseOneDetail(): 239 ...
ParseOneDetail(): 92 ...

Unhandled Exception: System.AggregateException: One or more errors occurred. ---
> System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetResponse()
   at HtmlAgilityPack.HtmlWeb.Get(Uri uri, String method, String path, HtmlDocum
ent doc, IWebProxy proxy, ICredentials creds) in D:\Source\htmlagilitypack.new\T
runk\HtmlAgilityPack\HtmlWeb.cs:line 1355
   at HtmlAgilityPack.HtmlWeb.LoadUrl(Uri uri, String method, WebProxy proxy, Ne
tworkCredential creds) in D:\Source\htmlagilitypack.new\Trunk\HtmlAgilityPack\Ht
mlWeb.cs:line 1479
   at HtmlAgilityPack.HtmlWeb.Load(String url, String method) in D:\Source\htmla
gilitypack.new\Trunk\HtmlAgilityPack\HtmlWeb.cs:line 1103
   at HtmlAgilityPack.HtmlWeb.Load(String url) in D:\Source\htmlagilitypack.new\
Trunk\HtmlAgilityPack\HtmlWeb.cs:line 1061
   at SimpleChartParser.AppAnnieParser.ParseOneDetail(ApplicationInfo appInfo) i
n c:\users\nhn60\documents\visual studio 2010\Projects\FunToolPack\SimpleChartPa
rser\AppAnnieParser.cs:line 90
   at SimpleChartParser.AppAnnieParser.<ParseDetails>b__0(ApplicationInfo ai) in
 c:\users\nhn60\documents\visual studio 2010\Projects\FunToolPack\SimpleChartPar
ser\AppAnnieParser.cs:line 80
   at System.Threading.Tasks.Parallel.<>c__DisplayClass21`2.<ForEachWorker>b__17
(Int32 i)
   at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass7.<ExecuteSelfReplicating>b__
6(Object )
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceled
Exceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationTo
ken cancellationToken)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int
32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWit
hState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](TSource[] ar
ray, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Act
ion`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEveryt
hing, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`
1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState
, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithE
verything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Act
ion`1 body)
   at SimpleChartParser.AppAnnieParser.ParseDetails() in c:\users\nhn60\document
s\visual studio 2010\Projects\FunToolPack\SimpleChartParser\AppAnnieParser.cs:li
ne 80
   at SimpleChartParser.Program.Main(String[] args) in c:\users\nhn60\documents\
visual studio 2010\Projects\FunToolPack\SimpleChartParser\Program.cs:line 15
4

2 回答 2

0

这本身不是一个答案，但是您可以尝试修改您的代码以使其像这样工作并让我们知道您是否仍然有错误？

public void ParseDetails()
{
    Parallel.ForEach(mListAppInfo.ToArray(), ai => ParseOneDetail(ai));
}

private void ParseOneDetail(ApplicationInfo appInfo)
{
    var htmlWeb = new HtmlWeb();
    var document = htmlWeb.Load(appInfo.AppAnnieURL);

    // get first one only
    HtmlNode nodeStoreURL = 
            document.DocumentNode.SelectSingleNode(Constants.XPATH_FIRST);
    appInfo.StoreURL = nodeStoreURL.Attributes[Constants.HREF].Value;
}
于 2012-08-29T09:21:41.113 回答
0

我会注释掉每一行ParseOneDetail并检查它是否完成运行。

如果是（应该），取消前半部分的注释，并检查它是否会再次完成。继续执行此二进制搜索，直到您准确确定是哪一行阻止它运行。该行中可能有一些不是线程安全的。

于 2012-08-29T09:36:54.063 回答