4

我在我的 c# 项目中使用 CSVHelper 并将一个大的 csv 数据文件（大约 2000 条记录）读入内存。

https://github.com/JoshClose/CsvHelper

如果记录低于 500，它工作正常，它总是在不同的阶段抛出 IOException，这取决于网络或数字是否上升。我目前部署到Azure云平台，所以从blob存储读取到服务器，应该不是网络问题。

CsvHelper.CsvParserException: A parsing error occurred.

Row: '995' (1 based)
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
   at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
   at CsvHelper.CsvParser.GetChar(Int32& fieldStartPosition, Int32& rawFieldStartPosition, String& field, Boolean prevCharWasDelimiter, Int32& recordPosition, Int32& fieldLength, Boolean isPeek) in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 445
   at CsvHelper.CsvParser.ReadLine() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 247
   at CsvHelper.CsvParser.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 108
   --- End of inner exception stack trace ---
   at CsvHelper.CsvParser.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 136
   at CsvHelper.CsvReader.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvReader.cs:line 173

它抛出 while (csv.read())

 var wc = new WebClient();

            using (var sourceStream = wc.OpenRead(fileUrl))
            {

                using (var csv = new CsvReader(new StreamReader(sourceStream)))
                {


                    while (csv.Read())
                    {
                        try
                        {

//some reading operation
}
 catch (Exception ex)
                        {
                            _logger.Error(ex);
                        }
                    }
                    _logger.InfoFormat("Finished {0} reading data #{1}");

                }
            }

在哪里设置流式阅读器超时值？

4

1 回答 1

2

使用云资源（无论是 Azure 还是任何其他云资源）时，不应直接读取文件。充其量你应该实现重试逻辑以确保你绕过任何瞬态错误（阅读关于瞬态错误here，here和there或者只是在互联网上搜索术语“Transient Error”）。

在您的情况下，我建议您将调用包装在CloudBlockBlob.DownloadToStream方法周围。因此，您仍然可以使用 Stream 来解析文件，但您还将在 .NET Azure Blob API 库的安全性背后工作，该库代表您处理所有暂时性错误。

您的代码将如下所示：

// get the CloudblockBlob object
using(MemoryStream blobStream = new MemoryStream())
{
    blobObject.DownloadToStream(blobStream);
    using (var csv = new CsvReader(new StreamReader(blobStream)))
    {
        while (csv.Read())
        {
            try
            {
             //some reading operation
            }
            catch (Exception ex)
            {
                _logger.Error(ex);
            }
        }
        _logger.InfoFormat("Finished {0} reading data #{1}");
    }
}
于 2014-03-11T10:30:35.393 回答