c# - Reading text from Amazon s3 stream

Question

I use the following code to read text file from Amazon S3, and processing it line by line. This code works but the problem is it is slow.

 GetObjectRequest getObjRequest = new GetObjectRequest()
    .WithBucketName(amazonSettings.BucketName)
    .WithKey(_fileKey);
using (AmazonS3 client = AWSClientFactory.CreateAmazonS3Client(
    amazonSettings.AccessKey, 
    amazonSettings.SecretAccessKey))
using (GetObjectResponse getObjRespone = client.GetObject(getObjRequest))
using (Stream amazonStream = getObjRespone.ResponseStream)
{                        
    StreamReader amazonStreamReader = new StreamReader(amazonStream);
    tempGsContact = new GSContact();
    while ((_fileLine = amazonStreamReader.ReadLine()) != null)
    {
        if (_fileLine.Equals("END:VCARD"))
        {
            // Make process 1
        }
        else if (!_fileLine.Equals(string.Empty))
        {
            //Make process 2
        }
    }                        
}

The question is: can I get more sufficient way to reduce the time cost?

Answer 1

There is a similar performance bottleneck on HTTPWebResponse in .NET, which is probably what the AmazonS3 class they made is wrapping.

It's caused by the object taking a long time to resolve proxy settings, there are a few potential solutions listed here, but the easiest option might be to add the following to your app.config file:

<system.net>
  <defaultProxy enabled="false">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

Alternatively you could replace the call here:

AWSClientFactory.CreateAmazonS3Client(amazonSettings.AccessKey, amazonSettings.SecretAccessKey)

with a call to an overload that accepts a third parameter of 'AmazonS3Config', where you can specify a null proxy via 'AmazonS3Config.ProxyHost = null' - which should effectively be the same as the above configuration change for only that request.

Answer 2

OR You can use this function also,

    private static void ReadS3Object(AmazonS3Client client)
    {
        GetObjectRequest request = new GetObjectRequest();

        request.WithBucketName(BUCKET_NAME);
        request.WithKey(S3_KEY);

        GetObjectResponse response = client.GetObject(request);

        StreamReader reader = new StreamReader(response.ResponseStream);

        String content = reader.ReadToEnd();

        Console.Out.WriteLine("Read S3 object with key " + S3_KEY + " in bucket " + BUCKET_NAME + ". Content is: " + content);
    }

Answer 3

This function is for calculating word count in text file.

private int WordCount(string awsBucketName, string awsFilePath, string wordForSearch)
    {
        string line =string.Empty;
        int counter = 0;
        if ((cloudKaseClient != null) & (token == tokenCfg))
        {
            var request = new GetObjectRequest()
            {
                BucketName = awsBucketName,
                Key = awsFilePath
            };
             using (var response = cloudKaseClient.GetObject(request))
            {
                    StreamReader reader = new StreamReader(response.ResponseStream);
                    while ((line = reader.ReadLine()) != null)
                    {
                        counter += line.Split(' ').Where(t => t.ToString().ToLower().Contains(wordForSearch.ToLower())).Count();
                    }
                    reader.Close();             
            }
        }
        return counter;
    }