I have a continuous webjob that is running on azure that after a larger deployment 8 hours ago gets the status never finished in some cases and in other cases completed the job. I have enabled all logging I could find and have been spending quite a few hours trying to figure out what the problem is.
The only log error information I seem to be able to find is from the job_log which states:
[11/15/2017 14:46:23 > e553e5: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found. [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase
1 cmd, Exception ex) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.cs:line 50 [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.<DeleteBlobImpl>b__33(RESTCommand1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 3349 [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 299 [11/15/2017 14:46:23 > e553e5: ERR ] --- End of inner exception stack trace --- [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50 [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.EndDelete(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 1729 [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 114 [11/15/2017 14:46:23 > e553e5: ERR ] --- End of stack trace from previous location where exception was thrown --- [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.Azure.WebJobs.Host.Protocols.PersistentQueueWriter1.<DeleteAsync>d__6.MoveNext() [11/15/2017 14:46:23 > e553e5: ERR ] --- End of stack trace from previous location where exception was thrown --- [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionInstanceLogger.<DeleteLogFunctionStartedAsync>d__e.MoveNext() [11/15/2017 14:46:23 > e553e5: ERR ] --- End of stack trace from previous location where exception was thrown --- [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext() [11/15/2017 14:46:23 > e553e5: ERR ] --- End of stack trace from previous location where exception was thrown --- [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor1.d__0.MoveNext() [11/15/2017 14:46:23 > e553e5: ERR ] --- End of stack trace from previous location where exception was thrown --- [11/15/2017 14:46:23 > e553e5: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.b__0() [11/15/2017 14:46:23 > e553e5: ERR ] at System.Threading.ThreadHelper.ThreadStart_Context(Object state) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) [11/15/2017 14:46:23 > e553e5: ERR ] at System.Threading.ThreadHelper.ThreadStart()
Can anyone give me some ideas on how to debug this, because I am out of ideas.
My webjobs main looks like this:
static void Main()
{
var host = new JobHost();
var config = new JobHostConfiguration();
config.Queues.MaxPollingInterval = new TimeSpan(0,0,0,30);
config.Queues.MaxDequeueCount = 3;
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
and the processqueuemessage looks like:
public static void ProcessQueueMessage([QueueTrigger("importqueue")] string msg)
{
try
{
WorkerWebJobCore wwjc = new WorkerWebJobCore();
wwjc.RunCore(msg, TableStorageAccessResources.ImportQueue,
TableStorageAccessResources.TableStorageDataOneId,
TableStorageAccessResources.TableStorageDataOnePassword);
}
catch (Exception e)
{
CommunicatorLog.Log.LogError("WebJobWorker","WebJobWorker","Error in processing queue message","ERRWJWF01");
}
}
So I have a catch around everything, therefore I don't get how it can fail?
Thanks in advance.