I've just started playing with the Grinder (love it!) but have encountered some curious behavior, as follows:
I had been using 5 workers to parse tweet text and send them to the URL I am load testing. Grinder was working without a hitch but I noticed that Python urlencode was failing on some of the unicode or latin encoded stuff (i.e. some threads were dying an untimely death as a result).
I thought, since Jython is really Java under the hood, that I would try using java.net.URLEncoder on suspect strings and see how it behaved.
So at the top of my Jython script I added
from java.net import URLEncoder
and wrapped the python urlencode in a try/catch which fell back on UrlEncoder.encode(myString,'UTF-8')
And it seemed to work flawlessly when I tested it using one worker. But once I upped the number of workers above 2 or 3, the workers would start, but then fail to communicate with Console, throwing the following error after an unreasonably long load time:
2012-11-15 15:59:04,287 ERROR worker-bootstrap: Error running worker process
net.grinder.communication.CommunicationException: Exception whilst sending message
at net.grinder.communication.AbstractSender.send(AbstractSender.java:57) ~[grinder-core-3.11.jar:na]
at net.grinder.communication.QueuedSenderDecorator.flush(QueuedSenderDecorator.java:60) ~[grinder-core-3.11.jar:na]
at net.grinder.engine.process.GrinderProcess.sendStatusMessage(GrinderProcess.java:638) ~[grinder-core-3.11.jar:na]
at net.grinder.engine.process.GrinderProcess.run(GrinderProcess.java:430) ~[grinder-core-3.11.jar:na]
at net.grinder.engine.process.WorkerProcessEntryPoint.run(WorkerProcessEntryPoint.java:86) [grinder-core-3.11.jar:na]
at net.grinder.engine.process.WorkerProcessEntryPoint.main(WorkerProcessEntryPoint.java:59) [grinder-core-3.11.jar:na]
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_37]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) ~[na:1.6.0_37]
at java.net.SocketOutputStream.write(SocketOutputStream.java:136) ~[na:1.6.0_37]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) ~[na:1.6.0_37]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) ~[na:1.6.0_37]
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1793) ~[na:1.6.0_37]
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699) ~[na:1.6.0_37]
at net.grinder.communication.AbstractSender.writeMessageToStream(AbstractSender.java:90) ~[grinder-core-3.11.jar:na]
at net.grinder.communication.StreamSender.writeMessage(StreamSender.java:70) ~[grinder-core-3.11.jar:na]
at net.grinder.communication.AbstractSender.send(AbstractSender.java:53) ~[grinder-core-3.11.jar:na]
... 5 common frames omitted
This looks essentially like a timeout to me (broken pipe, implying I guess that either Console got tired of listening or Agent got tired of trying to connect). But I'm not sure why. (Nor am I sure how I feel about seeing the word "whilst" in an exception. Strewth, Hacker! Didst thou ne'er see sinister leaks of memory? )
As far as the code problem at hand, I can do better then the (currently implemented) default behavior of Python urlencode - I do know a few tricks for improving on this but they tend to get messy and I'm kind of guessing that some of them are not going to work in Jython.
But I'm curious on a general level here. Java URLEncode works just fine - on 3 workers or fewer -- so why should it fail given more workers?
Guesses:
Am I running out of system resources?
Horrors - is each worker creating its own JVM?
Well, checked, and yes, it is, as it turns out. but this didn't stop things previously. why now? I have 2 processors in my mac - I would think if this were the problem, we should fail at 3 workers, but, nope. 4 or 5, yep.
Is using native Java libraries in Jython, for some reason, NOT a good idea?
Insights appreciated.
Thanks JB