I'm a security guy by day a coder by night and I usually write my tools in java. It was something I was taught in school so really its just the simple fact that I'm comfortable writing in that language.
I have been for a few years now been limited by the inability to take a response from a server and build it up by actually running the JavaScript. This has drastically hurt my ability to do so many things like spider and simply see an actual response as if it was built in a browser. In order to compensate for this Weakness I have tried to use a number of different headless browsers. I tried JX browser, CORBA,SWT, HTML unit, etc. You name it I have tried to use it. As many of you know none of these technologies can even accurately handle some of the most simple JavaScript applications. Therefore, I'm really looking for an alternative.
I've been looking into potentially using something like server-side JavaScript in order to render the HTML and manipulate the DOM. I was looking at things like Node.js,PhantomJS, etc. ideally I would like to be able to keep using my Java code and somehow use these technologies as a supplement for those headless browsers. However, it looks like being able to talk to these technologies will be difficult. It looks like I might be able to leverage the commandline with Java and write to files with those js technologies and read those files with my Java code. This does seem a little clunky so I thought maybe I should just flip over to fully using these to build my tools. I wanted to get others' ideas or opinions on this. Just to give you an idea of what I typically need to do I need to be able to I need to build the send HTTP requests and get responses back,crawl sites, things like that.
I'm really just looking to see if there are other options available or if I'm going about this the right way