I'm a little confused about some of the details of the issue, but I hate to see this question go completely unanswered in case someone else has this issue.
This plugin (mouseSwipe) overrides the default dragging functionality for mobile devices. Whereas normally devices would scroll the page on the mouse starting event, this plugin overrides that behavior to detect click movement across an element. Since it interrupts that functionality, dragging the opposite direction (for scrolling) is also broken. If the plugin were still being maintained by the owner (it doesn't appear to be), it could be updated to fix this issue, or emit events that could be used to manually create the functionality you're wanting.
I assume this is also what is giving you trouble for clicking to go to the specified page.
If you want my honest opinion, I would choose a different library, perhaps one that focuses solely on the swipability of mobile devices, and then handle desktop functionality separately (though, if you're using PhoneGap, it's likely you aren't even publishing this to a web platform for desktops). If it's going to be on the web, you can use modernizr (or the like) to figure out if the device supports touch input, and then implement something like the following:
http://labs.rampinteractive.co.uk/touchSwipe/demos/Image_gallery_example.html
For devices that do not support touch, you could fall back to button/arrow-based navigation (after all, as a desktop user, I do not expect to be able to drag it back and forth with the mouse).