Is it possible to get WS-Security working on an ASMX file or is ASMX too simplistic and I have to upgrade to WFC (which I really don't want to do)?
Yes, it is possible using Web Services Enhancements 3 (an add-on for Visual Studio 2005 and ASMX). See this MSDN page for a WSE-3-specific tutorial and use the usernameOverTransportSecurity
assertion, noting that this is not actually secure unless the connection takes place over a protected transport (i.e. SSL).
It is, however, not recommended that you do this, and I cannot fathom why you would not "want" to "upgrade to WCF" given the choice. Please note the following very important limitations of ASMX/WSE:
WSE is no longer a supported product. Although it still works, it no longer receives updates or even bug fixes.
No version of WSE will successfully integrate into Visual Studio 2008, or even Visual Studio 2005 running on Windows Vista x64 or newer.
WCF goes to a lot of trouble to provide thread-safe client operations and allow proxies to exist for long periods of time (which in turn provide significant per-operation performance benefits). WSE proxies, on the other hand, are disposable non-threadsafe objects that incur a setup time with every remote method invocation (even when using Secure Conversation). This also makes them largely unsuitable for Dependency Injection and many other widely-used patterns.
These are just some of the reasons why you shouldn't use WSE anymore. The reasons why you should use WCF on the client side are manifold, including but not limited to separation of the model and proxies, consumption of REST-based services, and better handling of collection types.
Unless you really must continue to use ASMX, please reconsider your refusal to move to WCF - unless the service does a lot of unusual things with XML serialization, it takes no more than 5 minutes to make the switch.