Firstly, well done on experimenting to find the best structure for your application. Often people will argue without trying out various options to get real performance data.
If you want to save build time, and your words file does not change very often, the obvious build speed improvement is caching the data structure. Whatever data structure you are using, build the structure once, and then store the structure to the SD-card (rather than just storing the strings). Standard java.util structures can be stored using Serialization.
If you want fastest build time, and your word list is sorted in alphabetical order, or can be, then you could just store in a String array. Build time will be very quick again, and search time will be similar to a TreeSet (using Arrays.binarySearch()).
If you want faster lookup, you might want to check out Perfect Hashing or Tries, but these aren't in the Java standard libraries.
A trie will be much more memory efficient than either of these, which may make it quicker. (Information on finding an implementation)
I'm surprised TreeSet is faster than HashSet in your experiments, which means that you might be operating in a situation where memory allocation is expensive. Did you remember to set the initial capacity when you allocated the HashSet? Remember to avoid an expensive rehash, you need to set the initial capacity to at least number of items/0.75 (the load factor).