We found that upgrading Automapper (to version 6.0.2) would be the way to go in our case. Using the updated AutoMapper and the same objects and mappings listed above we saw the ExampleDTO->Example object mapped in 1.57 seconds, and the reverse case mapped in 1.86 seconds. I'm not overly happy posting an answer that says use the upgrade, so I'll post a few options that gave some modest gains, and if anyone else has an actual answer, I'll gladly mark that one.

I tried creating a mapping for the ISet HashSet and this was about twice as fast as without that specified mapping, I cannot remember exactly what I did here, but I found it on the googles.

The other option I tried, was creating Non Mappable HashSets on the DTO object that simply returned the ISet. This was about 3x faster, but still not close to the performance gained by upgrading.

class ExampleDTO
{
    public int Id;
    public enum Type;
    public DateTime creationTime;
    public ISet<string> StringThings;
    [IgnoreMap]
    public HashSet<string> StringThingsHash
    {
        get
        {
            return StringThings;
        }
    }
    public ISet<int> IntThings;
    [IgnoreMap]
    public HashSet<int> IntThingsHash
    {
        get
        {
            return IntThings;
        }
    }
    public ISet<double> DoubleThings;
    [IgnoreMap]
    public HashSet<double> DoubleThingsHash
    {
        get
        {
            return DoubleThings;
        }
    }

and I used the following mapping

CreateMap<ExampleDTO, Example>()
  .ForMember(dest => dest.StringThings, opts => opts.MapFrom(src => src.StringThingsHash)
  .ForMember(dest => dest.IntThings, opts => opts.MapFrom(src => src.IntThingsHash)
  .ForMember(dest => dest.DoubleThings, opts => opts.MapFrom(src => src.DoubleThingsHash);
CreateMap<Example, ExampleDTO>();