If memory is not a problem, why not use a std::vector of some custom type? You could have the fastest access times, since all elements are in order, and just save a flag if an element is removed. Consider this proxy class:
template<typename RealType>
class MyProxy {
public:
RealType instance;
bool isused;
MyProxy(RealType) { /* TODO */ }
};
Which is then used within your std::vector:
std::vector<MyProxy<MyRealType> > vec;
For a lookup, you just have to check whether the index is within the bounds of the std::vector and that the isused flag is true. For removal just get the element of index i and set the flag to false. For insertion you have to push_back a new instance of MyProxy<MyType> instead of MyType.
The drawback of this approach is of course, that the std::vector is constantly growing, so you have to keep an eye on memory consumption and eventually freeing the vector, but this is possibly the fastest method for lookup, insertion and removal.