more generally, should classes which differ only a little in the domain model, and only in terms of behavior, be represented via inheritance in the implementation, ...
Behavior is not "only". Behavior is most important part of objects.
or is it better to simply push different kinds of behavior into the parent class?
Certainly not. That would be violation of Liskov substitution principle.
Inheritance should not be used for "easier access to common stuff".
Content of parent class muss be completely ubiquitous with child classes, avoid inheritance and use composition if child does not comply w/ it.
more generally, should very simple classes in the domain model be given their own classes in the implementation - e.g. for extensibility - or should that be pushed into the container class?
It kind a depends on how "deep" Your logic is going to go.
Usually it's a good idea to start decomposition only when you hit some limits.
Some people call it evolutionary design.
E.g. - I have "class must not be larger than ~200 lines of code", "object must not talk with more than 5-7 other objects", "method should not be larger than ~10 lines of code", "same logic should be written once only" etc...
Also - it's easy to underestimate semantics. if order.isOverdue() is much more easily readable and understandable than if order.dueDate<date.Now(). Introducing classes that reflect concepts of domain greatly "humanizes" code - increases abstraction level (think "asm vs java").
But decomposition for the sake of decomposition leads to unnecessary complexity.
It always must be justified.
Should Transition be its own class in the implementation and, if so, will I need to subclass it in order to support each kind of automaton (since, besides differing in terms of the state, they also differ in terms of what happens during transitions)?
There is nothing wrong with that as long as it complies with Your domain.
Creation of abstractions is artistic activity that highly depends on Your domain (e.g. advises from experts at formal languages & automata might be severely over-complicated if requirement that Your code is supposed to be as a teaching tool for a course is forgotten).