I found a way. It's not ideal, but it does work in my testing and for my case, but it uses the private NSSegmentedCell method:
- (NSInteger)_segmentHighlightState:(NSInteger)arg1;
This method returns 1 for unselected cells, 3 for unselected cells while they are pressed during tracking, 4 for selected cells, and 5 for selected cell while pressed during tracking.
So testing bit 1 (mask with 0x02) gives the desired "is pressed" state. Sadly this is useless if you want to ship on the Mac App Store, but since Keyboard Maestro is excluded from the Mac App Store because of the sandboxing requirement, that no longer concerns me.
It is still worth defending against any possible changes, with code like this:
- (BOOL) mySegmentIsPressed:(int)segment;
{
NSInteger highlightState = 0;
if ( [self respondsToSelector:@selector(_segmentHighlightState:)] ) {
highlightState = [self _segmentHighlightState:segment];
} else {
highlightState = ([self selectedSegment] == segment) ? 4 : 1;
}
return (highlightState&2) != 0;
}
The startTrackingAt method @arun.s suggested doesn't work because NSSegmentedControl uses a different and unknown tracking area )and besides, even rectForSegment:inFrame is a private method so you can't even really find out where the cell is without using a (less) private method.
I'd rather a documented way to do this, but short of implementing the entire control afresh, I don't see how.