If I understand correctly what you basically want is for the animated layer to decelerate and stop along the same vector as it's moving when you tap it. If you have the position when you tap it and the transform when you tap it I think you can find out the endpoint by doing the following:
CATransform3D transform = <Your layer's transform goes here>;
CGPoint startPoint = <Your layers's current position goes here>;
CGFloat distance = 40.f;
CGPoint v = CGPointMake(0, distance);
CGAffineTransform affineTransform = CATransform3DGetAffineTransform(transform);
CGPoint offset = CGPointApplyAffineTransform(v, affineTransform);
CGPoint endPoint = CGPointMake(startPoint.x + offset.x, startPoint.y + offset.y);
P.S. - This only works if you don't apply any scaling, skewing etc. as part of your animation (i.e. the transform should only represent rotations and translations)