You can create detectors for each shape you want to support. These detectors will tell, if a set of points form the shape.
So for example you would pass 4 points to the quad detector and it returns, if the 4 points are aligned in a quad or not. The quad detector could work like this:
- for each point
- find the closest neighbour point
- compute the inner angle
- compute the distance to the neighbours
- if all inner angles are 90° +- some threshold -> ok
- if all distances are equal +- some threshold (percentage) -> ok
- otherwise it is no quad.
A naive way to use these detectors is to pass every subset of points to them. If you have enough time, then this is the easiest way. If you want to achieve some performance, you can select the points to pass a bit smarter.
E.g. if quads are always axis aligned, you can start at any point, go right until you hit another point (again with some thresold), go down, go left.
Those are just some thoughts that might help you further. I can imagine that there are algorithms in AI that can solve this problem in a more pragmatic way, maybe neural networks.