Motor vibration may not be the only problem here.
Orientation estimation can go wrong due to multiple factors like,
- Bias due to incorrect calibration. Keep the sensor level and idle. Make sure Gyroscope reads close to (0,0,0) on average. Accelerometer should read either (0,0,9.81) or (0 0 -9.81) m/s^2 depending on the convention.
- Bias due to temperature changes. Even a 10 deg change in PCB temperature can change the bias in Gyro by 0.3 dps (according to the datasheet)
- Motor noise. Seems like you have already tried reducing this one.
If none of them work, you could try implementing your own Kalman or complementary filter based on the raw data from Gyro, Accel and mag. This way you can be sure about calibration process, estimator gains, how the estimator works.
If implementing Kalman filter is difficult, you could try this AHRS filter block/algorithm given here,
https://in.mathworks.com/help/nav/ref/ahrsfilter-system-object.html