Hello,
I have over 3 years of professional experience in development of software similar to your project - I was tasked with development of autopilot and IMU devices for a small UAV. I would be glad to assist you.
Before I am able to suggest anything algorithm-related, I would need to know, if the rotation matrices you are talking about contain information about the angular velocity of the object or its actual orientation in the global frame. Would you consider using quaternions for internal calculations inside the filter? They can be calculated based on the provided matrices, their main advantages are a bit smaller computational requirement and lack of the gimbal-lock phenomenon.
I look forward to hearing from you.
Best regards,
Piotr Juda