I've got the same issues so no you aren't alone and I know how incredibly frustrating it is. I got mine mostly solved by putting some easy liner in the battery compartment to stop the battery from sliding around. It seems like after playing a lot of beat saber the battery spring wears out from all the G's on it compressing it over and over which is why the tracking gets worse and worse as it's getting easier and easier to move the battery.
Check this out - https://imgur.com/a/QrfrZ
I'm also personally experimenting right now with my 3 sensor setup by aligning in a diagonal in my square room so my face is looking at a sensor and I have one on each side of me so there is always at least 2 cameras on each controller. It seems better but I killed my arms a couple of days ago and haven't been able to properly run it through its courses just yet lol.
Other solutions involve putting foil between the spring and the negative terminal but I find it to be a bit unreliable. Let us know if you find something that solves.