I never left my headset sitting in the sunlight, but I have the exact issue. I'm 6' 4" and have long arms. If I play a game like Beat Saber, it loses track of the controllers constantly. Only games where you keep the controllers in front of you all/most of the time do I never have issues.
Other than the problem of the head strap being too small for me(and breaking twice), this was the 2nd reason I ended up switching over to a Valve Index. Using the external sensors on that headset, I never have any tracking issues at all. I had hoped Oculus would make external sensors for the Quest 2, but it doesn't look like it's going to happen.