If you are going to take a short break, before removing your headset, try pressing your Quest power button and put it to sleep. Then when you are ready to play again, put your headset back on (sitting as close as possible to where you left off), and press the power button again to wake up your headset. Hopefully your Guardian will be where you left it.
I have this same issue with certain rooms. Sometimes I would have to turn it off and on for it to remember a guardian, and it's like it's not even trying to remember the previous guardian when I just look around or turn sleep mode off and on. Right now in my room, it's completely forgetting the guardian every time it goes to sleep.