I felt that sometimes, closer objects made my head hurt, so out of curiosity, I closed my left eye then my right eye. I expected them to line up on the horizontal axis, but they were offset vertically by a significant amount.. many times it was 45 degrees and sometimes even closer to vertical, instead of horizontal.
Keep in mind you have to make sure your head is absolutely level, and if there is anything in the background that isn't exactly aligned, your brain will be comparing the foreground object to that. You can try this out in real life.
Again you have to make sure your head is aligned with the home screen orientation (e.g. hold back and select reorient, then try this test without tilting your head anyway except up/down).
After more testing, it seems like sometimes it is calibrated well, if I take the phone out and re-insert it, the stereo alignment is a lot better. But it took several times of re-inserting to get a good one. 3/5 times it's bad for me.