For me, it's really easy to remember that one rem is sixteen pixels. It's then really easy to do the math based on that: half a rem is eight pixels. A quarter, or 0.25, is four. An eighth, or 0.125, is two. Sure, it took me a couple minutes to commit the full range to memory, but now I know without having to think about it what 0.625rem's going to give me (five-eighths of a rem, or ten pixels (a mere eighth less of a rem than 0.75rem, which is twelve pixels (itself a mere eight-of-a-rem step away from 0.825rem, or fourteen pixels))). These are fixed values. They're in my brain now and I keep them on the tip of my fingers and rattle off anywhere I need when I'm concerned about spacing or sizing in a design context. It's kind of a superpower.
Trying to keep track of wht an em is in any context gives me a headache. I like the idea of "let me give this a little more of whatever or a little less of something depending on the local context" but in practice if I'm going to say "give me half of something" I want to know what the something is. Like, definitively. Half of something in one place might be way more than I want. In another place it might be too little. It can also completely change depending on the local context of my local context. I don't have time for that. I'm a busy man. I have children to make dinner for.
It's like, if you told me that I can measure everything in inches (or centimeters, or cubits, or whatever) but that I'd get a different result for what an "inch" is based on whether I'm measuring it on my desk or in my garage or on top of a ladder in my neighbor's backyard...I don't want to carry that many tape measures around with me. I want one tape measure. That measures in inches. Neatly dividable into half inches and quarter inches and eighth-inches...