Me too.
You aren't going mad.
What a bizarre bug - and it's very very annoying. You are right the only way to exit the playback is via long press which takes you all the way out.
I noticed this a little while ago.
I just rolled my eyes and added it to the list of irritations on the NOW app on AppleTV. FF/RW are horrendous and now you can't even back out of the app cleanly!
Hoping they fix it but not holding my breath.
