As many of you have already stated, there is nothing like Utena (or if there is, I haven't stumbled across it yet). I've seen some of the anime recommended above and imo nothing comes even close. I've just finished watching Princess Tutu and I'm not a fan, I rather enjoyed Madoka but I don't get what's all the fuss about it, I've yet to pull through Simoun because the first episodes drove me away with their pathos, same for Evangelion, where the first episodes drove me away with their techno-babble.

But I've just started watching Star Driver (from one of the Utena writers) and so far I truly enjoy it a lot...

It's not an anime but the manga Kaguya Hime reminds me of Utena a bit with the mentions of incest, homosexual relationships, nightmare fuel, tomboy girls, girly bishonen, with both garnering a bit of same sex crushes, and mind screw elements.