In the west, If you listen to the song "girlish" your friends will surely think you're affeminate, but if you listen to a song like "About You" people will surely think you're weird because the song is in japanese. lol.
There's no scape
I'm a simple straight old guy and I listen to Ayu because her songs make me feel very good. Her music makes great company. Her songs keep me awake when I'm working late or way home in the bus while all the guys listening to metal-hiphop-reggaeton or whatever already fell asleep.
The most of the messages in the lyrics are touching and inspiring for both genders, so, Ayu's songs works very well for male or female ears.
BTW, Ayu is hot... Feel myself as affeminate is impossible, specially if you find yourself watching the Sparkle PV with your mouth wide open

or looking for a woman at least 10% as fascinating than Ayu.
The main concern for us wouldn't be if the fact of listen to ayu makes us look affeminate or not. IMO the important think is if we are free-independent-open minded guys or like little-kiddy-chickens, affraid to be ourselves no matter what people might say.