The Virgin Megastore is in Times Square, which sells a fair amount of foreign pop music. The Japanese selection is kind of sucky, though. I wouldn't rely on it to find anything worthwhile.

Other than that, there is a Kinokuniya by Rockefeller Center (49th-ish, between 5th + Ave of the Americas).

I think there is a Book-Off, too. I vaguely remember finding it on a side-street somewhere, but I'm not positive. I don't think they had so much music as books, though (unlike the Book-Off stores in Tokyo -- music up the wazoo).

Edit: Yes, there is a Book-Off! It's right near Grand Central, if that's how you're coming to the city. It's on 41st between 5th and Madison, right across from the NY Library.