I'm glad there's a person speaking from recent experience. When I went to Japan the first time, my family and I were initially under the impression that there'd be a lot of people who could speak English but we were wrong. When we wanted to find a street (their address system is somewhat odd) there was a lot of pointing and gesturing. Realistically, though, you can get around without knowing the language.

Also, regarding use of sporadic English and the possibilities of fluency: think of it this way, it's just like how so many non-Japanese speaking members here use phrases like "kawaii", "sayonara", "konnichi wa", "ohayou", "sugoi". You know the basics and possibly say it perfectly but that doesn't mean you can converse, right? It's just like how many people memorize Ayu's lyrics & the translation.