Yeah, I can also associate some songs with colors or memories, but it's more like I mentally visualize them, but I never actually get to see the color or feel the smell at all, so I can clearly say I don't have synesthesia at all.
But how can it make sense? I'd guess it's probably related to neurone synapses on the brain cortex and hippotalamus, related to sight (occipital lobe) and hearing (temporal lobe). It's rather complex, and I confess I've hardly read anything about synesthesia before.
EDIT:Oh, I found an interesting, actual (2007) article (but I can't guarantee on the validity of the informations though) that talks about possible hypothesis of what could happen in the brain of those who have synesthesia:
http://serendip.brynmawr.edu/exchange/node/189