It depends how loud the music is, and for how long you listen to it. Continued exposure to noise at or above 85 dB(A) over time will cause damage to your hearing and warning signs will include ringing in your ears or a 'dull' quality to your hearing eg. after attending a rock concert. Sometimes there are no symptoms at all, but you could still be damaging your hearing.
As a rule of thumb, if music makes your ears hurt, turn it down, go into a different room from the music, put ear-plugs in or turn the music off completely. If you can't hear what people around you are saying, then that's a sign you are in danger.