Being assertive and feeling guilty

[Halo Effect]

There are 7 billion people in this world. They all have their own wants and needs. The moment you place the needs and opinions of all others above your own you voluntarily give up any power you have over your life and happiness.

Something that might help, Guyintheback, and I'm curious for your answer:
Think about this very problem from the perspective of the annoying guy you said no to. He annoyed you and made you feel awkward for weeks. Did he feel bad? Did he feel guilty or apologize? Did he accommodate you in any way? How would you feel and act in his spot? Visualize it.