This is something that always sits badly with me.
I literally don't have a jingling pocketful of cash, and when I'm on my way somewhere I'm almost always in a hurry.
I am pretty empathetic/compassionate and feel like most homeless people deserve help, even if they use my help to get drugs - because sleeping rough sucks without some form of comfort, and no one chooses to end up there.
But I find myself bowing my head and muttering "...sorry" under my breath. I hate it. I need to do better.