Turned away from an apartment building

The area I deliver in is a particularly bad slice of NY, so customers are almost always apprehensive when I buzz them. Today I had my first ever denial. I buzzed the apt, customer replied "Who's this", I reply Amazon, he replies " no, no thank you".

I of course assume there was a language barrier, so I call him, he picks up, speaking English clear as day and before I can say anything he says "no thank you" and hangs up. I was literally stunned, years of doing this and I've never had this. Anyone ever run into a customer like this?