Am I a bad person?

No, you're just stressing right now. When he does get you into a conversation, can you just steer him into a topic that might interest you both; like favorite foods, hobbies, etc.?