For me this was the main reason which kept me from going vegan I have to admit. That I’d always have to justify myself for something you seriously shouldn’t have to justify for at all.
And yes, at first I got reactions like „this is so exaggerated“, „but what are you allowed to eat now?“, „isn’t that unhealthy?“, „Vegan? Oh come on..“ etc. But I think if you really stand up for something you believe in it’s not a big deal. If people won’t leave you alone just say something like „Please let’s not talk about this anymore. I respect you and your choice, please also respect mine.“. It’s also helpful when you know what you’re talking about, for example the facts why veganism is a good thing and where you get all your nutrients from.
And don’t worry about being impolite when refusing food someone offers you. I know it can be difficult when someone for example made a cake and you don’t want to hurt that person’s feelings. But you can just let them know that it’s nothing personal and that it looks good and that you’re sure it also tastes great. Hopefully they will understand.