You can be the most enlightened, loving, open-minded person on the planet and STILL get really annoyed by some people. Don't feel guilty about your feelings and don't try to internalize them in order to keep the peace. There are three very positive and understandable reasons why you can be a beacon of spiritual growth, and yet, not like everybody, at the same time...
1) Your Higher Vibration
The challenge you may face with certain people is that they vibrate differently from you. It is a physical reaction that can feel like the shrill of nails scraping down a chalkboard. It is energy that the other person is producing that physically makes you uncomfortable. In turn, this is also why you automatically "click" with certain people right away... it's because you are vibrating on the same level.
Go by your feelings and intuition. It's okay to not want to be around particular individuals, sharing the same physical space with them. Removing yourself from the room is always an option.
2) Your Personal Power
The difference between you and another person can be a matter of personal power. Power lies in our words and actions in the present moment. If those around us are living in the past or the future, then they may easily be an annoyance. When a person is stuck in the past or constantly worried about the future, they have a tendency to moan and groan and complain about their life and take on the role of "victim."
However, when you use your personal power, you instinctively know that you are being given two options: the opportunity to speak up and take a positive stand in order to change something you don't like, OR the chance to accept whatever it is and move on. On the contrary, those individuals who come from a place of raw, uncontrolled emotion, get nowhere other than to upset those around them and pull more negative energy in for themselves.
3) Your Positive Vision
You are more than likely a free-thinking, non-discriminatory, non-judgmental person who is open to possibilities. You know that positive thoughts create positive results, while negative thoughts create negative results, and that, you always have the power to choose. So when you are presented with the views of another that are negative and closed-minded, recognize that they are choosing that life experience for themselves... it has nothing to do with you. All knowledge is a form of light, and they simply aren't able to see the bigger picture just yet... it's all about awareness.
So remember, next time someone says or does something that clearly annoys you, do yourself a huge favor and take it with a grain of salt.