should your boyfriend br friends with you X?

So i am happy to say that i do have a boyfriend, he is prob the best thing that has happend to me in the longest time! He's is more then someone to kiss and hug, he is my bestfriend, that makes it so much better. Well we have been together for a while now and its been great!
But there is just one thing that bugs me so much.... He is like really good friends  my X of a year and 3 months..... 
I just can put my finger on it, i think its because at the end of my X and i relationship was the worst. All The name calling, controlling, threating, just all the  FIGHTING. I just strongly belive that he is a bad person. But even though my boyfriend now knows what has happend between him they are still friends. Or maby i just dont want my name to com up in one of there conversations. Like last weekend i had called him on his cell phone, i new he was hanging out with his friend i just wasent sure who, but right when he answerd he said ill call you back and practicly hung up. later i found out it was because my X was right by him and he didnt want to make it akward. I dont want to tell him who he can and can't hang out with because thats just not me. I have told him though that it really weird to me but they still hang out at least 2 a week.

should i just get over it and let them hang out.
or speak up?