Not inviting children

I have spent hours planning my dream wedding. Now all the little details are in order I am left thinking about guests and children. My dilemma is that my niece is going to be a flower girl and is a very quiet child, but I do not want to invite some other children in our family, particularly the ones who terrorised the last wedding we attended! My fiance and his family think we have to invite them but my mother and I don’t agree. I find myself torn between doing what I want for ‘my day’ and keeping people happy! I need help.