I agree with Father. And it's not unique to straight single men. Try going out in public with your 5 kids.
It's not persecution, though it can feel like a burden. (I don't mean the kids...I mean the looks and the comments. In fact, even the positive attention, which we do get sometimes, usually from sweet older ladies, can be a bit of a burden.) We've all got stuff to deal with.
I think after three is when the "line" is crossed where people think it's OK to start giving looks. Unless you're downtown where no one has more than two and heterosexuals are often outnumbered. But I agree with Father. Looks and comments are one thing. Having words with someone is another. And actually being persecuted is still another.
Having met my children I would rather bear the cross of the looks and comments than the cross of never having known and loved them.