Not interested in attending bf family gathering?

My bf's family has been pretty sh*tty to me for the past year that we've lived together. Previously his mother lived with us and she wanted to be queen and run him and me and my son, and with all respect I told her that was not happening. Things got completely out of hand and she even tried to fight me, I had to call the cops on her, because my bf repeatedly said he'd told her to leave me alone but there was nothing else really he could do (except put her out-duh). Every time something happened she went back and told his dad which made me look bad, and his family as a whole (mom, dad, brother) refer to me as an outsider, literally they never call me by my name but instead "the outsider". He has gone to his father's house at least 10 times since we have been living together and not once have I gone with him, now his fathers sister is having a family gathering for the fourth. I don't want to go. My family has however been the exact opposite to him, and he's gone to thanksgiving, xmas, birthday dinners, everything everyone knows him (we've been together almost 3 years). Am I wrong to not want to go. I know he's gonna be upset, when he told me I said I didn't know, but I mean he's done a crappy job of making me feel included since we're supposed to be getting married in the future, and his family is very seclusive so I'd actually rather stay home. Especially with their being a chance of his mother popping up as he says she has at past family events.