Should I gain muscle or lose fat first?

Okay so long story short, I was a fat kid. Then I lost the majority of the weight and Im fairly healthy now but Im still a little flabby. (Don't worry not like disgustingly fat) Everytime I say i need to lose some extra weight or go to the gym people say I'm crazy. I think I just have some fat where muscle should be, ya know like overly fat underly muscled kinda deal. So what should I take care of first?