Is it bad for a guy to look really young?

I'm 23 but people tell me I look 19 or 20 all the time- I'm getting tired of being called "cute" and "innocent" and people always saying "awww" and that I have a "babyface." I'm 6'2' 150lbs, so I'm actually really skinny and scrawny for my age I guess. Would gaining 20-30 pounds and a little muscle help me to look older and more mature? And are most girls turned off by a guy that looks really young?