How do get bigger in body size?

Im an 18 year old man, My height is 5'7 or 5'8, (i forgot) i weigh about 162.42 Pounds, about muscle and a little bit of fat in my ab area especially i even have love handles (fat on my hips), but ive been like this for quite a while now, my weight some times goes up and down, i go to the gym about 3 to 4 to even 5 days a week, i awlays lift heavy weights, i work on all essential muscle groups usually on different workout days i do, i sometimes workout at home by doing pushups and other excercises regarding to it,, situps, and all kinds of pull up excercises but im still like this weight, and dont feel like im getting bigger, im getting stronger but not bigger though and when i see other people, theyre the same age but bigger than me! I dont get it?? Even sometimes the same age, i usually take protein powder, i take any kind like muscle milk, serious mass etc. But what food i eat, i eat around 1500 to 1750 calories per day almost sometimes 1250 per day, i usually eat around 34 to 65 grams of protein, also tryin to lose this fat on my belly to get rid of the love handles by running and swimming, but i also need to gain more muscle to get bigger, what should i do? Does it have to with how much i consume calories or protein? Or how many days i work at the gym? Or protein powder? PLEASE HELP ME!!!!!