Boobs are a sweet sight … as long as you’re not looking in a mirror. Unfortunately, up to 65% of men deal with some degree of “gynecomastia,” or an excess of male breast tissue, according to a recent study from the Baylor College of Medicine.

“We call it the silent disease,” says Rod Rohrich, M.D., a board certified plastic surgeon at the University of Texas Southwestern Medical Center and the Dallas Plastic Surgery Institute. “A lot of men deal with this, but they don’t talk about it, so other people aren’t really aware of how common it is.”

Gynecomastia often shows up when a guy’s going through puberty and his hormones are out of whack. “In many of those cases, it often resolves on its own, so we don’t recommend any surgery or treatment until you hit 18,” Rohrich says.

But man boobs can be a trickier problem when it persists—or strikes during a dude’s 20s, 30s, or beyond. In those cases, it’s likely not going away without surgery, Rohrich says. The good news: It’s a simple same-day procedure, after which you should be rid of your breasts for good, he says.

But how do you avoid man boobs in the first place? Keep Reading.

Body fat is metabolically active, and has been shown to drive up a man’s supply of the female sex hormone estrogen while simultaneously tanking his supplies of testosterone. This hormonal swing can fuel gynecomastia, shows research from the Mayo Clinic.

A healthy diet and regular exercise can help you avoid excess fat, Rohrich says. Keep in mind: It’s not just what you eat but how you eat that contributes to binge eating and weight gain.

Anabolic steroids can mess with your hormones in ways that promote man boobs. “I see this a lot in weight lifters and ex-athletes,” Rohrich says. His warnings extend to over-the-counter supplements—those mass-building pills marketed as hardcore performance or strength boosters. “Even if you have a procedure to remove the excess tissue, it could come back if you take this stuff,” he says.

Soy-based foods like tofu are a great source of protein amino acids. But there’s some research to suggest soy may up a guy’s levels of the female sex hormone estrogen, while simultaneously knocking down his levels of testosterone, according to a study from the University of Minnesota. Again, this rise in estrogen and drop in T could drive the growth of breast tissue.

Many products contain soy. So it’s unrealistic (and probably unnecessary) to avoid the stuff altogether. But if you’re dumping soy milk on your breakfast cereal and into your coffee, you might want to switch to regular milk, almond milk, or other non-soy alternatives, the research suggests.

While it’s inconclusive, research going back decades has linked a heavy marijuana habit to gynecomastia, Rohrich says. Chemical compounds in pot can interact with your body’s estrogen receptors in ways that may drive the formation of man boobs, the studies suggest. “If you’re smoking a lot of pot, you’re at risk,” he says.

Unfortunately, Rohrich says gynecomastia is often genetic. “If your dad and grandfather had it, chances are you will to,” he says. If that’s the case, you can’t do much to prevent it. But, again, a procedure to remove the tissue should resolve your problem, he says.