There’s this pervasive idea that transgender people aren’t actually the gender that they outwardly express. Men without penises can’t be men, and women with them certainly aren’t women. Or are they?

Despite what people might think, our body parts don’t actually have much to do with our gender. So regardless of what’s happening in someone’s pants, straight people are still straight—even if their partners are trans.

First, some biology. One of the major reasons that people tend to think that all transgender folks are gay is the idea that sex and gender are the same thing. And that your sex, or your sexual organs as they appear on your body, are what determine your gender.

“Gender can be conceptualized as an inherent or a deep sense of self regardless of our body parts,” says Dr. Reece Malone, a sexologist and sex therapist. “I often talk about how people can lose a limb due to illness or injury, however, that doesn’t change their gender identity. In other words, if a man loses his prostate due to cancer, it doesn’t make him less of a man. We’re still our gender regardless of the body parts we have or don’t have.”

“While labels can be important in terms of feeling a sense of belonging and security, we often get too wrapped up in the rules of that label.”

And simply put, a person who is transgender is someone who has sexual organs that don’t align with their gender. Although we tend to think of gender as one of two (male or female), much like the Kinsey scale of sexual preferences, gender is a spectrum—with some people being planted firmly on one side or the other, but with many others somewhere in the middle.

Just because your outward body might not align with your inward gender expression, that doesn’t mean that a person is automatically gay.

“All of us have a sexual orientation regardless of what gender we are,” Dr. Malone explains. “Similar to non-trans people, transgender people may identify as heterosexual, gay, lesbian, bisexual, queer, pansexual, asexual or another sexual orientation altogether.”

So, what if you meet a woman, and it turns out she’s a trans woman? It doesn’t make you queer, says Dr. Malone—you’re still straight.

“Most straight men who have sex with, or are in a relationship with a transgender woman, [it’s] because he’s attracted to her and vice versa. Being gay refers to someone attracted to someone of the same gender. In this scenario, if he sees his gender is male and is exclusively attracted to women, and if she sees her gender is female and is exclusively attracted to men, then they are by definition straight unless they say otherwise.”

But it’s the way we teach sex education that creates the disconnect. Dr. Malone says that one of the first things he does with patients is ask them to process feelings around sexuality and gender, the hope being they overcome their fear of labels. “While labels can be important in terms of feeling a sense of belonging and security, we often get too wrapped up in the rules of that label,” he says. It’s cultural labels that keep straight cisgender people from exploring relationships with heterosexual trans people—missing out on an opportunity to fall in love with the potential person of our dreams.

“If we allowed ourselves more permission to be curious, to explore and to focus on mutual pleasure, we can become more empowered and in control in our sexuality. Unfortunately, aspects of our culture, stigma and sex-shaming prevent those from being their authentic sexual selves.”

It’s also important to figure out exactly why the idea of being gay is so terrible, and break down that bias as well. Because no one is asking you to do something you don’t want to, only to get rid of the labels that are possibly keeping you from building meaningful connections.