Go ahead and google it. If you search “emotional fidelity,” the engine will come back at you with: “Did you mean: ‘emotional infidelity’?” Nope, you didn’t.

In my twenty-five years of psychiatry and couple’s work, I’ve come to understand that emotional fidelity is the most overlooked aspect of relationship happiness. After the discovery of an affair, usually the focus is on sexual fidelity and blaming the partner who broke that vow. Yet the other component of infidelity, which I think looms larger and may actually be more destructive to relationships, is the challenge to the bonds of emotional fidelity.

Marriages often work best when both emotional fidelity and sexual fidelity are intact. So what exactly does that mean? Let’s start with the easier one to understand: sexual fidelity. It means that you expect your partner will not have sexual relations with anyone else unless you agree to it. Even in most nonmonogamous relationships, in which couples grant each other explicit permission to have sex outside of their marriage, many couples still expect that they will remain each other’s primary sexual partner.

Emotional fidelity is a similar concept. I define it as committing to keep your partner as the central person in your life—your main source of connection and your primary confidant. Think back: when you met the person with whom you fell in love, you already had parents, maybe siblings, and likely close friends and other loved ones. When you fell in love, this former stranger suddenly became your best friend, your closest connection. You sometimes put your new lover’s feelings even before your own. This stranger either took the place of your closest confidants or at least entered your inner circle of intimate relationships. As you fell in love, you talked every day. You started seeing each other every night. If something big happened in your life, the first person you wanted to tell was him/her. If something bothered you, theirs was the first shoulder you wanted to cry on. If something bad happened to you, you didn’t blame it on this other person; instead, you saw them as a source of support and guidance. Some might call this codependency. Regardless, this emotional connection is generally what leads the couple to declare, “We are in love.”

Please allow me to state this clearly: You did not fall in love because you had sex. You fell in love because of emotional fidelity and your intention to continue it.

Over the course of most long-term relationships, just as the sexual passion dies down, so too does this emotional connection. If you have children, your children (who are completely dependent on their parents) become more important (as well they should). A sick relative may also become more important as they need you more. Many people and activities can wind up taking the place of your spouse. A new job. Golf, basketball, or a card game. Aging parents. New and old friends who wind up getting way more attention and positive regard than your spouse. This connection to your spouse—the same person you once rushed to tell everything—fizzles. Instead of, “I can’t wait to see you,” when you walk in the door you might even think, “You again?”

This may be part and parcel of the extinguishing of the sexual flame, but I believe that the loss of emotional fidelity is the most destructive loss to the couple. The loss of emotional fidelity is essentially the failure to keep your promise of holding the other person in high regard and having their back at all times.

About half of people cheat simply because they can. They have no complaints with their marriage, and they report being happy with their spouses.

But many of those who are in the other half give reasons for cheating. I often hear, “My partner is just not that into sex anymore.” When I ask my therapy group of male cheating patients what a spouse can do to prevent cheating, “Give them plenty of sex, and they’ll never leave” is what they advise. Yet the more common reason that my patients, male and female alike, cite as why they cheat is because they don’t feel that they can count on their partners. That special emotional connection they once shared has all but vanished.

Excerpted from Infidelity: Why Men and Women Cheat by Dr. Kenneth Paul Rosenberg. Copyright ©2018. Available from Da Capo Lifelong Books, an imprint of Perseus Books, LLC, a subsidiary of Hachette Book Group, Inc.