There are a variety of reasons why people gain weight: a lifetime of unhealthy habits, lack of motivation, depression, body-image issues, hormones, injury, or illness.

Those same reasons (and more) can keep you from losing weight as well. One in particular can be harder to overcome than others: fear of losing weight.

Anyone who has tried to lose weight — and failed — knows that mental roadblocks can play as big a role as diet and exercise in your success.

The Psychology of Losing Weight

“If it was only a matter of ‘calories in and calories out,’ everyone would be losing weight,” says Elizabeth Lowden, M.D., a bariatric endocrinologist at the medical weight-loss program of Northwestern Medicine in the Chicago area. “But there’s a psychological component to losing weight for most people.”

Lowden’s assessment of weight-loss patients includes find out how how they gained it, what has tripped them up and kept them from losing in the past, and what about losing weight scares them, “to find out what’s standing in the way that we need to deal with,” she says.

“Weight is intertwined in people’s sense of security and self-confidence,” says Matt Traube, Ph.D., a psychotherapist in Southern California. “So it has a far-reaching emotional impact in a lot of important areas in people’s lives.”

Burying your fears or dismissing them as silly could sabotage your plans to get healthier. The first step: Identifying what could be holding you back. Here are the most common reasons people are afraid to lose weight.

Fear of the Unknown

There are a million wild cards related to weight loss, especially if you’ve been on the heavier side most of your life. Some fears concern the physical.

People worry about what the “thin version” of themselves might look like: Will my face look too gaunt? Will I be left with extra loose skin if I lose a lot of weight?

But many fears are psychological: “We all have a baseline understanding of what normal is, or our norm,” Traube says. “It’s what we know and what we feel comfortable with. Sometimes even when we know our current behavior or physical characteristics aren’t the healthiest, changing that is scary.”

A big weight loss is a major life change, which can be frightening. “You might wonder, for example, ‘What would life be like if I were skinnier?’” says Traube. “Not knowing the answer can provoke anxiety.”

Fear That Your Relationships Might Change

We’ve all seen that rom-com where the overweight girl loses the extra pounds, gets a makeover, then gets mean-girled by “friends” for shedding her role as the less attractive sidekick.

It’s a common trope for a reason: Consciously or subconsciously, people develop expectations of friends and family members, which include elements of people’s appearance and self-esteem. Partners, too, or even co-workers, might feel threatened or resent your weight loss, particularly if they struggle with weight themselves. They might feel abandoned when your progress surpasses theirs.

“People get used to you looking a certain way or being a certain way, so changing that could make others feel betrayed or jealous,” Traube says. Even if loved ones are genuinely happy about your weight loss, there might still be a fear that people in your life just won’t view or support you the way they did before, adds Lowden.

The possibility of new relationships after a weight loss can inspire fears, too.“Extra weight can be a great defense mechanism, because it provides an excuse not to have to face uncomfortable new experiences such as dating or establishing new friendships,” Traube says.

Fear of Unwanted Attention

Extra weight also can create a protective barrier between the person carrying it and people in the world who can make you feel vulnerable.

“Clearly, the amount of attention women get in our society is significant compared to the attention men get,” Traube says. “And unfortunately, what’s said on the street isn’t always respectful toward women, which can be scary.”

A lot of women might not relish the idea of eliciting attention from strangers on the street, but it’s a more common fear and a greater source of anxiety for women with a history of abuse. After a trauma that made someone feel vulnerable or taken advantage of, extra pounds can be comforting because a person might feel he or she generates less attention at a heavier weight.

And although the possibility of more people expressing sexual interest in you might be a reason to lose weight, the flip side is that it also might raise the risk of having to deal with uncomfortable and unfamiliar experiences in the dating world.

Fear of Failure

“People often tell themselves, ‘If I’m skinny, I’m going to find the love of my life,’ or ‘I’ll be successful in my career once I lose weight,’” Traube says. “It’s a typical cognitive distortion, thinking, ‘I just need to change this and everything will be better.’”

The idea that you could do all that work and then still have many of the same problems can be terrifying, he says. “It can be scary to have to confront the thought that it wasn’t the weight, it’s you.”

Still others fear that if they lose a noticeable amount of weight and gain it back over and over again, people will criticize or ridicule them.

Fear of Giving Up What’s Comfortable

Weight gain is interwoven with all sorts of coping mechanisms. It’s extremely common for people to hide behind their weight or eat to make themselves feel good, no matter what their level of actual hunger.

Eating is a common way to quell anxiety and depression, so when you lose that effective coping mechanism, it can make you feel anxious, Traube says.

The snack-as-security-blanket habit is one of the most difficult for many people to let go of, he says. Also common for introverts who struggle with weight is using it as an excuse to stay home and isolate themselves.

“If you have social anxiety, weight is a wonderful way to keep you at home,” Traube says. “This is a big reason why losing weight is scary for a decent percentage of people.”

Fear of Embarrassment

Losing weight often involves “putting yourself out there” literally as well as figuratively. Most doctors advise people trying to shed pounds to get more movement into their days, starting with walking or easing into a workout routine at home.

People comfortable flying under the radar might feel anxious when experiencing such exposure, Lowden says.

Want to fight your fears? Here are some steps to take to kick your fears out of the way and get healthier, starting now.

How to Overcome Your Fears

1. Start Small

Sure, it’s good to set a goal like dropping a set amount of pounds to get down to a healthy weight. But it’s just as important to identify smaller goals that will set you on a steady, long-term path in the right direction.

“I’m a firm believer that small steps equal big progress,” Traube says.

“When the bar is set too high, it can create paralysis in people so they don’t even want to get off couch and start.”

“People can sabotage themselves by jumping into something when they aren’t ready yet,” Lowden adds.

When we push too far and too fast mentally and physically, it’s hard to stay motivated, Traube says, because who wants to do something that’s so difficult? He suggests to patients to start with very small goals, such as making healthier shifts in diet habits, and build from there.

“I describe it as reverse-engineering a process and tell them to give it some time,” he says. “Don’t expect to change everything in a weekend.”

2. Log Your Fears

You’ve heard this advice before — because it works: Keep a journal, even if you’re already tracking calories, Lowden says. Focus on how you feel while you’re eating, what you eat how you feel before and afterwards.

After a few days, you might be able to identify some patterns, such as heading for a snack after a stressful work call or meeting, or starving yourself until your kids go to bed.

“You could discover an emotional component to your eating or that maybe you don’t have much time to yourself and need some more ‘me time,’” she says.

When you’re conscious of what’s happening on an unconscious level, you have more control of it, Traube says: “Once a fear becomes a tangible idea rather than something you can’t put your finger on, you can begin to make choices over it.”

3. Seek Out Support

Much like writing things down can help you identify patterns and issues, talking about them also can help bring them to the forefront. Support groups at weight-loss clinics or online can be great resources, Lowden says, adding that apps and websites have the added bonus of anonymity, so you can share thoughts and fears without embarrassment.

4. Plan for the Long Term

You might lose weight fast by drastically restricting calories, but once you reach your goal weight, you’ll most likely return to old eating habits and gain all the weight back.Losing weight in a healthy way might take longer, but your chances of establishing healthier changes for longer are much greater.

“Having a plan to maintain weight is an important but often overlooked aspect of weight loss, Lowden says. “If people can identify what trips them up and stick with the diet humans are supposed to be on, rather than radically reducing calories, they have more long-lasting success,” she adds.

Conquering your fears can be daunting proposition, but by figuring out the root causes and employing smart strategies, you can move closer to your goal of losing weight and getting healthy.