“Harry is frightened and feels the only person who can do anything about it is his father,” a royal insider tells PEOPLE

Prince Harry’s desperation to protect his wife and kids has deepened his estrangement from his father, King Charles.

When the Duke of Sussex, 39, returned to the U.K. in May to celebrate the 10th anniversary of one of his proudest achievements, the Invictus Games, his father was notably absent from the ceremony.

The two were unable to meet during Harry’s visit, which the prince’s spokesperson attributed to the King’s “full schedule.”

However, sources close to Harry suggest a deeper issue. Harry wanted to discuss his ongoing battle for security, which he has been fighting for in court for more than four years and believes Charles, 75, has the power to reinstate. (Buckingham Palace will not comment on security provisions, but a palace source tells PEOPLE the notion that Harry’s security is in Charles’s hands is “wholly incorrect.”)

The family rift that first became public in 2020 when Harry and his wife, Meghan Markle, stepped back from their royal roles has been marked by blistering attacks and unresolved tensions. But for Harry, no single issue is a greater sticking point than that of security, multiple well-placed insiders in his circle tell PEOPLE in this week’s exclusive cover story.

Despite a positive meeting between the father and son in February following the King’s cancer diagnosis, their communication has since deteriorated. Those close to Prince Harry say the monarch no longer takes his son’s calls or responds to his letters.

“He gets ‘unavailable right now,’ ” says a friend of Harry’s. “His calls go unanswered. He has tried to reach out about the King’s health, but those calls go unanswered too.”

Amid the estrangement, Prince Harry remains deeply concerned for the safety of his wife and children, Prince Archie and Princess Lilibet, and has, according to sources, repeatedly asked his father for help.

“Harry is frightened and feels the only person who can do anything about it is his father,” a royal insider tells PEOPLE. Another source close to the situation says, “Harry is determined to protect his own family at all costs.”

Now the fight is boiling over. In the ITV documentary Tabloids on Trial, which aired in the U.K. on July 25, Prince Harry voiced his fears that negative press could provoke a violent attack on his wife.

“All it takes is one lone actor,” he said, citing these “genuine concerns” as his reason for not bringing Meghan, 42, Archie, 5, and Lilibet, 3, back to the U.K.

The Duke of Sussex has also taken the battle to the courts, where he lost his bid to restore his taxpayer-funded security earlier this year. (He has plans to appeal.)

Exactly what is stopping Harry from securing the protection he seeks is a matter of bitter contention. Constitutionally, the monarch has no governmental power in the U.K., and the power to bestow police protection lies with the Executive Committee for the Protection of Royalty and Public Figures (RAVEC), which operates on behalf of the U.K. government.

But Harry, whose offer to personally cover the cost of police protection was also rejected in court, feels that as King, Charles could intervene to ensure such protection is extended.

The issue has created an impenetrable wall between Harry and Charles. The conversation has now shifted from frustration to “complete silence” from the King, claims the friend.

Sources close to the Duke and Duchess of Sussex, who have established a new life in Montecito, California, say the palace is mindful of what happened when Princess Diana left royal life — and instantly eclipsed the institution.

“Charles’ fear is a repeat of the past,” the insider says. “When his wife divorced the institution, she took the headlines, overshadowed his work and became a global superstar. Seeing his son leave with his wife was not part of the plan.”

Safety has always been a cornerstone of why Meghan and Harry say they decided to leave. The Duchess of Sussex has cited negative press and the lack of royal support during her time as a working royal as reasons she contemplated suicide.

The couple “didn’t feel they had a choice,” says the friend. “The only option was to leave — for their sanity. I do think they thought if they got out of this bubble, there would be less focus on them.”

At the Sandringham Summit in January 2020, which decided their fate as non-working royals, Prince Harry emerged from the meeting — attended by the future King Charles, Prince William, Queen Elizabeth and palace staff — confident that their security would remain intact during their transition. Court documents also reveal that the Queen, who died on Sept. 8, 2022, at age 96, deemed it “imperative” for the Duke and Duchess of Sussex to have “effective security” after their departure. Despite this, Harry was informed just weeks later that their police protection had been withdrawn.

“The Queen made it clear that effective security was necessary due to the threats against them, but somewhere along the line there was interference,” the royal insider alleges.

The security downgrade was ultimately enacted by RAVEC. Its membership includes the Home Office (the U.K. version of Homeland Security), the Metropolitan Police and the Royal Household. Although the King has no direct involvement with RAVEC, his staff sits on the committee. A government spokesperson says they provide “rigorous and proportionate” security and do not disclose details “as doing so could compromise their integrity and affect individuals’ security.”

Harry, whose team notes that the Sussexes’ annual threat assessment has not been updated by the government since 2019, perceives the roadblock to be his father, even as the palace disputes this.

RAVEC’s decision could have international implications as well. If the U.K. acknowledges the threat to them, other countries are likely to follow suit in how they’re protected.

Faced with covering their own security expenses — and with other financial support from the King cut off — Harry and Meghan quickly struck deals to generate income when they came to the U.S. in 2020, including Harry’s lucrative book deal for his 2023 memoir Spare and the couple’s 2022 Netflix docuseries, Harry & Meghan.

A key factor in their decision to move to California was the ability to have armed private security in the U.S., say sources in their circle, pointing to daily social media threats, potential home intrusions and alarming incidents such as a chaotic paparazzi chase in New York City in May 2023.

“The U.K. government acknowledges numerous threats against them, but we are not privy to specific details due to lack of cooperation,” says Joe Funk, senior vice president of Torchstone Global Security, which oversees the couple’s private protection in the states.

Prince Harry, who is fifth in line to the throne, is also worried about the public’s safety when he is out, as well as the risk to his staff.

“The staff is concerned when they travel with them,” says a former employee of their Archewell Foundation.

The fact that Harry and Meghan stepped away from royal duty should not be a factor in their protection level, says the source close to the situation: “‘Working royal or not, this is the King’s son.”

Neil Basu, a former head of counterterrorism for the Metropolitan Police, has spoken out about the fact that Meghan faced serious threats while in the U.K., noting that both Harry and Meghan had among the highest threat assessments within the royal family. (In recent years various people have been arrested and imprisoned for making threats against the couple.)

“Harry’s military service, his global recognition and his marriage to a mixed-race woman all contribute to his high threat level,” Basu tells PEOPLE.

With both sides firmly entrenched—and Harry’s brother Prince William, 42, seemingly aligned with his father—any hope for reconciliation appears distant at best.

Some close to the palace suggest that Harry’s memoir and public interviews have compromised the family’s trust in him; others close to Harry counter that if he had proper security, he wouldn’t need to speak publicly in order to help pay for it. If the matter of security is resolved, says the friend, “it’s ‘swords down.’ ”

Indeed, nothing “would give [Harry] more happiness than being able to rekindle his bond with his father. At the end of the day,” adds the friend, “you can’t undo bloodlines. He’s not asking his dad for a nicer house or nicer cars. He’s asking because of the reality of the situation. He is at risk.”