Defra's agricultural transition plan puts 14,000 farms in the uplands and commons 'at risk', a charity which protects commoners' rights has warned.
The Foundation for Common Land (FCL) has written to the prime minister to warn of the 'impending car crash' for thousands of farms.
The group also warned that the government's plan 'threatened' the environmental outcomes of the land commoners steward.
Commons provide more public benefits per area of land than almost any other type of land, figures by the charity highlight.
Over half (58%) of common land is designated as SSSIs, 80% is in designated landscapes and 38% of moorland is registered common land.
Defra's transition plan, unveiled this week, sets out that the Basic Payment Scheme (BPS) income will be halved by 2024 and gone entirely by 2028.
The Environmental Land Management scheme (ELMS) will be introduced to incentivise sustainable farming practices while boosting the environment.
But the FCL warned this would leave upland and common farm businesses 'not knowing which fork on the path to take'.
The charity also said the post-Brexit plan failed to provide figures on the value of the future environmental land management schemes.
Julia Aglionby, executive director of the group, said the government was seeking carbon storage, biodiversity and natural flood management 'on the cheap'.
"Defra confirmed last week that until further notice, payments will be based on income of foregone approach instead of a natural capital approach for payments."
She added that Defra did not provide an impact analysis on how its roadmap proposals would affect the viability of farms currently heavily dependent on BPS.
"Our modelling is not pretty; the family farm is at risk, including activities such as pastoral commoning," Ms Aglionby said.
She asked: "How can farmers restructure to deliver for nature and the climate without any details? What does ‘public money for public goods’ actually mean?
"Defra’s Path to Sustainable Farming is rather like a road atlas that only shows the motorways. We need a much more granular scale plan.”
The charity said those farmers who looked after sensitive landscapes 'risked being first against the wall' in this restructuring of British agriculture.
It highlighted Defra data which shows that upland farms' BPS income is '104 percent of farm business income'.
"We ask the government to address our concerns as a matter of urgency," Ms Aglionby said.
The Liberal Democrats echoed the charity's warning, saying the new plan for farming risked 'driving thousands of family farmers out of business'.
It has called for current payments to be maintained until the new scheme is rolled out in full.
Liberal Democrat farming spokesperson Tim Farron: “Reports that farmers could lose half their money in the next four years are deeply concerning.
“Their plans risk driving thousands of family farmers out of business, leaving no one to deliver their new plans to protect our environment."
