Asylum seekers refusing to board the controversial Bibby Stockholm barge are being threatened with having rights to government support withdrawn.
The first group of 15 people boarded the vessel on Monday, but 20 refused.
If they do not move on to the accommodation barge by the end of Tuesday, their housing assistance could be withdrawn, government sources said.
Justice Secretary Alex Chalk said the threat to withdraw state-funded support was "unlikely" to be illegal.
He told BBC Radio 4's Today programme: "That is something that the courts would have to consider but I think it's unlikely, bluntly, that it would be illegal to do so but all cases are considered on their facts.
"What is perfectly legal is for the British people to say this is what we are offering and it's not four-star accommodation but it is perfectly safe, it's perfectly decent and it complies with the fire safety checks and goodness knows what.
"It is sparse and it is a bit austere but, frankly, that is not unreasonable."
Monday saw the first asylum seekers board the Bibby Stockholm - moored in Portland Port in Dorset - after a series of delays over safety concerns.
It is the flagship of the government's latest plan to "stop the boats" and deter dangerous Channel crossings by migrants and up to 500 men aged 18-65 will eventually live on the vessel while they await the outcome of asylum applications.
Some human rights groups have called the scheme "inhumane", but ministers insist it is safe and will save money.
One migrant who boarded the vessel on Monday told the BBC he had arrived in the UK on an aircraft, had a wife still in Iran and had been in Britain for six months.
The man - who the BBC is not identifying - gave no reply to the question of whether he was scared aboard the barge, but when asked said he had eaten a "good" breakfast aboard including "eggs, cheese, jam and butter".
Cheryl Avery, Director for Asylum Accommodation at the Home Office, said some 15 people had boarded the barge on Monday, but a group of about 20 people had refused to board.
"There have been some challenges - some minor legal challenges - and I can't go into the detail of those, but accommodation is offered to individuals on a no-choice basis," she said.
Sky News quoted a letter to one of the group of 20 as saying that if they failed to travel to Portland on Tuesday "arrangements for ceasing the support that you are receiving from the Home Office may commence".
"Where asylum seekers fail to take up an offer of suitable accommodation without a reasonable explanation, there should be no expectation that alternative accommodation will be offered," the letter added.
Heather Jones of the Portland Global Friendship Group, set up to provide support for those moved to the barge, said she had been in touch with eight asylum seekers who did not want to leave their Bournemouth hotel.
"They were worried about moving - they have a community in Bournemouth that supports and looks out for them and they have friends. To be moved again would have been so unsettling," she said.
The Care4Calais charity said it was providing legal support to 20 asylum seekers wanting to challenge being moved to Portland.
"Housing people fleeing threats and persecution on an overcrowded barge is appalling, and in itself likely to cause extreme distress.
"These are people who have endured horrific and often life-threatening journeys; being trapped in small, cell-like rooms behind the high metal fences will induce fear, anxiety and panic," it said.
The 222-room, three-storey barge arrived in Portland Port more than three weeks ago, chartered by the government to reduce what it says is the £6m-a-day cost of placing asylum seekers in hotels.
Ministers plan to increase the numbers aboard up to 500, despite safety warnings from the Fire Brigades Union (FBU) which has raised concerns over narrow exits and possible overcrowding.
It includes catering, a TV room, a multi-faith prayer room and a gym.
Migrants will be free to leave on hourly buses to Weymouth and Portland, although they are encouraged to return by 23:00 each night.
The Home Office has said the barge occupants will undergo security screening and Dorset Police has said it does not expect any impact on the local community.
https://www.bbc.co.uk/news/uk-england-dorset-66438086
