The militants raided two jails, leading to the escape of more than 100 inmates, according to Mujiv Hataman, the governor of a Muslim self-ruled area that includes Marawi.
Duterte imposed 60 days of martial law Tuesday on the island of Mindanao, which encompasses the southern third of the nation and is home to 22 million people.
He vowed to be "harsh".
Mr Duterte said on Wednesday that one of the policemen killed was caught at a checkpoint set up by the militants, before being beheaded. If there is open defiance, you will die.
Duterte said his own version of martial law meant security forces could conduct searches and arrest people without warrants.
A woman touches an image of Jesus Christ shortly after attending a mass for victims of the recent attack in Marawi by Muslim militants linked to the Islamic State group and also for extrajudicial killings inside a church in Man.
Marawi mayor Majul Gandamra said some rebels from the Maute were still holed up in buildings and sporadic gunfire could be heard. "I'm just praying that the bullets will not find its way to my house and hit us", he said as he left the city.
"I've realigned myself in your ideological flow and maybe I will also go to Russian Federation to talk to Putin and tell him there are three of us against the world: China, Philippines and Russian Federation".More news: MS politician calls for lynchings over removal of Confederate statues
More news: National Football League awards Super Bowls to Tampa, Los Angeles
More news: HSBC Holdings plc Reaffirms "Buy" Rating for Barclays PLC (BARC)
But, even though the region is a known hotbed of Islamist militants, the troops were taken by surprise when dozens of gunmen emerged to defend Hapilon, then go on a deadly rampage throughout the city.
Archbishop Socrates Villegas, president of the Catholic Bishops Conference of the Philippines, said the militants forced their way into a cathedral in Marawi and seized a priest, 10 worshippers and three church workers.
Under martial law, enhanced security measures will be implemented, including strengthened border controls, restrictions on movement of people and vehicles, closures of public places if necessary and increased police surveillance of public areas.
The gunmen have pledged allegiance to ISIL. The Philippines launched an airstrike that wounded him in January, but he got away. Meanwhile, Duterte has scaled down joint military exercises with the US.
Marawi is located in Lanao del Sur province, a stronghold of the Maute, a fierce, but little-known group that has been a tricky opponent for the military. The Maute is one of less than a dozen new armed Muslim groups that have pledged allegiance to the Islamic State and formed a loose alliance, with Hapilon reportedly designated as the alliance's leader.
He said troops had isolated the guerrillas but were not engaging them. "The Mautes are embedded in the population".
The Maute has been blamed for a bombing that killed 15 people in southern Davao city, Duterte's hometown, last September and a number of attacks on government forces in Lanao, although it has faced setbacks from a series of military offensives.
A military spokesman says troops are using helicopters to help clear militants from the besieged city of Marawi in the southern Philippines.