Oregon Gov. Kate Brown announced Wednesday that the Trump administration had agreed to a phased withdrawal of federal agents from Portland—but President Trump and acting Department of Homeland Security Secretary Chad Wolf weren’t on the same page. “They have acted as an occupying force & brought violence,” Brown wrote in a tweet. “Starting tomorrow, all Customs and Border Protection & ICE officers will leave downtown Portland.” She said state police would be the main police presence from Thursday, and would “protect Oregonians’ right to free speech and keep the peace.”

While Wolf said in a statement that an agreement had been reached with Brown, he said federal agents would remain until the department is “assured” that federal buildings, including the Mark O. Hatfield Courthouse, wouldn’t be attacked. In a call with reporters Wednesday afternoon, he said no agents would be leaving until federal agencies determine that state police are stopping protesters from damaging the courthouse. He said state police would be setting up a secondary perimeter around the courthouse to add protection.

Asked by reporters Wednesday morning, Trump said federal agents wouldn’t leave until the area was secure. “If they don’t secure their city soon, we have no choice, we’re going to have to go in and clean it out,” he said,