President Donald Trump insisted on Tuesday that not all of the facts were known yet about the aftermath of a white supremacist rally in Virginia that turned violent at the weekend and that both left- and right-wing groups used force.

Trump, taking questions from reporters in the lobby of Trump Tower in New York, reverted to his initial comments on Saturday blaming "many sides" for the violence in Charlottesville, but on Monday had explicitly condemned neo-Nazi groups.

"They came at each other with clubs ... it was a horrible thing to watch," Trump told reporters at what was supposed to be an announcement of his administration's infrastructure policy. He said left-wing protesters "came violently attacking the other group."

Trump has faced a storm of criticism from Democrats and members of his own Republican Party over his initial response to the violence around the rally in the Southern college town of Charlottesville.

The trouble erupted after hundreds of white nationalists converged in Charlottesville to protest plans to remove a statue of General Robert E. Lee, commander of the pro-slavery Confederate army in the U.S. Civil War.

Street brawls broke out as the white nationalists were met by crowds of anti-racism demonstrators. A car then plowed into a group of the counter-protesters, killing one woman and injuring 19 other people. A 20-year-old Ohio man, James Fields, said to have harbored Nazi sympathies, was charged with murder, malicious wounding and leaving the scene of a fatal accident.

On Tuesday, Trump explained his initial restrained response by saying: "The statement I made on Saturday, the first statement, was a fine statement, but you don't make statements that direct unless you know the facts. It takes a little while to get the facts." Trump earlier Tuesday hit back at business leaders who quit a presidential advisory panel in protest, calling the executives "grandstanders."

Three business leaders quit a Trump panel in protest on Monday and on Tuesday, Scott Paul, president of the Alliance for American Manufacturing, said on Twitter he was also resigning.

Trump bowed on Monday to two days of pressure for a more forceful response, singling out groups behind the "Unite the Right" rally that were widely seen as stoking the disturbances. But he was still clearly frustrated over the reaction to his response.

"For every CEO that drops out of the Manufacturing Council, I have many to take their place. Grandstanders should not have gone on. JOBS!" Trump said on Twitter on Tuesday.