The European Union's special envoy to Georgia said the talks hit an impasse because of "procedural difficulties."
A compromise allowing the participation of representatives from South Ossetia and Abkhazia proved impossible to find, comments from Russian and Georgian participants showed.
Russia and Georgia fought a five-day war in August and remain at odds over South Ossetia and Abkhazia, two breakaway Georgian provinces that Moscow has recognized as independent states under its protection.
"All of the parties expressed their points of view. Today we encountered procedural difficulties. That is why we decided to suspend the discussions this afternoon and pursue the process of discussions," EU envoy Pierre Morel said at a briefing.
He said new talks had been provisionally set for Nov. 18.
The European Union, United Nations and Organization for Security and Cooperation in Europe had organized the one-day meeting, which they had hoped would lead to talks every two weeks to build confidence and help resolve the conflict.
In Brussels, Georgian President Mikheil Saakashvili blamed Russia for the failure of the first day's talks.
"Russia has just walked out of the Geneva talks ... which basically means that Russia has no interest whatsoever at this stage in any diplomatic process," he told reporters.
But the head of Russia's delegation, Deputy Foreign Minister Grigory Karasin, said the meeting had initiated a process to resolve the conflict with the participation of Abkhazia and South Ossetia.
"The event was de facto broken up by Georgia, which refused to take part in the plenary session," Karasin said, Interfax reported. "They failed to break up the meeting, it took place, and that is extremely important."
Feverish diplomatic efforts to find an acceptable format for the talks included a news blackout and a ban on photographers from taking pictures of the delegations as they entered the UN European headquarters in Geneva.
The United States, which sees Georgia as an ally in the volatile Caucasus region, also took part in the talks.
Also Wednesday, the UN's highest court ordered Russia and Georgia to ensure the security of all ethnic groups in the breakaway Georgian regions of South Ossetia and Abkhazia and adjacent areas of Georgia.
In a provisional ruling on a lawsuit filed by Georgia that alleged human rights violations by Russia in the region, the International Court of Justice said Georgia and Russia must refrain from sponsoring any act of racial discrimination.
It also ordered both parties to do everything in their power to ensure the security of persons, freedom of movement and the protection of refugees' property.
The court ruled it had jurisdiction to order the provisional measures and ordered both parties to inform it of their compliance.
Court rulings, including provisional orders, are binding but the court has no police force to enforce its decisions. A judgment on the merits of the case proper could take at least another year.
A Message from The Moscow Times:
Dear readers,
We are facing unprecedented challenges. Russia's Prosecutor General's Office has designated The Moscow Times as an "undesirable" organization, criminalizing our work and putting our staff at risk of prosecution. This follows our earlier unjust labeling as a "foreign agent."
These actions are direct attempts to silence independent journalism in Russia. The authorities claim our work "discredits the decisions of the Russian leadership." We see things differently: we strive to provide accurate, unbiased reporting on Russia.
We, the journalists of The Moscow Times, refuse to be silenced. But to continue our work, we need your help.
Your support, no matter how small, makes a world of difference. If you can, please support us monthly starting from just $2. It's quick to set up, and every contribution makes a significant impact.
By supporting The Moscow Times, you're defending open, independent journalism in the face of repression. Thank you for standing with us.
Remind me later.