The commandos ambushed a Gazel minivan carrying the insurgents, along with arms and explosives, on a road near Russia's border with Azerbaijan, the Dagestani branch of the FSB said in a statement.
The FSB commandos fired several rocket-propelled grenades into the minibus and sprayed the rebels with automatic gunfire, the statement said.
Two FSB officers were wounded in the fighting, one of whom died later in a hospital.
Wednesday's sting operation came after two recent crackdowns by security services on the same Islamist rebel network, which is active in southern Dagestan, local security officials said.
Three policemen were killed in a Sept. 7 clash, as were five militants, including Ilgar Mollachiyev, whom the Chechen rebel leadership had appointed last fall as commander of the Dagestani insurgents. The following day, three policemen were killed in a skirmish with members of Mollachiyev's group whom law enforcers attempted to encircle near the village of Sirtych.
Dozens of law enforcement officers are killed each year in Dagestan, where ethnic rivalries and Islamist separatism -- exacerbated by police brutality and official corruption -- prompt young men to join violent anti-government groups.
Dagestani leader Mukhu Aliyev has repeatedly demanded -- to no avail -- that police desist from brutal attacks on local residents. A campaign is under way in the republic to oust Dagestani Interior Minister Adilgirei Magomedtagirov, who answers to the federal government.
Sergei Markedonov, a Caucasus expert at the Institute for Political and Military Analysis, said that because of pervasive corruption and entrenched nepotism in Dagestan and Ingushetia, deadly violence has become the only available form of political protest for those unhappy with the status quo.
"Also, the latest events in Dagestan demonstrate that the Kremlin's claims of having crushed the insurgent networks in the Caucasus are not true," he said.