The European-drafted resolution had been watered down to try to avoid the vetoes, dropping a direct reference to sanctions against Damascus.
But Moscow and Beijing said the draft contained no provision against outside military intervention in Syria.
The US envoy to the UN said Washington was "outraged" by the vote.
The result is a huge blow to European and US efforts on the Syria issue, the BBC's Laura Trevelyan in New York says.
More than 2,700 people have been killed across Syria since the crackdown began in March, the UN estimates.
The government of Syrian President Bashar al-Assad says it is in the process of introducing reforms and is speaking to members of the opposition. It blames the unrest on armed gangs.
'Unacceptable'Nine countries backed the resolution in the 15-member council, while four more abstained during the vote late on Tuesday.
But the resolution - which was drafted by France with the co-operation of Britain, Germany and Portugal - was still defeated because of the vetoes from two of the council's five permanent members.
Continue reading the main story
“Start Quote
China believes that under the current circumstances sanction or threat of sanction does not help resolve the question of Syria. Rather it may further complicate the situation”
Li BaodongChina's UN AmbassadorThis was despite the fact that the text had been changed three times by the European allies, who had tried to accommodate Russia's and China's objections.
The resolution referred to "targeted measures" - instead of sanctions - if the clampdown in Syria continued.
But Russian Deputy Foreign Minister Gennady Gatilov earlier said the resolution was "unacceptable" because it envisaged sanctions and did not call on President Assad's government to start talks with the opposition, Russia's Interfax news agency reported.
Moscow also said that the draft contained no provision against outside military intervention in Syria, or respect for non-interference in its domestic affairs.
China's UN ambassador Li Baodong said that Beijing "believes that under the current circumstances sanction or threat of sanction does not help resolve the question of Syria. Rather it may further complicate the situation".
The Libyan conflict has sharpened divisions among Security Council members, with both Moscow and Beijing saying that the resolution authorising the use of force to protect civilians was misused by Nato to bring down Col Muammar Gaddafi's regime.
Our correspondent says Tuesday's vote exposes the deep rift at the UN between the major powers, and the completely difference of approach in how to deal with Mr Assad's government and its crackdown.
After the vote, France's UN Ambassador Gerard Araud said the veto showed "disdain for the legitimate interests that have been fought for in Syria" since the protests in the country began.
The US envoy, Susan Rice, said: "The United States is outraged that this council has utterly failed to address an urgent moral challenge and a growing threat to regional peace and security."
Washington had said before the vote that it hoped the council would send a strong message to Syria. Germany said the world had to show solidarity with Syrians on the streets.
=============
Dilma irrita Itamaraty com abstenção na ONU
Agencia Estado, 4/10/11
A decisão do Brasil de, mais uma vez, abster-se em uma votação no Conselho de Segurança das Nações Unidas partiu diretamente da presidente Dilma Rousseff e, desta vez, incomodou profundamente o Itamaraty. A resolução votada na noite de hoje teve o Brasil como um de seus principais negociadores e, em sua versão final, foi ainda mais branda do que o texto com o qual os diplomatas brasileiros já haviam concordado. Ainda assim, Dilma, consultada em Bruxelas, onde participava da Cúpula Brasil-União Europeia, instruiu a missão em Nova York a se abster.
A decisão foi tomada pela presidente para manter uma paridade com as posições de outros países dos BRICs, já que a Índia e África do Sul haviam informado sua decisão de se abster, e Rússia e China, sempre mais resistentes em condenar o governo sírio, avisaram que iriam usar seu poder de veto.
Na votação final, além de Índia, Brasil e África do Sul, apenas o Líbano também se absteve. O país do Oriente Médio, que ocupava a presidência do Conselho até a semana passada, sempre resistiu a condenar o regime de Bashar Al-Assad e havia impedido, na sua presidência, o avanço de um texto com sanções.
A avaliação de diplomatas ouvidos pelo Grupo Estado é que o Brasil, depois de mais esse veto, ficou em uma posição de difícil defesa, já que a missão brasileira foi uma das principais negociadoras do texto. O País já tinha concordado com uma resolução que daria um prazo máximo de 30 dias para que o governo sírio parasse com a violência ou embargos econômicos, de armas e viagens poderiam ser colocados em prática. O texto final foi desidratado para tentar conquistar pelo menos a abstenção de China e Rússia, retirando até mesmo menções aos possíveis embargos.