The Egyptian-drafted resolution had been withdrawn after Israel asked Donald Trump to intervene but it was proposed again by Malaysia, New Zealand, Senegal and Venezuela.
The US has traditionally sheltered Israel from condemnatory resolutions.
Israeli PM Benjamin Netanyahu said he would not abide by the vote.
Israel announced its ambassadors to New Zealand and Senegal had been ordered to return for consultations. Israel has no diplomatic ties with Malaysia and Venezuela.
The Palestinian leadership welcomed the UN resolution, which was passed by 14 votes to zero, with one abstention.
President-elect Trump, who will be inaugurated on 20 January, tweeted after the vote: "As to the UN, things will be different after Jan. 20th."
The issue of Jewish settlements is one of the most contentious between Israel and the Palestinians, who see them as an obstacle to peace.
About 500,000 Jews live in about 140 settlements built since Israel's 1967 occupation of the West Bank and East Jerusalem. The settlements are considered illegal under international law, though Israel disputes this.
US policy reverse: By Barbara Plett Usher
The resolution reflects an international consensus that the growth of Israeli settlement-building has come to threaten the viability of a Palestinian state in any future peace deal.
It is a view strongly shared by the Obama administration, and for that reason the US reversed its policy of vetoing any UN Security Council criticism of Israel.
It is a decision that was taken after months of debate within the administration about whether and how President Obama might be able to define his position on a two-state solution before leaving office.
But his successor Donald Trump has made clear he intends to strongly support Israeli government positions, even making a highly unorthodox intervention before the vote by publicly urging Mr Obama to veto the resolution.
In a statement, Israeli Prime Minister Benjamin Netanyahu's office said: "Israel rejects this shameful anti-Israel resolution at the UN and will not abide by its terms.
"At a time when the Security Council does nothing to stop the slaughter of half a million people in Syria, it disgracefully gangs up on the one true democracy in the Middle East, Israel, and calls the Western Wall 'occupied territory'.
"The Obama administration not only failed to protect Israel against this gang-up at the UN, it colluded with it behind the scenes. Israel looks forward to working with President-elect Trump and with all our friends in Congress, Republicans and Democrats alike, to negate the harmful effects of this absurd resolution."
Image copyrightAFPImage captionDonald Trump has taken a strongly pro-Israel stanceThe US ambassador to the UN, Samantha Power, said the resolution reflected the "facts on the ground" that settlement growth had been accelerating."The settlement problem has gotten so much worse that it is threatening the two-state solution," she said.
She criticised Mr Netanyahu's support for settlement expansion, saying: "One cannot simultaneously champion expanding settlements and champion a two-state solution that would end the conflict."
However, Ms Power added that the US had not voted in favour of the resolution because it was "too narrowly focused" on settlements.
She added that even if all settlements were dismantled, both sides would still have to acknowledge "uncomfortable truths" and make "difficult choices" to reach peace.
A spokesman for Palestinian President Mahmoud Abbas said: "The Security Council resolution is a big blow to Israeli policy, a unanimous international condemnation of settlements and a strong support for the two-state solution."
On Thursday, Mr Trump had urged the Security Council to defeat the motion.
"Peace between the Israelis and the Palestinians will only come through direct negotiations between the parties, and not through the imposition of terms by the United Nations," he said in a statement. "This puts Israel in a very poor negotiating position and is extremely unfair to all Israelis."
