Sanchez won the confidence motion on Friday by 180 votes to 169 with one abstention.
That all changed on May 25 when the Socialists filed a no-confidence motion against Rajoy, a day after a court found former PP officials guilty of receiving bribes in exchange for awarding public contracts in a vast graft scheme between 1999 and 2005.
Sanchez will be sworn in Saturday at 11 a.m. (0900 GMT, 5 a.m. EDT) by King Felipe VI at the Zarzuela Palace in Madrid and will appoint his Cabinet over the coming days.
Unlike the new government in Italy, Sanchez and his party are staunch supporters of the European Union and the shared currency.
The bespectacled 63-year-old leader got up and shook hands with Sanchez before leaving the lower house without a word.
The incoming prime minister has outlined that his priorities will be social issues - such as measures to help young people and the elderly - before calling elections, though he hasn't said when there might be a vote.
Sanchez's victory resulted in the first ouster of a serving Spanish leader by parliament in four decades of democracy.
Rajoy became Spain's first sitting prime minister to give evidence in a trial when he was called as a witness previous year.More news: This Is Why Kim Kardashian Visited The White House Yesterday
More news: Missouri Governor Eric Greitens Resigns
More news: Cavs' Kevin Love Still In Concussion Protocol, Game 1 In Jeopardy
The no-confidence motion made odd bedfellows of 22 widely divergent parliamentary groups, including anti-establishment left-wing national parties, the political arm of the now-defunct armed Basque separatist group ETA, and regional Catalan nationalist parties hankering to negotiate an independent Catalan republic. Many Spanish voters, exasperated by corruption scandals involving the traditional centre-right PP and centre-left Socialist parties, have abandoned them for newcomers like the left-wing Podemos (We Can) and centrist Ciudadanos (Citizens), as well as regional parties.
Madrid fired Catalonia's government and took control of the running of the region after an illegal and unsuccessful declaration of independence by the Catalan parliament in October.
In its ruling, the court said the credibility of Rajoy's testimony "should be questioned".
Sanchez not only inherits Spain's worst political crisis in almost four decades.
The PSOE only holds 84 of 350 seats in the Spanish parliament, meaning it is not clear if the party will be able to govern effectively.
In his first cabinet list presented on May 19, Catalan leader Quim Torra included candidates either in exile as a outcome of the independence bid or under investigation for their role in it.
The country's new leader will preside over a minority government propped up by the leftist Podemos bloc and other parties, including Basque and Catalan nationalists.
In a reversal from Rajoy's aversion to Catalonia's aspirations for greater autonomy, Sanchez has promised to open a dialogue with new Catalan President Quim Torra despite having called the fervent separatist's comments "xenophobic".