Within six minutes his side were a goal down through a Rodrigo Galo strike, with Croatian striker Marko Livaja heading in a second five minutes after the break.
Celtic had been looking to make it to the Champions League group stage for the third year in a row but despite Scott Sinclair's goal to push for a comeback they ultimately fell short.
The Scottish champions drop down to the Europa League play-off, where they will face Latvian side Spartaks Jurmala or Lithuanian side Suduva, and Rodgers said: "The Europa League is still a prestigious competition".
Red Star, who have never qualified for the Champions League group stage, will meet Salzburg after the Austrian team strolled into the playoffs with a 4-0 defeat of Macedonians Shkendija.More news: Ndidi Rated Best Passer In Leicester Defeat To Man United
More news: NASA launches spacecraft to 'touch sun' - USA
More news: Rupee hits lifetime low of 70 per dollar
Earlier, Boyata released a video on Instagram showing him receiving treatment, accompanied by a caption saying: "Unfortunately I could not play tomorrow because I'm not yet 100% operational".
Boyata's agent Jacques Lichtenstein claimed the Parkhead club had broken promises to his client, who was the subject of a failed £9million bid from Fulham before the English transfer window closed.
"I still felt this was a game in which we had more than enough to win but you have to defend better than we did".
Former Celtic player Andy Walker says the Hoops need to sell Dedryck Boyata and that his team-mates "can not trust him". "We have to find a way to get a result".
"My concentration really until I get back is only with the players that are here".