The Gers were leading 1-0 after Alfredo Morelos’s strike when Malmo went down to 10 men on the stroke of half-time with Bonke Innocent sent off.
But the Swedish champions responded with a double strike within four minutes from Antonio Colak to turn the tie on its head, with Malmo having won the first leg 2-1.
Gerrard’s side dominated possession but couldnt find a way to win, condemning them to three straight defeats, including losing to Dundee United in the Scottish Premiership on the weekend.
“The most frustrating part is we’ve lost the second half to 10 men,” Gerrard said.
“To be 1-0 up and then go 2-1 down, a bit of fear and panic set in. We’ve got the second-half performance wrong and that’s on all of us.
“We’ve lost three games out of four and, at Rangers, that’s not good enough.”
Elsewhere in qualifying, Red Star Belgrade were bundled out by Moldovan minnows Sheriff who won the second leg 1-0 to seal a 2-1 aggregate victory.
Greek champions Olympiacos were knocked out in a shock by Bulgarian outfit Ludogorets on penalties after a 2-2 second leg draw.
Hungarian champions Ferencvaros got past Slavia Prague, while Slavia Prague were beaten comfortably by Monaco.
Shakhtar Donetsk, PSV Eindhoven, Young Boys, Benfica and Dinamo Zagreb also all progressed.
The winners will now meet in the playoff round to determine the final 32 Champions League participants.