Sheriff, competing in the UEFA Champions League for the first time, had won their opening group fixture 2-0 over Shakhtar Donetsk.
The Moldovans backed that up with an improbable result in the Spanish capital, after an unmarked Jasurbek Yaxshilboyev headed them in front in the 25th minute.
Karim Benzema curled an effort wide before half-time from the edge of the box, before the Frenchman levelled the game up from the spot in the 65th minute after Vinicius Junior won a penalty.
Sheriff had a goal disallowed for offside, before Luxembourg international Sebastien Thill netted a stunning left-foot 89th minute winner to leave Ancelotti’s side shell shocked.
“It’s small details that cost us this game,” Ancelotti said after the game.
“It’s a lesson for us in the future as it’s a defeat we didn’t deserve.”
Real had 31 shots compared to Sheriff’s four, with 11-3 on target, as well as dominating 76-24 per cent possession with almost 500 more passes.
Ancelotti added: “More than worried, we’re annoyed. We played with intensity and were looking good.
“We lost because of small details, even though the side played well. We could’ve been better when it came to the final third but it’s tough to explain.”
The result means Sheriff are outright top with six points from two games, while Real are second with three points.
Inter and Shakhtar drew 0-0 in the other group game, meaning they remain third and fourth with one point each.
“We’ve missed out on three points and the group is wide open,” Ancelotti said. “We need to win our next game.”