Both sides were reduced to 10 men as England scored a 90th minute from Raheem Sterling before Joe Gomez gave away another to Iceland moments later.
However, the hosts spurned their penalty with Birkir Bjarnasson firing over the crossbar in the 93rd minute, to ensure England avoiding a draw.
“If we had scored the late penalty and that had been the end of it, we’d have been talking about a performance of great character,” Southgate said.
“We then give a really poor penalty away and it feels like we got out of jail.”
Kyle Walker was sent off for England for a second bookable offence in the 70th minute which left Southgate disappointed but he declined to publicly criticise him.
“Kyle knows the situation,” he said. “We all have to learn from that incident because we’ve gone out of tournaments before because we’ve been down to 10 men.
“That’s not a situation we’ve had in the four years I’ve been in charge. So the response to it from the rest of the team was good.
“But it put us in a really difficult situation and also we lose him now for the Denmark game (on Tuesday).”
Earlier in the game, England appeared to have a legitimate goal disallowed when Harry Kane tapped in at the back post.
“Obviously Harry’s goal should stand and that would gave a big bearing on the glow of the game because of that slight anxiety that you have when you haven’t got a goal,” he said.
“But then I thought, we were good until the final third, but that lack of match fitness showed in our creative play.
“Defensively, we were reasonably solid, we defended the set plays well but were caught on a few counter attacks.
“The sending off was a key moment as we were even more in control at that stage and it’s very difficult to win any international game when you’re down to ten men.
“But at least we leave here with a win, which gives us some momentum going forward.”
England play Denmark, who lost 2-0 at home to Belgium, on Tuesday in Copenhagen, while Iceland travel to Brussels.