Djokovic could not find his rhythm, eventually winning his first service game after 15 minutes but going on to lose the first set in one hour and 25 minutes.
He won four consecutive games in the third set as his opponent faltered but Istomin came back in the fourth set to take it to a tie-break.
Both players served aggressively as they received vocal support from the crowd, with Istomin taking the match to a deciding set with a brutal ace.
Istomin, who broke in game five, remained strong on his own serve and raised his arms to the crowd after Djokovic sliced his final shot into the net.
Uzbek Istomin will next face Spain’s Pablo Carreno Busta, who beat Britain’s Kyle Edmund on Thursday.
‘I surprised myself’
“It is the biggest win [of my career] for me and means so much, now I feel I can play with these guys and be with them on the same level,” said Istomin.
“From the third set I had cramp in my leg, I don’t know how I held it. I was playing so good. I surprised myself.”
Istomin had two years out of the game after breaking his leg in a car accident in 2001 and spending three months in hospital.
Coached by his mother Klaudiya, he dropped out of the top 100 in 2016 and was given a wildcard to play in the Australian Open.
Prior to his win over Djokovic, Istomin had won just one of 33 matches against a player ranked in the world’s top 10.
His best Grand Slam result is reaching the last 16 at Wimbledon in 2012 and the US Open in 2013, where he lost to Murray.
Djokovic’s difficult seven months
Djokovic has struggled for consistency since winning his first French Open title in June 2016 and completing a career Grand Slam.
He was knocked out in the third round at Wimbledon by American Sam Querrey but looked to have returned to form when he won the Rogers Cup in July.
However, he went on to lose to Del Potro in the first round of the Olympics and was knocked out of the doubles competition the following day.
He struggled physically in the US Open final, losing in four sets to third seed Stan Wawrinka, before he lost the world number one ranking to Murray in November.
Murray also ended his run of four consecutive ATP World Tours Finals titles in the same month.
“Djokovic is not the same Djokovic we saw this time last year, who was at the peak of his career,” two-time Australian Open finalist Pat Cash told BBC Radio 5 live.
“It’s clearly the mental edge. He’s done so much and worked so hard to grab those four Grand Slams, I think he’s just lost the edge.”
