The seven-times grand slam champion pulled up with what appeared to be a groin strain after lunging for a return with the German leading by a game and 30-0 in Melbourne.
It was just too painful to move, fourth seed Williams told reporters after aggravating an injury that troubled her in the previous round.
The last 48 hours I did as much pain management and recovery that I could. A lot times you get this adrenaline that blocks pain. I just didn't get enough of that today.
Williams, a Melbourne finalist in 2003, strained a muscle in the same area during her second round victory over Czech Sandra Zahlavova on Wednesday.
Her retirement leaves a grand slam without an American woman in the last 16 for only the fourth time, the last time coming at the French Open in 2008.
Petkovic moves on to face former champion Maria Sharapova in the fourth round.
It's super disappointing, said Williams. This is just not how I envisioned my Australian Open being. I've never had to retire from a grand slam.
I was just hoping for some magic that I could recover. I'm just going to focus on getting healthy and coming back, added the former world number one, who last won a major at Wimbledon in 2008.
I love tennis and I've got a lot of great tennis in me. I love my job, so (there's) no end in sight.
Petkovic had mixed feelings after going through, and earning A$38,500 -- the difference between winning and losing in the third round -- for her four minutes on court.
I feel very weird about tonight, said the German, who matched her fourth round appearance at last year's U.S. Open.
Obviously I'm one round further which is great. But on the other hand I really felt great today. I really wanted to give it a real go.