Pejić was born in Tuzla and has one older brother, Igor. His mother, Jadranka Savić is a Bosnian Serb, and his father, Vlada Pejić, is a Bosnian Croat. They divorced shortly after Andrej's birth. During the Bosnian War, Andrej and Igor fled to Serbia with their mother and grandmother, settling in a refugee camp near Belgrade. When asked if they found their childhood difficult, Pejić said:
No, I was happy. The camp was a community. We went to school and I had lots of friends – mainly girls. Everybody played outside. My memories are very carefree.
After the refugee camp, the family settled in Vojska village near Svilajnac.
The siblings met their father for the first time when Andrej was four years old. Their mother wanted them to have a relationship with their father, and to that end, Andrej and brother Igor would spend a month every summer with him.
After the 1999 NATO bombing of Serbia, Pejić's mother felt unsafe and decided to initiate the immigration process for Australia. In 2000, the family moved to Melbourne, Victoria, Australia as political refugees when Andrej was eight years old.