Jim Hytner, the global chief operating officer, will take over the leadership of the UK on an interim basis.
IPG Mediabrands has had a tough year with the loss of the £90 million Tesco business to MediaCom in April.
Jones became the chief executive at IPG Mediabrands UK and Ireland two years ago and aimed to turn the company around to compete with its bigger rivals.
Jones was previously the chief executive at UM London between 2003 and 2013, and also worked as joint managing director and chief operating officer at the agency.
Jones began his career in 1983 as a sales executive at ITV. He then moved agency side two years later to McCann Erickson.
Hytner became the chief executive for the Mediabrands G14 region in 2013. He has previously worked as the worldwide chief executive at Initiative, the EMEA chief executive at Universal McCann, and the UK marketing and global brand director at Barclays Bank.
Henry Tajer, the global chief executive at IPG Mediabrands, said: "Andy has been an outstanding and loyal employee for a long time.
"We will be announcing our plans for the UK office in the coming weeks, but for now, this is an opportunity to express our gratitude to Andy, although he will be with us into 2016 to assist with the handover."
