The eldest son of the Hon. Edward Ward (Member of Parliament for county Down) and his wife, Lady Arabella Crosbie [d. 1813], Edward Southwell Ward was also the grandson of Bernard Ward, 1st Viscount Bangor. His uncle, Nicholas, 2nd Viscount Bangor, died without children in 1827; thus Edward succeeded him as 3rd Viscount. Ward married Harriet Margaret Nugent [d. 1880] on 14 February 1826. They had four sons, of whom Edward and Henry William Crosbie became the 4th and 5th Viscounts, respectively.