James Agar succeeded his father as Earl in 1868. He earned a Master of Arts, and was Deputy Lieutenant for Wiltshire and Hampshire. In 1873, he was created Baron Somerton of Somerley, Hampshire. He married Caroline Susan Augusta [d. 1915], daughter of the 6th Viscount Barrington, in 1856. They had a large family of four sons and four daughters; the second son, Sidney James, succeeded his father as 4th Earl.