Campbell married Lady Elizabeth Georgiana [d. 1878], eldest daughter of the 2nd Duke of Sutherland. They had a very large family of five sons and seven daughters. He was a Knight of the Orders of the Garter and Thistle, a Doctor of Laws and Doctor of Civil Law, a Privy Counsellor, and a Fellow of the Royal Society. He served as a Chancellor of the University of St. Andrews and as a Trustee of the British Museum. From 1852-55, 1859-66 and 1880-81, he was Lord of the Privy Seal; from 1868-74, Secretary of State for India. He married secondly in 1881, to Amelia Maria [d. 1894], widow of Col. Hon. Augustus Henry Archibald Anson, and daughter of the Bishop of St. Albans. The Duke married a third time in 1895, to Ina Erskine [d. 1925], formerly Extra Woman of the Bedchamber and Private Secretary to Queen Victoria.