The eldest son of James George, 3rd Earl of Courtown, Knight of the Order of St. Patrick, and his wife, Mary (daughter of the 3rd Duke of Buccleuch), James Thomas succeeded his father as Earl in 1835. On 4 July, 1822, he married his cousin, Lady Charlotte Albinia Montagu Scott [d. 1828], daughter of the 4th Duke of Buccleuch. They had a son, James George Henry, who later succeeded his father as 5th Earl. James Thomas married secondly on 29 October 1850 to Dora Pennefather [d. 1859], daughter of Rt. Hon. Edward Pennefather, Chief Justice of Queen's Bench, Ireland. With her he had three more sons and a daughter.