Charles Amory was married to Martha Babcock Greene Amory [1812-1880], grand-daughter of the artist John Singleton Copley [1738-1815]. The Amorys had two sons, Copley Amory [1841-1879] and Edward Linzee Amory [1844-1960], and a daughter Susan Greene Amory, later Mrs, Franklin Gordon Dexter [1840-1924].
Linzee, John William. The Linzee Family of Great Britain and the United States of America. 2 vols. Boston, 1917: 2:766.