Julius Fisher Gage was the elder son of prominent inventor and machinery manufacturer John Henry Gage and his wife, Catherine Ann, of Nashua, New Hampshire. He is pictured at four years of age with his parents and younger brother, Edward John, in a group portrait dated to 1846. The portrait (now in the NGA), which was painted by an unknown American folk artist, descended in the Gage family and was in Julius Gage's possession as of 1901.