James David Barry was the nephew of Captain James Barry (d. 7 January 1808) and his wife Joanna Gould Barry (d. 18 October 1811). The Barrys were imigrants of Irish descent who settled in Baltimore in 1793. They came from Lisbon, Portugal, and James served as the Portugese counsel general in the US. The Barrys had two daughters, Ann (d. 17 July 1808) and Mary (d. 17 November 1805), both of whom were unmarried. The Barry family was painted by American artist Gilbert Stuart; the portraits of the Barry daughters are now in the NGA (1954.9.3 and 1954.9.4). James David Barry married Joanna Coombs in 1811. He and his brother Robert Barry both spent time in London.