Joseph Willson was probably the younger brother of Samuel Willson, both of them sons of the elder Joseph Willson, and grandsons of the Samuel Willson who came to New York City in 1674, and died in 1689. Only Samuel is named as a grandchild in the elder Samuel's will, dated December 1688, as Joseph had not yet been born at that time. The younger Joseph Willson married his first wife, Ann Hopkins, on November 20, 1722, and their daughter, Mary, was born November 20, 1723. Ann Hopkins Willson died in 1746, and Joseph was married again in 1754, to Elizabeth Lamb (sister of General Lamb). They had two children, Cornelia and Ann. Willson inherited a portrait (NGA 1957.11.9) thought to be of his older brother Samuel, painted in 1720 by an American "naive" artist now referred to as the Schuyler Limner. When Joseph Willson died the painting was apparently inherited by his widow, who herself remarried, to Samuel Hallett. The Halletts went to New Brunswick, Canada, as Loyalists, and ownership of the painting descended through their daughter, Catherine Hallett Sinott.