Jeffery's wife was a descendant of Paul Smith Palmer; through her, Jeffery inherited companion portraits of Paul Palmer and his wife, Hannah Eells Palmer, painted by Erasmus Salisbury Field around 1835/38. The pictures (now in the NGA) descended in the Palmer family.