Baker was the third generation of the family to be born at Bayfordbury Manor. In 1824 Baker inherited the property from his grandfather. During the late 1820s and 1830s he made many improvements to his property. Baker probably took most of his photographs during the 1850s and early 1860s, using materials and methods that represented the height of technical perfection possible with negative paper processes.