Henry Pollard was the second son of Sir Christopher Willoughby, Bt., and his second wife, Martha Evans. He succeeded his brother, Sir Christopher William Willoughby, 2nd Bt., who died unmarried in 1813. Sir Henry served as a Member of Parliament for Tarmough Isle of Wight, and later Evesham. He also died unmarried, and thus was succeeded by his younger brother, Sir John Pollard Willoughby, as 4th Bt.