Enslaved laborers and craftsmen built Chatham Manor in 1771 at the direction and financing of William Fitzhugh. The large Georgian structure’s location atop Stafford Heights and its visibility from Fredericksburg put the Fitzhughs’ wealth and status on display for everyone around. In addition to owning hundreds of thousands of acres and at least one hundred enslaved people, the Fitzhughs were related to and good friends with other well-known Virginians like George Washington, Thomas Jefferson, and George Mason. The non-stop social requirement of hosting friends and relations drained Fitzhugh's purse. As a result, Fitzhugh sold Chatham 35 years after moving in.
At the time of the Civil War, Confederate supporters Betty and J. Horace Lacy owned Chatham. The Lacys resided at Chatham in the winter and lived in their other home, Ellwood, in the summer. At the outbreak of war, J. Horace Lacy volunteered to serve in the Confederate Army. Betty, meanwhile, packed up the household and left Chatham empty, which al