Spencer was raised in Milwaukee, WI. He received his BA from Duke University in 1992 and
his PhD in economics from Duke in 2001. He and his wife Melissa Ruby Banzhaf have two children, Elizabeth and John. They attend North Avenue Presbyterian Church, where Spencer is an ordained elder and previously served as Treasurer.
His wife Melissa Banzhaf is also an economist, specializing in the labor market. With a BA in economics from Duke and PhD from the University of North Carolina at Chapel Hill, she serves as the administrator of the Atlanta Research Data Center.