Two priests from Christ Church, Savannah, were the first to serve the Episcopal residents of Milledgeville: the Reverend John W. Bartow, rector, and the Reverend Edward Neufville. They held services in Milledgeville as early as 1832. Nine years later, on April 4, 1841, St. Stephen’s was incorporated by the state of Georgia and is the eighth oldest Episcopal Church in the state and the third oldest in what is now the diocese of Atlanta. The structure is the oldest church building in Milledgeville.
During the brief occupation of Milledgeville by General Sherman’s forces in bitterly cold weather in late November 1864, soldiers from the 107th New York Infantry Regiment took shelter in St. Stephen’s, as well as other churches on the square. They not only burned pews but also poured syrup into the pipes of St. Stephen’s organ. When the nearby magazine and arsenal were blown up as the troops left on their famous March to the Sea, the roof was damaged and the windows were blown out.
The stained glass window over the altar was a gift of Christ Church, Savannah, in appreciation for hospitality extended during the Civil War years. The other lancet arched windows in the church date from the turn of the century and celebrate the lives of some of the communicants as well as the Reverend J. M. Stoney, rector from 1873 to 1884. They are unusual in that portions of them are painted.