The community of Stoughton began when Luke Stoughton, a Vermont Yankee living in the Janesville area, purchased 800 acres of land nestled in a large bend of the Catfish River (now the Yahara), on July 3, 1847. He platted the town, built a dam, lumber mill, and general store, and began to publicize his community to relatives and friends. The village grew steadily, and in 1853, Luke Stoughton offered free land to the railroad if it would pass through Stoughton. The railroad's presence assured the city's future prosperity, and made the village a center for agriculture.

By 1868, when Stoughton became an incorporated village, the population had grown to 950, with most citizens being of Yankee descent. Most of the town lay between the river and the railroad tracks. The community's economic base was still agricultural. After the Civil War, however, a manufacturing economy began to develop, with T.G Mandt's Wagon Works being the most important industry. A second wagon works was opened, and by World War I, a total of over 800 men were employed. At the same time, leaf tobacco became the important local crop, with hundreds of people, many of them women, employed in the seventeen tobacco warehouses in town.