The origin of Albion College lies not in the city of Albion, but about 10 miles (16 km) southeast of the present location of the college. On March 23, 1835, Methodist Episcopal settlers in Spring Arbor Township obtained a charter for the Spring Arbor Seminary from the Michigan Territorial Legislature. Foundations for a building were begun in 1837 at a location about 3 miles (4.8 km) southwest of the current village of Spring Arbor but were soon abandoned due to the economic turmoil caused by the Panic of 1837. No classes were ever held at the Spring Arbor location. The trustees applied to move the seminary to Albion in 1838, and the legislature approved the move in 1839.

With 60 acres (243,000 m²) of land donated by Albion pioneer Jesse Crowell, the cornerstone was laid for the first building in 1841. The seminary, now named the Wesleyan Seminary, first held classes in 1843, in the local Methodist Church. In 1844, classes began in the newly constructed Central Building, which was rebuilt as the present Robinson Hall in 1907.

The Albion Female Collegiate Institute was founded in 1850 by the Wesleyan Seminary Corporation. The two schools merged in 1857 under the name The Wesleyan Seminary and Female College at Albion.

On February 25, 1861, both schools were completely merged under the name Albion College when the school was fully authorized by the State legislature to confer a full four-year college degree upon both men and women.