Built on the eastern side of the quadrangle, it sat almost across from the Industrial Arts Building constructed three years prior.  On the ground floor was one of the first indoor swimming pools in east Texas (34x38 feet); it was later covered over by a gym floor and allegedly forgotten until the building was being prepared for demolition.  The second story contained classrooms, offices, and the gymnasium; the third floor contained the gym gallery and a larger classroom.

An addition to the north was completed in 1960 that extended the building amazingly close to the Peabody Library.  At one time the university post office was located in the gym's basement.

At the suggestion of the Campus Master Plan the gym was demolished; the site is now home to the Rather Communication Building.

Two gymnasiums for men have existed on campus: the first, a frame building, was removed in the 1950s, while the second still stands as Academic Building Three. Both men and women’s gymnasium facilities were later relocated to the Health & Kinesiology Center.