In 1953-54 the library was started with 2,300 books donated by the citizens and the Texas State Library. October 17, 1954, the public was invited to an open house. In 1957, a lot donated by the heirs of E.M. Scarbrough to the city was offered as the site for a proposed George Sessions Perry Memorial Library. Perry was an author and native son of Rockdale. Hope for a Perry Library faded, however, due to funds being unavailable at that time.
In 1961, Dr. George Hill Patterson made a $12,500 donation in memory of his mother who was born in Rockdale. The city provided matching funds and on May 25, 1963 the dedication of the new Lucy Hill Patterson Memorial Library was held. Dr. Patterson was pleased with the acceptance and operation of the library and provided a generous gift in his will for library expansion. In 1980 the library was completely remodeled, tripling in size with a second story.
With a staff of three and an active volunteer program, a variety of programs and services are offered to patrons of all ages. The library currently works closely with the Milam County Genealogical Society and is a member of the Central Texas Library System.