The RMA was established in order to operate as a history room in the basement of the City of Richmond’s Carnegie Library finished in 1910. The building became a branch library in 1949 after World War II dramatically changed the demography of Richmond and then a children’s library in the 1960s. In the early 1980s, the Carnegie Library underwent an extensive renovation in order to convert the building into a space that was suitable for a local museum. The City of Richmond retains ownership of the Carnegie Library and allows the RMA to operate a museum here.

The SS Red Oak Victory was constructed in the Kaiser Shipyards in Richmond on November 9th, 1944 and carried ammunition to allied ships in the Pacific theater during the final days of World War II. The ship was used by various commercial enterprises after the war, only to be reactivated for cargo missions during the Korean and Vietnam conflicts. In 1969, the ROV was decommissioned for the last time and stricken to languish in the Naval Ghost Fleet in Suisun Bay. When the RMA learned the ROV was to be disassembled for scrap metal in the 1990s, they enlisted the assistant of Congressman George Miller to save it. In the late 1990s, Congress passed an act of legislation to convey ownership of the ship to the RMA and it has been under restoration since that time.