The evocative basement of the palace contains the remains of a Romanesque cloister and the crypt of the "Ugonian Basilica" (11th cent.), decorated with very rare coeval frescoes and a Roman sarcophagus with the myth of Selene and Endymion that housed the coffin of the patron saint Rufinus. The museum preserves interesting and valuable artwork from the cathedral, enhancing its historical and religious significance. Completing the site visit are the Cloister and Crypt of St. Rufinus, both from the 11th century. The crypt, an evocative environment located under the nave of the cathedral, is one of the places where Francis used to retire in prayer when he preached in the cathedral.

Beneath today's Town Square, dominated by the Roman Temple of Minerva, the ancient Roman Forum can be visited, which was found during excavations in 1836. The remains of the Forum include the base of the temple with access gates to a monumental cistern, a podium with seats for magistrates, and a tetrastyle temple dedicated to Castor and Pollux.