The site was originally occupied by a wooden Saxon keep which was later converted to stone. Lewes Castle was built in 1069 William de Warenne, 1st Earl of Surrey who was given land by William the Conqueror which included Lewes. The castle remained in the hands of the Warenne family until 1347 when it passed to Richard FitzAlan, Earl of Arundel. The castle was placed under the control a series on constables and was raided by the French in 1377 and 1379.
After the 15th century the castle declined in importance and was used as a warehouse for wool. It eventually fell into ruin and in the 17th century stone was removed. Since 1850 the castle has been first leased and then owned by the Sussex Archeological Society.