A couple of prehistoric artefacts have been found in Padbury, one Neolithic to Bronze Age flint side scraper at Grange Farm and one Neolithic stone axe that is not closely provenanced. There is also an Iron Age hillfort or enclosure at Norbury Camp; earthworks recorded on a sixteenth century map and crop-marks have been identified on aerial photographs. Other early remains include a few finds of Roman pottery near Grange Farm. A Roman road is known to pass through the parish.
Medieval finds have been made at Grange Farm and a possible medieval or post-medieval moat or enclosure was recorded in field survey on Main Street. There are historical records that Padbury, or Overbury as it was sometimes known, had a watermill from the time of Domesday and the current mill building dates to the seventeenth century, though there are later alterations. The only surviving medieval building is St Mary's church. This has a thirteenth century nave and chancel, fourteenth century aisles and seventeenth century tower. There was some alteration made in the sixteenth century and restoration in the nineteenth century. Later Victorian work on the church uncovered some fourteenth century wall-paintings in the north aisle.
There are many listed buildings in Padbury, many of them timber-framed and dating back to the sixteenth or seventeenth centuries like Stratfords Cottage, Trefoil Cottage or the old vicarage. The village cross at Downe Cross was known to be standing in the sixteenth century, though it is now gone. There are also some later, brick built, eighteenth and nineteenth century houses, like the White House. There are many other monuments from the nineteenth century, such as the Village Hall, which was a school and several railway bridges around the parish. The clay-pit at Norbury is all that is left of a brick and tile-works that is known from trade directories of the nineteenth and early twentieth centuries. The nineteenth century railway station was closed in 1964 and has since been built over with a housing estate. The parish also had a windmill, as known from historic maps and the field-name Windmill Furlong.