The organoleptic characteristics of virgin olive oil (VOO), together with its nutritional and health properties, have led the olive tree to be cultivated beyond the Mediterranean basin, reaching latitudes with colder climates, with minimum temperatures below 0◦ C and with a higher probability of early frosts. The freezing of olives generates destruction within the tissues and promotes degradation reactions. In this study, the impact of mild frost occurring at different harvesting times on the composition of volatiles and phenolic compounds in VOO were investigated. Arbequina variety olives were harvested at different stages of ripening. Half of the olives were subjected to oil extraction and the other half were frozen at −3 ± 1◦ C for 12 h prior to oil extraction. A significant decrease of phenolic compounds with harvesting time was observed in both types of oils (fresh and frozen olives). Oils from frozen olives presented a slightly higher content of total phenols, except in the advanced ripening stage (September), and a slightly lower content of volatile compounds at all harvesting times. In addition, a higher content of 3,4-DHPEA-EDA was observed in oils from frozen olives, which is attributed to an early action of the endogenous β-glucosidase enzyme on oleuropein in freeze-damaged olive fruits. Principal component analysis and Discriminant Partial Least Square Regression allowed the oils to be classified according to the type of fruit (fresh and frozen) and the month of harvest. This study would indicate that mild frost would have a low impact on the chemical composition of virgin olive oil, although, this depends on the ripening stage.