In this work, we assess both the morphological and genetic diversity of 68 important olive cultivars from three Southern Italian regions: Calabria, Campania and Sicily. Twenty-five phenotypic traits were evaluated and 12 simple sequence repeat (SSR) markers were analysed. All SSR primers were polymorphic and reliable. The total number of alleles per locus varied from 5 to 19 with an average number of 13.1 and a mean polymorphic information content (PIC) of 0.81. These results suggested high genetic diversity within these three olive germplasm collections. Morphological traits also showed significant variability amongst cultivars. Two cases of identity were found and ten statistically significant cases of putative parent/sibling were discovered by performing a SSR-based parentage simulation analysis with CERVUS. The Mantel test indicated low but significant correlations between the morphological data and SSR allelic frequency, origin and SSR allelic frequency, and origin and morphology. Structure software allowed inference of relationships between the three olive germplasm collections and allowed us to obtain the most consistent grouping and to identify putative admixed or exchanged cultivars. Cluster and multivariate analysis, based on morphological traits, revealed geographic grouping in agreement with UPGMA dendrogram and structure analysis using SSRs. Sicilian cultivars showed a more homogenous genetic makeup, probably due to geographical isolation, whilst Calabrian and Campanian cultivars seemed to have a less distinct genetic structure, with a greater degree of intermixing. A correlation between the presence of certain SSR alleles and fruit size was also found.