Phenotypic variation controls the species interactions which determine whether or not species coexist. Long-standing hypotheses in ecology and evolution posit that phenotypic differentiation enables coexistence by increasing the size of niche differentiation. This hypothesis has only been tested using macroscopic traits to date, but niche differentiation, particularly of microscopic organisms, also occurs at the molecular and metabolic level. We examined how phenotypic variation that arises at the level of gene expression over evolutionary time affects phytoplankton species interactions and coexistence. We predicted that similarity in gene expression among species would decline with phylogenetic distance, and that reduced similarity in gene expression would weaken competition, increase facilitation and promote coexistence. To test this, we grew eight species of freshwater green algae in monocultures and bicultures for 46 days in a laboratory microcosm experiment. We quantified the strength of species interactions by: (i) fitting Lotka–Volterra models to time-series densities and estimating interaction coefficients, and (ii) calculating relative densities that compare species’ steady-state densities in biculture to those in monoculture. We used Illumina high throughput sequencing to quantify the expression of 1253 families of homologous genes, including a set of 17 candidate genes that we hypothesized a priori to be involved in competition or facilitation. Synthesis. We found that closely related species had greater similarity in gene expression than did distantly related species, but as gene expression became more similar, species experienced weaker competition or greater facilitation, and were more likely to coexist. We identified gene functional categories that were uniquely differentially regulated in association with particular species interaction types. Contrary to common thinking in ecology and evolution, similarity in gene expression, and not differentiation, was associated with weaker competition, facilitation and coexistence.