Evolutionary theory has produced two conflicting paradigms for the adaptation of a polygenic trait. While population genetics views adaptation as a sequence of selective sweeps at single loci underlying the trait, quantitative genetics posits a collective response, where phenotypic adaptation results from subtle allele frequency shifts at many loci. Yet, a synthesis of these views is largely missing and the population genetic factors that favor each scenario are not well understood. Here, we study the architecture of adaptation of a binary polygenic trait (such as resistance) with negative epistasis among the loci of its basis. The genetic structure of this trait allows for a full range of potential architectures of adaptation, ranging from sweeps to small frequency shifts. By combining computer simulations and a newly devised analytical framework based on Yule branching processes, we gain a detailed understanding of the adaptation dynamics for this trait. Our key analytical result is an expression for the joint distribution of mutant alleles at the end of the adaptive phase. This distribution characterizes the polygenic pattern of adaptation at the underlying genotype when phenotypic adaptation has been accomplished. We find that a single compound parameter, the population-scaled background mutation rate $\Theta_{bg}$, explains the main differences among these patterns. For a focal locus, $\Theta_{bg}$ measures the mutation rate at all redundant loci in its genetic background that offer alternative ways for adaptation. For adaptation starting from mutation-selection-drift balance, we observe different patterns in three parameter regions. Adaptation proceeds by sweeps for small $\Theta_{bg} \lesssim 0.1$, while small polygenic allele frequency shifts require large $\Theta_{bg} \gtrsim 100$. In the large intermediate regime, we observe a heterogeneous pattern of partial sweeps at several interacting loci.

Hybridization is common in nature, even between "good" species. This observation poses the question of why reinforcement is not always successful in leading to the evolution of complete reproductive isolation. To study this question, we developed a new "quasi-linkage disequilibrium" (QLD) approximation to obtain the first analytic results for the evolution of modifiers that increase mate discrimination against hybrids and heterospecifics. When such modifiers have small effects, they evolve more readily under a one-allele than a two-allele mechanism (sensu Felsenstein 1981). The strength of selection on the modifier decreases as hybrids decrease in frequency, and so further reinforcement may not occur once hybridization is sufficiently rare. The outcome is qualitatively different when modifiers have large effects, however, for example when a single mutation can cause complete reproductive isolation. In this case modifiers in a two-allele mechanism can be selected as or more strongly than those in a one-allele mechanism. Further, they can spread under quite general conditions. Thus whether complete closure of genetic introgression by reinforcement occurs may depend on the size of effects that mutations have on the sensory systems used in mate choice.