Two hypotheses, discussed by Burke and Arnold (2001):
The role of epistasis in adaptive evolution has been a controversial issue ever since Sewall Wright and R.A. Fisher first formalized their views in the early 1930s. According to Wright (113, 114), natural selection retains favorably interacting gene combinations. Therefore, as a result of the highly integrated nature of the genome, selection may lead to the production of what Dobzhansky (43) has termed "coadapted" gene complexes. In contrast, Fisher (48) argued that natural selection acts primarily on single genes, rather than on gene complexes. In Fisher's view, therefore, selection favors alleles that elevate fitness, on average, across all possible genetic backgrounds within a lineage. Such alleles have been termed "good mixers" (75). Regardless of the role of epistasis within lineages, however, negative epistasis in a hybrid genetic background, or hybrid incompatibility, is fully consistent with both the Wrightian and Fisherian worldviews. This is because allelic fixation occurs in any one lineage without regard to the compatibility (or lack thereof) of new alleles with those in any other lineage. Hybridization then produces a vast array of recombinant genotypes that have never before been subjected to selection. On average, these genotypes will be less well adapted than their parents, giving rise to some level of selection against hybrids.
Hybrid breakdown, or the reduction in fitness of segregating hybrid progeny that often results from intercrossing genetically divergent populations or taxa, has long been taken as evidence of unfavorable interactions between the genomes of the parental individuals (e.g., 39, 42, 43, 75, 80). The most widely accepted genetic model for the occurrence of such incompatibilities was first described by Bateson (15, as cited in 83), and later by Dobzhansky (39) and Muller (79, 80). In short, the Bateson-Dobzhansky-Muller (BDM) model assumes that an ancestral population consisting solely of individuals of the genotype aa/bb is broken into two parts that are temporarily isolated from each other. In one subpopulation, a new allele (A) is then assumed to arise at the first locus. Meanwhile, a new allele (B) is assumed to arise in the other subpopulation. Because individuals of the genotype aa/bb, Aa/bb, and AA/bb can interbreed freely, the A allele can then spread to fixation in the first subpopulation; likewise, individuals of the genotype aa/bb, aa/Bb, and aa/BB can interbreed freely, and the B allele spreads to fixation in the second subpopulation. However, although A is compatible with b, and B is compatible with a, the interaction of A with B is assumed to produce some sort of developmental or physiological breakdown, such that hybridization between the two subpopulations leads to the production of offspring with decreased levels of viability and/or fertility. Although this model focuses on negative interactions between differentiated regions of the nuclear genome, similar interactions between one or more regions of the nuclear genome and some component of the cytoplasm (e.g., the chloroplast or mitochondrial genome) could also play an important role in hybrid incompatibility. Unfortunately, the BDM model does not provide any mechanistic explanation as to how mutations that are neutral (or beneficial) within a given lineage will produce strongly disadvantageous incompatibilities when combined in a hybrid background (Burke and Arnold 2001, emphasis added).
References:
Burke JM, Arnold ML. 2001. Genetics and the fitness of hybrids. Annu Rev Genet 35:31-52. DOI link
