Science China Life Sciences General Environmental Science
Uncovering global gene action patterns via learnable genotype encoding
Abstract
Genomic prediction models such as GBLUP and its additive-dominance extension (GBLUP-AD) rely on predefined fixed genotype encodings, which implicitly assume symmetric allele effects and often fail to capture non-canonical gene action. We propose LatentGP, an end-to-end differentiable Gaussian process framework that learns genotype encoding as a model parameter and jointly estimates latent genotype representations and variance components through marginal log-likelihood maximization. Simulations demonstrated that LatentGP can recover complex gene action modes, particularly asymmetric super-additive patterns that were poorly captured by conventional models. In hybrid pig and maize populations, LatentGP achieved relative accuracy improvements ranging from 0.1% to 10.9% compared with baseline methods. Beyond predictive performance, visualization of the learned embeddings revealed asymmetric super-additive patterns underlying pig production traits and nonlinear dominance patterns for maize yield. In human disease cohorts, LatentGP automatically converged to a linear additive encoding, providing a negative control that highlights its ability to self-regulate according to the underlying genetic architecture. By shifting from fixed genotype coding to adaptive representation learning, LatentGP uncovers diverse global gene action patterns while preserving the statistical rigor of linear mixed models. The software is freely available at https://github.com/hyacz/LatentGP .