Signal, Image and Video Processing Обработка сигналов
Rsatlip: generative adversarial CLIP text-to-image synthesis with recurrent affine transformation and semantic-spatial awareness
Аннотация
Generating images that are consistent with text descriptions remains challenging. In CLIP-based generative adversarial networks (GANs), conditional affine transformations can fuse text and image features, but isolated fusion layers may not share global text context and uniform spatial modulation can affect text-irrelevant regions. We propose RSATLIP, which combines (i) recurrent affine transformation (RAT), using an LSTM-conditioned state to propagate text information across fusion layers; (ii) a semantic-spatial-aware convolutional network (SSACN), whose spatial mask gates text-conditioned affine modulation and is learned from the generation objectives without explicit mask labels; and (iii) the Efficient Multi-Scale Attention (EMA) module, adapted to enhance multi-scale feature modeling along the recurrent path. Experiments on CUB, COCO, Oxford-102, and the SPH dataset show competitive results, with advantages in CLIP-Score and on the fine-grained SPH benchmark.