Science China Information Sciences Компьютерные науки (общие)

Expert policy support expansion: alleviating the overfitting issue of adversarial imitation learning for high-dimensional tasks

Han, Fuguang, Zhang, Zhilong, Li, Yichen, Chen, Feng, Yu, Yang, Zhang, Zongzhang
Springer Nature · 2026-11-04 · DOI 10.1007/s11432-024-5205-7

Аннотация

We consider imitation learning (IL), which mimics a policy from finite expert trajectories. A prevalent framework of IL is adversarial imitation learning (AIL) that trains a discriminator to distinguish samples from the expert and the imitator. Although AIL achieves impressive performance on some artificial tasks, we observe that the learned discriminator may overfit to the limited expert data, especially when encountering tasks with high-dimensional state spaces. We find that the imbalance between expert demonstrations and generated samples during the training of the discriminator can affect its training margin, consequently resulting in overfitting of the discriminator. Building upon this, we propose expert policy support expansion (EXPOSE). Apart from the discriminator and the generator, EXPOSE additionally learns two random network distillations (RNDs), namely, an expert RND (e-RND) and a policy RND (p-RND). The p-RND incentivizes the imitator to explore the state-action space, while the e-RND filters out expert-level samples from the imitator and adds them to the expert dataset, explicitly expanding the support of the expert policy. Extensive empirical results on tasks from the DeepMind Control Suite and Meta-World benchmarks demonstrate that EXPOSE achieves an average improvement of 40% in overall performance compared to baselines.

Читать у издателя →