Artificial Intelligence and Law Искусственный интеллект

Can general large language models (LLM) pass a hands-on patent attorney test? A study on LLM capabilities

Bhakti Khera, Rezvan Alamian, Pascal A Scherz, Stephan M Goetz
Springer Nature · 2026-09-20 · DOI 10.1007/s10506-026-09543-4

Аннотация

Abstract The legal field already uses various large language models (LLMs) in real-world applications. The practical use in the patent domain concentrates on general pre-trained and foundation models without domain specificity. However, little is known and documented about their performance; their limitations are poorly studied and documented. Quality control and any targeted improvement of natural language processing in patent law require such information. We evaluated several open-source and proprietary LLMs—including the GPT series, Anthropic, DeepSeek, and Llama-3 variants—on parts of the European qualifying examination (EQE) for future European patent attorneys. The chosen tests are intentionally technical and designed with little room for interpretation so that models cannot hide wrong associations and answers behind apparent eloquence. OpenAI o1 led with 0.82 accuracy and 0.81 F1 score, whereas AWS (Amazon Web Services) Llama 3.1 8B lagged at 0.50 accuracy, and a Python-deployed Llama 3.1 8B scored 0.55. The latter two are within the range of mere guessing for the two-option forced-choice design. None of the evaluated models could have fully passed the examination, as accuracy never exceeded the average threshold of 0.90 required for professional-level standards—also not models that are regularly promoted for their assumed beyond-PhD- and bar-admitted-lawyer-level performance. GPT-4o excelled at integrating text and graphics, whereas Claude 3 Opus often struggled to maintain formatting coherence. Human patent experts evaluated the textual justifications to identify and understand problems at the argument level. They uncovered various critical shortcomings of each model. They valued clarity and legal rationale over the raw correctness of the answers, which revealed misalignment between automatic metrics and expert judgment. Model outputs were sensitive to modest temperature changes and prompt wording, which underscores the continued necessity of expert oversight. Although the arguments and our analysis of the training corpora demonstrate that the relevant legal texts were part of pre-training, the models, e.g., their embeddings, demonstrated potential for improvement in domain-specific language aspects. Future work should target logical consistency, robust multimodality, and adaptive prompting to approach human-level patent proficiency. In summary, despite the outstanding performance of recent large models, the general public might overestimate their capabilities. The field has a long way to go to develop a virtual patent attorney. This paper aims to point out several specific limitations that need solutions.

Читать у издателя →