EVALUATING THE ACCURACY OF CHATGPT GENERATED TRANSLATIONS OF LEGAL EXPRESSIONS USING DIFFERENT PROMPTING STRATEGIES

The growing use of ChatGPT in translation has expanded to legal translation. However, the specialized nature of legal language poses significant challenges for machine-generated translations, particularly in preserving the accuracy of legal meaning. This study investigates the types of legal expressions found in selected legal agreements and evaluates the accuracy of ChatGPT-generated translations produced using different prompting strategies. Using a descriptive qualitative approach, this study analyzed 85 legal expressions extracted from English legal agreements. The expressions were classified according to Veretina’s (2012) legal language characteristics and Varo and Hughes’s (2014) classification of technical terms, then translated by ChatGPT using three prompting strategies: Basic, Role-based, and Domain-specific. Translation accuracy was assessed by an expert legal translator using Nababan et al.’s (2012) translation quality assessment instrument. Six types of legal expressions were identified: Semi-Technical Terms (27.06%), Doublets (23.53%), Purely Technical Terms (18.82%), Triplets (11.76%), Archaisms (9.41%), and Foreign Terms (9.41%). Basic Prompting achieved the highest accuracy score (2.17), followed closely by Domain-specific Prompting (2.16) and Role-based Prompting (2.11). Although Domain-specific Prompting produced the greatest number of accurate translations, all three strategies were classified as Less Accurate according to Nababan et al.’s (2012) accuracy scale. The minimal differences in accuracy scores suggest that prompting strategies have only a limited effect on the accuracy of ChatGPT-generated translations of legal expressions. Therefore, expert human review remains necessary, particularly for expressions involving specialized legal terminology and legal drafting conventions.