The State Bar of California has revealed that certain multiple-choice questions on its recent February 2025 bar exam were created with the assistance of artificial intelligence (AI), shedding new light on the challenges faced during the administration of the test. This development comes amid widespread complaints and technical difficulties experienced by examinees who undertook the problem-plagued exam earlier this year.

According to the State Bar’s Monday news release, it plans to request the California Supreme Court to adjust the scores for those who sat the February exam. The exam itself was marked by significant issues, including repeated crashes of the online testing platform prior to some candidates even beginning, along with problems such as trouble finishing and saving essays, screen lags, error messages, and restrictions on copying and pasting text. Such difficulties sparked frustration and concern among test-takers and legal educators alike.

Further details emerged during a recent presentation by the State Bar, which disclosed that of the 171 scored multiple-choice questions, 100 were created by Kaplan, a well-known test preparation company, while 48 questions were sourced from a first-year law student exam. Notably, a smaller group of 23 questions, scored and developed with the involvement of artificial intelligence, were produced by ACS Ventures, the State Bar’s psychometrician.

Leah Wilson, executive director of the State Bar, expressed confidence in the fairness and accuracy of the AI-assisted questions, stating to the Los Angeles Times, “We have confidence in the validity of the (multiple-choice questions) to accurately and fairly assess the legal competence of test-takers.”

The admission of AI use has generated strong reactions within the legal education community. Mary Basick, assistant dean of academic skills at the University of California, Irvine, Law School, described the situation as “worse than we imagined” and found the use of AI by non-lawyers to draft questions for the exam “just unbelievable.” Speaking to the Los Angeles Times, Basick said, “The debacle that was the February 2025 bar exam is worse than we imagined.”

Similarly, Katie Moran, associate professor at the University of San Francisco School of Law, noted the gravity of the disclosure. “It’s a staggering admission,” Moran said. She raised concerns about the process, stating, “The State Bar has admitted they employed a company to have a non-lawyer use AI to draft questions that were given on the actual bar exam. They then paid that same company to assess and ultimately approve of the questions on the exam, including the questions the company authored.”

Andrew Perlman, dean of Suffolk University Law School and an advisory council member of the American Bar Association Task Force on the Law and Artificial Intelligence, commented on the broader implications of AI’s use in legal examinations. Although he had not previously heard of AI being utilised to develop bar exam questions or associated standards governing such uses, he acknowledged the growing role of AI technology in legal education. Perlman remarked, “AI can be useful for developing questions for assessment, but a critical guard rail is making sure that everything that comes from an AI tool is vetted carefully by experts in the subject matter.” He anticipates that AI’s role will continue to expand, adding, “we will be worried in the future about the competence of lawyers who don’t use these tools.”

The controversies surrounding this particular bar exam and the involvement of AI in question development have opened new discussions about the integration of technology in legal certification processes and the necessary safeguards to maintain exam integrity. The State Bar of California’s decision to seek score adjustments reflects ongoing efforts to address the challenges experienced by this cohort of candidates.

Source: Noah Wire Services