New GPT-4 Passes All Sections Of The Uniform Bar Exam. Maybe This Will Finally Kill The Bar Exam.

// When the ChatGPT hype machine first blew up, folks started to wonder if the GPT engine underlying the product — GPT-3.5, to be precise — could pass the bar exam. It couldn’t. Humans 1, Robots 0.

Alas, the human race failed to yell “stop the count” as ferociously as some law firms did back in 2020, and our future AI overlords have evened the score. Casetext announced this morning that it collaborated with Dan Katz and Michael Bommarito — the research team behind the original GPT-3.5 bar exam challenge — to put GPT-4 through its paces. And it passed.

googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-top-300x250" ); });

From the announcement:

Sponsored Introducing Filevine’s 2023 Customer Benchmark Report Explore new data on how modern law firms are managing their work. From Filevine and Above The Law

Katz and Bommarito had earlier examined the performance of a large language model released in late 2022, finding it could not pass any portion of the UBE. Their forthcoming paper shows that GPT-4, however, passed the multiple-choice portion and both components of the written portion, exceeding not only all prior large language models’ scores, but also the average score of real-life bar exam test takers.

“Exceeding the average score of real-life bar exam test takers.”

googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-middle-300x250" ); }); googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-storycontent-440x100" ); }); googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-in-story-youtube-1x1" ); });

Congratulations to all the bar examiners out there who fought tooth and nail to preserve this anachronistic and quantifiably ill-conceived hazing ritual amid bizarre threats, rank failure, and abject cruelty. In refusing to overhaul attorney licensing, you’ve hitched the profession’s star to a test that a computer can now easily game.

Casetext recently unveiled CoCounsel, providing ridiculously impressive AI-assistance to a number of legal tasks and — more importantly — building a system that could intelligently grasp what it didn’t really know. With GPT-3.5 already caught hallucinating legal conclusions in other contexts, Casetext’s focus on building the right architecture around the engine to keep it from misleading the attorney reflects the whole mission of legal AI over the coming years.

Sponsored Sponsored Understanding Big Law: A Must-Read Guide To One Of The Most Challenging Workplaces In The World Is Now Available From Elizabeth M. Bennett Sponsored What Do Millennials Think Of Law Firm Life? We want to know your views on law firm policies and culture. From Above The Law Sponsored Introducing Filevine’s 2023 Customer Benchmark Report Explore new data on how modern law firms are managing their work. From Filevine and Above The Law Sponsored What Do Millennials Think Of Law Firm Life? We want to know your views on law firm policies and culture. From Above The Law

And the impetus for the company to get involved with Katz and Bommarito on this experiment is the other half of today’s announcement that CoCounsel is now powered by GPT-4.

The implications of GPT-4 for the legal industry go far beyond passing the bar exam, though. GPT-4, when paired with Casetext’s deep legal practice and data security expertise, has made possible a first-of-its-kind professional-level solution lawyers and their clients can rely on. “CoCounsel combines the power of next-generation AI with the security and data privacy law firms require,” said Casetext Chief Technology Officer Dr. Ryan Walker. “Client data is never used to train the models, and law firms retain complete control over their data. CoCounsel is the most secure AI in legal technology.”

GPT-4 promised to be a significant upgrade from GPT-3.5. Passing the bar exam doesn’t mean the engine is ready to replace real lawyers. After all, there are a lot of people who’ve passed the bar exam who shouldn’t be real lawyers. But it does demonstrate a level of competency that can be responsibly focused toward jump starting a lot of legal tasks.

googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-bottom-300x250" ); });

But it does set up the awkward inevitability that CoCounsel will find itself working for a lawyer that it outperformed on the bar exam — demeaning itself to take orders from that one attorney in the firm that’s measurably less competent.

Welcome to being an associate, buddy.

Sponsored Sponsored The Global Legal News You Need, When You Need It News and analysis from this company could be a game-changer for legal professionals working globally. From Ethan Beberness Sponsored Why CRM Is An Essential Tech Tool For Law Firms In 2023 CRMs are powerful tools to help law firms grow and scale without needing more staff. Read on for a brief overview of the basic benefits… From Brittany Cianflone

Earlier: Legal AI Knows What It Doesn’t Know Which Makes It Most Intelligent Artificial Intelligence Of All

Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.

Topics

Bar Exams, Casetext, GPT-4, Technology