Everyone is chattering about ChatGPT. Can it pass the bar exam? No, though it performs well on some sections. Which should force a serious reevaluation of the test’s ultimate value to the profession, but instead will convince bar examiners to introduce cavity searches. And, as The Onion points out, ChatGPT was as depressed to take the test as the rest of us.

Can it pass law school exams? Yes… but only if you consider Cs as passing in law school. Perhaps it needs to be programmed to send itself rejection letters from firms so it can take a hint before plunking down tuition for 2L.

googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-top-300x250" ); });

But perhaps these experiments place the cart firmly ahead of the horse. SCOTUSBlog decided to ask ChatGPT 50 questions about the Supreme Court. The results were… well…

Sponsored How George Mason’s Scalia Law School Prepares New Attorneys For A Changing Future Dean Ken Randall spearheads the school’s charge towards interdisciplinary, inclusive degree opportunities. From Above the Law

ChatGPT’s performance was uninspiring. The bot answered just 21 of our questions correctly. It got 26 wrong. And in three questions, its responses were literally true but struck us as incomplete or potentially misleading. You can read all of the questions and ChatGPT’s responses, along with our annotations, here.

That’s pretty bad. Though none of the bot’s whiffs compared to when it couldn’t even figure out which side of Obergefell the justices were on.

googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-middle-300x250" ); }); googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-storycontent-440x100" ); }); googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-in-story-youtube-1x1" ); });

This is what a glitch in the Matrix looks like.

Sponsored Sponsored How George Mason’s Scalia Law School Prepares New Attorneys For A Changing Future Dean Ken Randall spearheads the school’s charge towards interdisciplinary, inclusive degree opportunities. From Above the Law Sponsored What's Your Type (Of Law Department)? A firm grasp of your identity can help you build a next-generation team. From Paragon Legal Sponsored How Are Millennials Changing Law Firm Culture? We want to know about your priorities. From Above The Law Sponsored How Are Millennials Changing Law Firm Culture? We want to know about your priorities. From Above The Law

That’s only half-joking. ChatGPT pulled this idea that Thomas and Ginsburg swapped sides in Obergefell from somewhere online when it spit it out as fact. Someday soon, a few content spamming bots are going to start citing each other about Nelson Mandela dying in prison while eating Stouffer’s Stuffing. The sheer volume of content they put out will rewrite the internet’s “consensus” of history in a flash crash of misinformation.

Can you imagine the repercussions this could have for the legal sector? Imagine basic constitutional history being wholesale rewritten, erasing centuries of accumulated legal understanding as the appellate process cherry-picks and amplifies out-of-context snippets all the way up the chain?

Yeah, I wonder what that would be like.

googletag.cmd.push( function() { // Display ad. googletag.display( "div-id-for-bottom-300x250" ); });

No, Ruth Bader Ginsburg did not dissent in Obergefell — and other things ChatGPT gets wrong about the Supreme Court [SCOTUSBlog]

Sponsored Sponsored The State Of Today’s Corporate Law Departments A report from our friends at Thomson Reuters will help you benchmark legal spend, find greater efficiency, and innovate for the future.  From Thomson Reuters and Above the Law Sponsored Streamline Legal Research And Focus On Decision Making LexisNexis’ Fact & Issue Finder combines AI and human-assisted search to provide digestible, thorough results. From Ethan Beberness

Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.

Topics

Clarence Thomas, Courts, Obergefell v. Hodges, Ruth Bader Ginsburg, Supreme Court, Technology