Can a chatbot earn a JD? This one averaged C-plus on law school exams

  • Home
  • Daily News
  • Can a chatbot earn a JD? This one averaged…
  • Artificial Intelligence & Robotics Can a chatbot earn a JD? This one averaged C-plus on law school exams
    •  
    • !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs'); Tweet  
    • lang: en_US  
    • reddit_target='law';  
    • Print.

    shutterstock_Screen with ChatGPT

    Image from Shutterstock.

    An artificial intelligence tool called ChatGPT averaged a C-plus on exams at the University of Minnesota Law School, according to four law professors who gave it a try.

    The law professors used ChatGPT to answer the questions and then blindly graded the answers, along with answers by real students, report Reuters and Insider.

    The average C-plus grade was still below that of law students, who had a B-plus average. And ChatGPT’s performance, while earning passing grades, was at or near the bottom of the class.

    The professors’ findings are available here.

    Reuters described ChatGPT as a “wildly popular” free AI tool that “generates sophisticated, humanlike responses based on requests from users and mountains of data, including from legal texts.”

    ChatGPT earned its highest grade of a B in a constitutional law class focusing on federalism and separation of powers but still placed 36th out of 40 students. It got a B-minus in employee benefits (placing 18th out of 19 students), a C-minus in taxation (placing 66th out of 67 students), and a C-minus in torts (placing 75th out of 75 students).

    The chatbot’s performance on multiple choice questions was generally worse than the essays, but the number of right answers was better than chance. It did the worst on questions involving math, dragging down its score on the taxation exam.

    ChatGPT did better on essay questions, but its performance was “highly uneven,” the study found. Sometimes it matched or even exceeded the performance of real students. Its essays “were typically clear and well-crafted—perhaps even suspiciously so compared to real students writing a time-limited exam,” the study said. And ChatGPT “displayed a strong grasp of basic legal rules.”

    But when the essay answers were incorrect, “they were dramatically incorrect, often garnering the worst scores in the class,” the study said. ChatGPT “struggled to identify relevant issues and often only superficially applied rules to facts as compared to real law students.”

    Lead study author Jonathan H. Choi spoke with Reuters about the findings.

    “Alone, ChatGPT would be [a] pretty mediocre law student,” Choi said.

    But he and the other authors of the study pointed out that ChatGPT was competing against students at a “highly selective” law school, and its grades were sufficient to earn a JD degree.

    See also:

    ABAJournal.com: “AI program earned passing bar exam scores on evidence and torts; can it work in court?”

    ABAJournal.com: “Should lawyers embrace or fear ChatGPT?”

    (adsbygoogle = window.adsbygoogle || []).push({});

    $( document ).ready(function() { var slidesFetched = false; var handleGalleryClick = function() { var $oldItem = $('.gallery_container'); $('.gallery_container .arrows a').click(function(e) { e.preventDefault(); $arrow = $(this); if (! slidesFetched) { $arrow.closest('.gallery_slide').css('opacity', '0.3'); var url = $(this).attr('href').replace('/gallery/', '/gallery/slides/'); $.get(url, function(data) { var $slides = $(data); $slides = $slides.filter('.gallery_container'); $('.gallery_container').replaceWith($slides); $slides = $('.gallery_container'); $slides.find('.arrows a').click(function(e2) { e2.preventDefault(); var $slideWrapper = $(this).closest('.gallery_container'); var $arrowContainer = $(this).closest('.arrows'); if ($arrowContainer.hasClass('previous')) { $slideWrapper.removeClass('active'); $slideWrapper.prev().addClass('active'); } else if ($arrowContainer.hasClass('next')) { $slideWrapper.removeClass('active'); $slideWrapper.next().addClass('active'); } changeSlide($('.cell-'+$(this).attr('data-slideid'))); }); changeSlide($('.cell-'+$arrow.attr('data-slideid'))); slidesFetched = true; }); } changeSlide($('.cell-'+$(this).attr('data-slideid'))); return false; }); var changeSlide = function($cell) { $('.slide-'+$cell.attr('data-slideid')).addClass('active').siblings().removeClass('active'); $cell.addClass('active').siblings().removeClass('active'); moveFilmstripCells($cell); // Hide filmstrip arrows if we're on the first/last slide. if ($('.gallery_container.active').find('.arrows.previous').length > 0) { $('#filmstrip_container .arrows.previous').css('visibility', 'visible'); } else { $('#filmstrip_container .arrows.previous').css('visibility', 'hidden'); } if ($('.gallery_container.active').find('.arrows.next').length > 0) { $('#filmstrip_container .arrows.next').css('visibility', 'visible'); } else { $('#filmstrip_container .arrows.next').css('visibility', 'hidden'); } }; var $cells = $('#filmstrip_container .cell'); var totalCells = $cells.length; var totalVisibleCells = $cells.filter('.visible').length; var pivotPoint = Math.floor(totalVisibleCells / 2); var moveFilmstripCells = function($activeCell) { var $visibleCells = $cells.filter('.visible'); var activeCellIndex = $cells.index($activeCell); var activeCellVisibleIndex = $visibleCells.index($activeCell); var firstVisibleCellIndex = $cells.index($cells.filter('.visible:first')); var lastVisibleCellIndex = $cells.index($cells.filter('.visible:last')); var cellsToMove = 0; var startingPoint = 0; // Do nothing if active cell is at the pivot point. if (activeCellVisibleIndex == pivotPoint) return; // Active cell is left of the pivot point. if (activeCellVisibleIndex < pivotPoint) { // Do nothing if the cell's index and visible index are identical, because that means // we don't have room to shift. if (activeCellIndex

    Introducing Jobbguru: Your Gateway to Career Success

    The ultimate job platform is designed to connect job seekers with their dream career opportunities. Whether you're a recent graduate, a seasoned professional, or someone seeking a career change, Jobbguru provides you with the tools and resources to navigate the job market with ease. 

    Take the next step in your career with Jobbguru:

    Don't let the perfect job opportunity pass you by. Join Jobbguru today and unlock a world of career possibilities. Start your journey towards professional success and discover your dream job with Jobbguru.

    Originally posted on: https://www.abajournal.com/news/article/artificial-intelligence-chatbot-averaged-a-c-plus-on-law-school-exams/?utm_source=feeds&utm_medium=rss&utm_campaign=site_rss_feeds