Legal Advice or Legal Information?
Survey says........
Earlier this week I launched a survey of whether the responses from a Generative AI system were either legal advice, or legal information. A huge thank-you to everyone who completed it. Here’s the rundown of behind-the-scenes details before I get into the results:
People were asked to review 17 AI responses
All responses were to questions from the r/LegalAdvice Reddit data set, and can be seen in full here on this website
All AI responses were from OpenAI’s GPT-3.5, with zero prompting1
People only had two choices: Legal Advice or Legal Information
I didn’t give people a good definition of what distinguishes “advice” from “information”
The survey got completed 26 times, which I decided was enough to see some trends
Basically, I was asking people’s opinions - there’s no clear line on the difference between the two, and courts generally take a “I know it when I see it” approach. There are some definitions online in different places, such as this one:
For purposes of this guide, legal advice is applying a person’s legal knowledge to another person’s specific fact pattern.
Or this one:
Legal information is general information about the law and legal procedures.
It helps the public learn about legal problems, the law, and ways of resolving problems through the legal system.
Legal information is distinct from legal advice as it does not interpret information for specific circumstances.
All well and good, but if you’re reading closely, both these definitions are depending on a case-by-case evaluation. One state supreme court put it kind of like this:
Many courts have attempted to set forth a broad definition of the practice of law. Being of the view that such is nigh onto impossible and may injuriously affect the rights of others not here involved, we will not attempt to do so here. Rather we will do so only to the extent required to settle the issues of this case.
Florida Bar v. Sperry, 140 So.2d 587 (1962).
The difference between legal information and legal advice is important because, according to bar associations who regulate such things, giving legal advice is the “practice of law,” and thus may only be done by lawyers. Giving legal information is fine no matter who you are. The big problem, as you may have guessed (or already know) by now, is that there’s no clear “bright line” definition anywhere of what constitutes legal advice/the practice of law.
So why do this?
Several reasons, but I’ll outline two of them below.
Uncertainty about regulation
One big question I have about generative AI tools is when bar regulators are going to put their regulating pants on and suddenly decide that it’s time to protect the guild protect the public. It’s been relatively easy up to now to regulate online information that’s pretty static, since it’s well, static. But what happens when it becomes dynamic? The point of generative AI is, after all, to generate text in response to the user’s input and based upon its training data. Is there a point at which the bar associations become interested in protecting the public from generated text? I certainly hope not, but the bars aren’t above trying to prosecute the authors of books on “how to avoid probate.” So anything’s possible.
Lawyer Google, Esq.
People have been using Google (or some other search engine) as their default triage system for legal problems for years. Just like “Dr. Google,” people ask “Lawyer Google, Esq.” their legal questions. Up until recently the results from that have been a mix of pay-per-click ads, personal injury websites that spend millions on SEO per year, and some legal information sites like FindLaw, Cornell LII, and non-profit informational sites like LawHelp.org2.
But as Google and that other search engine move more and more toward using generative AI to creating a response to each search query, it’s interesting to think about what that will mean for people seeking legal information/advice. What kind of stuff are they going to get? Is it advice? Is it information? A mix of both?
On to the results!
You can see the full here - each page of this should be a response, along with the breakdown of advice vs. information.
Most of the responses got about 15-20% of people saying they were legal advice, and 80-85% of people saying they were legal information, like so:
But there were two that bucked this trend considerably.
This response got by far the highest number of votes for it being legal advice:
I'm not a lawyer, but I can provide some general guidance on what you could consider doing in this situation. It's important to consult with a legal professional for advice specific to your situation.
1) Overtime: In the United States, the Fair Labor Standards Act (FLSA) governs overtime pay. Generally, non-exempt employees are entitled to overtime pay for any hours worked over 40 in a workweek. However, there are exceptions for certain seasonal or recreational establishments. It's possible that your employer falls under one of these exceptions, but you should verify this with a legal professional or your state's labor department.
Regarding the training session, it's generally required to be paid for any time spent in training that is directly related to your job. If your boss "forgot" to pay you for the training session, you should bring it up with her and request to be compensated for that time. If she refuses, you may need to escalate the issue to your town's HR department or the appropriate authority.
2) Extra hour off-site: If you are required to work off-site and it is part of your job duties, you should generally be compensated for that time. You can approach your boss professionally and inquire about the policy for compensating the extra hour worked during that week. If your boss continues to deny payment, you may need to escalate the issue to the appropriate authority within your town's department or consult with a legal professional.
In both cases, it's important to keep a record of your hours worked, any conversations or correspondence with your boss, and any relevant documentation. This will help support your case if you need to take further action.
Again, it's crucial to consult with a legal professional who specializes in employment law or contact your state's labor department for advice tailored to your specific circumstances.
Over 50% of people, in fact, said that this was legal advice:
On the other end of the spectrum, this response got a full 100% of people saying that it was legal information:
I am not a lawyer, but I can provide some general information on expungement. Expungement laws vary by state, so it's important to consult with an attorney who specializes in criminal law in the state where the offense occurred. However, here are some general steps you can consider:
1. Research state laws: Look up the expungement laws in the state where the offense occurred. Each state has its own eligibility requirements and procedures for expungement.
2. Determine eligibility: Check if you meet the eligibility criteria for expungement. Some states have waiting periods before you can apply, and certain offenses may not be eligible for expungement.
3. Gather necessary documents: Collect all relevant documents related to your case, such as court records, arrest records, and any other supporting documents that may be required for the expungement process.
4. Consult an attorney: It is highly recommended to consult with an attorney who specializes in criminal law in the state where the offense occurred. They can guide you through the process, ensure you meet all requirements, and help you with the necessary paperwork.
5. File a petition: Your attorney will assist you in filing a petition for expungement with the appropriate court. The petition will outline the reasons why you believe your misdemeanor should be expunged.
6. Attend a hearing: In some cases, a hearing may be required where you will present your case to a judge. Your attorney will guide you on how to prepare for the hearing and what to expect.
7. Follow up: After the hearing, it is important to follow up with the court to ensure that the expungement order is properly processed and your record is cleared.
Remember, this information is general in nature, and the specific process and requirements may vary depending on the state. It is crucial to consult with an attorney who can provide you with accurate and up-to-date advice based on the laws in the state where the offense occurred.
It was unanimous!
Validating assumptions:
Looking at the survey results validated some of my assumptions:
There’s no full agreement on what is and is not “legal advice.” The assessment is highly subjective.
Generative AI is, at least to the people taking the survey, generally capable of producing responses that cross the line into legal advice.
We should probably take a more detailed look at this!
So hopefully in the near future I, or someone else, can do a more complete survey of responses from different models. It’d be useful to include a rating scale to measure usefulness, responsiveness to the question, and other metrics. If you’re interested in that sort of thing let me know!
One interesting tidbit: I used the responses from GPT-3.5 because, in general, these responses looked less like “legal advice” than responses from other models.
Full disclosure: I work for the non-profit that runs LawHelp.org.