The real challenge in day-to-day legal work in somewhat more complex cases is probably to quickly find relevant precedents, create suitable and understandable legal documents and offer comprehensible advice. This process is time-consuming and often requires the involvement of legal experts who specialize in the specifics of the individual systems.

The research presented here shows that AI can quickly become overwhelmed in the current phase of development and one (legal personal) should therefore rely on a more graduated, if not granulated procedure.

https://hugodutka.com/posts/answering-legal-questions-with-llms/

 

"

The Problem

A great answer to a legal question must at least:

• Be based on sound reasoning;
• Quote the source text of the law to be verifiable; and
• Take into account all relevant parts of the law.

To meet these requirements, we had to put the relevant documents in the prompt - we couldn’t just hope that the LLM was trained on all of the law. We also limited our scope to a single regulation. It made the problem approachable, and we could scale later.

In our first attempts, we fed the entire 226 pages of the EU´s AI Act into GPT-4 and asked a sample question:

Does the deployment of an LLM acting as a proxy to optimize SQL queries fall within the regulatory scope of the EU’s AI Act?"

And we found that GPT-4 couldn’t give us a good answer.

A lawyer would start by asking some helper questions:

1. Does the proxy meet the definition of an AI system?
2. Can the proxy be classified as a high-risk AI system?
3. Will the proxy process personal or sensitive data?

But in a single response, GPT-4 couldn’t both break down the question and answer it. The former task requires a high-level analysis of the document, and the latter - low-level focus on details.

 

The Solution

We split answering the question into subtasks.

1. Make GPT-4 figure out which subquestions it should ask; then
2. Answer each subquestion independently; and
3. Aggregate the findings into a single response.

"

These are certainly valuable tips for legal practitioners when dealing with AI-supported software.