Artificial Intelligence (Al) is transforming the way we do business, and it’s changing the way we do legal. Forever!
According to Seun Adebiyi, former corporate attorney at Goldman Sachs and participant in the AI vs lawyers study:
“We are seeing disruption across multiple industries by increasingly sophisticated uses of Artificial Intelligence. The field of law is no exception.”
But who do you think would win in a contract reviewing contest – a lawyer or AI?
And how would you define the parameters of “winning”?
Faster? More accurate? Something else entirely?
In a landmark study conducted by Superlegal’s enterprise brand Lawgeex – AI vs Lawyers – 20 lawyers were put to the test against our revolutionary AI and you’re going to be shocked by the results!
Twenty US-trained lawyers, with decades of legal experience were asked to issue-spot legal issues in five standard NDAs. These lawyers competed against an AI system that was trained on tens of thousands of contracts.
This study represents a major landmark in the history of legal technology.
Before we dive into the groundbreaking results of AI vs Lawyers – let’s look into how the AI was custom-built for the challenge of contract review.
AI for contract review
The typical Fortune 1000 company has 20,000 to 40,000 active contracts at any one time. I know, crazy!
And the International Association for Contract & Commercial Management (IACCM) found that 83% of businesses are dissatisfied with their organization’s contracting process.
So the need for a contract review solution that can take care of the thousands of routine contracts businesses must sign on a daily basis is apparent and growing!
But training AI to perform work just like a lawyer has its challenges!
Legal AI Challenges
-
Legalese
“Legalese” is a technical legal language that is often complex and counterintuitive.
For the purpose of Al training, this form of language cannot be considered a natural language. For contract review, Natural Language Processing (NLP) and off the-shelf solutions simply don’t work.
No existing computational language models could read legalese coherently.
-
High Accuracy
The primary role of a lawyer is to minimize legal risk for their clients, so accuracy is vital.
In legal Al training, single document analysis requires much higher accuracy levels than, for instance, big data “sentiment” analysis (which is the process of using text analytics to mine data sources for opinions in order to predict trends).
Training the AI
The Superlegal technology has been developed through a combination of supervised and unsupervised learning techniques, that according to Professor Yonatan Aumann, lecturer in the Department of Computer Science at Bar Ilan University, has created:
“the most advanced technology for the automatic analysis of legal documents.”
The Superlegal Al was trained on tens of thousands of contracts, using custom-built machine-learning and deep learning technology.
This level of technology for analyzing legal documents has only been possible with advances in computing over the last ten years.
The Al has been trained to detect issues on more than a dozen different legal contracts, ranging from software agreements to services agreements to purchase orders.
AI training techniques
The following techniques were applied when training the AI:
- Unsupervised learning was used for teaching the Al engine the core legalese language.
- Supervised learning, using deep learning multi-layer LSTM and convolution technology, was then used to train the system for fine-tuned issue-spotting.
- A unique augmentation algorithm was applied to boost the AI’s learning from human-annotated documents.
- Given that NLP and other off the-shelf solutions wouldn’t cut it, new models had to be created for the task of AI legal review: New Legal Language Processing (LLP) and Legal Language Understanding (LLU).
Legalese
Teams of lawyers and engineers taught the Al legalese by exposing it to a wide range of legal documents.
Training an Al engine is similar to training a new lawyer — exposure to different examples is crucial in developing a deep understanding of the legal practice.
Once the Al learned legalese, legal trainers pointed out the concepts it is required to recognize.
The LLP technology then allows the algorithm to identify these concepts – even if they were worded in ways never seen before.
Concepts not keywords
Monitoring concepts, not keywords — the Al operates in a far more sophisticated manner than a blunt “keyword search.”
Keyword searches can be over and under-inclusive, as words may be absent from relevant documents, or present in irrelevant documents.
True Al recognizes a concept however it is phrased or wherever it appears in a document.
AI vs Lawyers – the Study
It’s time to get into the landmark study.
In the AI vs lawyers study, twenty lawyers reviewed five NDAs alongside the AI.
US lawyers with decades of experience in corporate law and contract review were pitted against the AI algorithm to spot issues in five Non-Disclosure Agreements (NDAs).
Why NDAs were chosen
NDAs are routine business contracts used to protect information provided by one business to another.
These confidentiality agreements set out the terms of how that information handed over can and cannot be used.
NDAs represent the most commonly used contracts in business dealings, so their review and approval is crucial.
However, NDAs take companies a week or longer to approve, a process that frustrates other teams and slows down deals.
Click here to find out more about NDAs – including the top 3 NDA negotiation killers according to Superlegal’s data!
Setting up the study
The study was conducted with input from academics, data scientists, and legal and machine-learning experts, and was overseen by an independent consultant lawyer.
A team of prestigious law professors and veteran lawyers established and reviewed a list of 30 proposed legal issues that might appear in a standard NDA.
These legal issues then formed the basis of those used to test the issue-spotting accuracy of the participant lawyers and the Al.
This test replicated a real-world scenario in which a new contract is uploaded for the first time to the platform by one of our customers.
The lawyers
The lawyers chosen for this study were thoroughly vetted to ensure participation of only highly-experienced and US-trained lawyers with significant commercial experience in this exact form of NDA review.
The agreements
The agreements for this lawyers vs AI challenge chosen were five publicly available NDA agreements from the Enron Data Set, (the industry standard for such documents).
These documents had never been processed by the AI algorithm before.
Test instructions
The lawyers participating in the study were given four hours to identify and highlight the relevant issues where they appeared in all five of the NDAs.
The lawyers were asked to:
- Set aside what they knew about specific issues and instead stick to the definitions provided.
- Use a simple drop-down menu to identify the correct issues and where they appeared in each of the five contracts.
In the post-interview questionnaire, 100% of participants said the test was clear, credible, and precisely modeled on the way they currently review NDAs.
How was this study different
While this study was not the first to pit Al against humans in the field of the law, it is certainly the most evenly-matched scenario ever undertaken.
The Al vs Lawyers challenge was unlike any previous study in the way that the study ensured:
1) significant expertise of the lawyers in the exact area of law it was assessing
2) that all participants carried out precisely the same task.
AI vs Lawyers – the Results
When it comes to lawyers vs AI – it wouldn’t be a huge surprise if we told you that the AI was faster, but would you believe us if we told you that the AI was also more accurate?
Results:
- On average, the lawyers took 92 minutes to finish reviewing the contracts compared to the AI taking 26 seconds.
- The AI finished the test with an average accuracy rating of 94%, while the lawyers achieved an average of 85%.
On average, it took 92 minutes for the lawyers to complete all five NDAs. The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time to complete the task by a lawyer was 51 minutes.
In contrast, the Al engine completed the task of issue-spotting in a matter of seconds. Speeding up the initial contract review from an average of 92 minutes to 26 seconds.
It’s time to talk about accuracy.
Few would be surprised that Artificial Intelligence works faster than lawyers on certain non-core legal tasks.
However, the general consensus is that machines cannot match human intellect for accuracy in fundamental legal work. Given that lawyers are trained rigorously, with meticulous research skills based on a deep study of case law.
But the Al engine achieved 100% accuracy in one of the contracts. The highest individual score for a lawyer on a single contract was 97%.
How can AI be more accurate than lawyers?!
As we talked about at the start of the article, it comes down to the way the Superlegal AI was trained:
- Custom-built machine-learning and deep learning technology
- Exposure to tens of thousands of contracts
- Creation of new language models
- By teams of the industry’s top lawyers and engineers
This is confirmed by Professor Erika Buell, Director of the Program in Law & Entrepreneurship at Duke Law, who believes that
“Use of Al should provide consistency and predictability in a client’s contracts, providing better client protection.”
You may be interested in
Legal Technology Trends in 2024: AI, Automation, and Beyond
Conclusion
So the jury is in – AI is better than lawyers for routine contract review. And by better, we mean faster and more accurate!
As a matter of fact, according to Professor Gillian Hadfield:
“This experiment actually understates the gain from Al.
The lawyers who reviewed these documents were fully focused on the task: it didn’t sink to the bottom of a to-do list or get rushed through while waiting for a plane.
The margin of efficiency may be even greater than the results presented here.”
Not to mention that AI is infinitely cheaper!
Hua Wang, a lawyer participant in the study commented:
“Law firms would charge thousands of dollars for such an assignment!”
Final Comment
So it’s established. Legal AI is better, and legal AI is the future.
In the words of John O. McGinnis, Professor of Constitutional Law at Northwestern University School of Law, and Professor Russell G Pearce at Fordham University School of Law:
“Intelligent machines will become better, both in terms of performance and cost. And unlike humans, they can work ceaselessly around the clock, without sleep or caffeine.
Such continuous technological acceleration in computational power is the difference between previous technological improvements in legal services and those driven by machine intelligence.”
Superlegal
Four years after the groundbreaking AI vs lawyers study, Superlegal still harnesses the power of this revolutionary AI and it’s only getting better, faster and stronger!
Plus – our contract reviews are verified by top legal professionals to guarantee speed, accuracy and cost-efficiency – so you can get the best of both worlds!
With Superlegal you’ll get your contracts back in just 24 hours.
Click here to try Superlegal for free!
About the Author
By entering your email, you agree to our Terms & Conditions and Privacy Policy.