AI implementation 03: AI in practice – a review case study

We've explored some of the challenges of AI implementation, from navigating the different perspectives on AI in your organisation to managing the many moving parts. In the last article in the series, we examine how these are all brought together in practice.

The challenge: One week. Thirty-four thousand documents to review. A Court-ordered further discovery deadline

Our client, a party involved in proceedings in the Queensland Supreme Court, faced a complex discovery process. Over 16 months, several tranches of discovery had been completed, totalling over 4,500 documents. However, when our client’s opponents were granted leave to amend their pleadings, new issues were introduced, necessitating further discovery.

With tight Court-ordered deadlines, a re-review of the document corpus was required within a matter of weeks. Traditional search and analytics techniques identified 34,000 documents for review, which needed to be assessed against 10 discrete issues. A traditional approach involves a number of steps including multi-staged human review and a number other quality control steps. The timeline was tight: first-line review had to be completed in just five working days to allow for quality control, second-line review, and production.

What about AI?

We've already outlined five key considerations when deploying AI solutions. For this matter, we evaluated the suitability of an AI-augmented review workflow within RelativityOne. Here’s how we addressed the critical factors:

1. Security – RelativityOne was vetted by our security team during procurement. It has a range of security related certifications and it forms part of our own ISO27001 security certification, incorporates multi-factor authentication, and ensures data sovereignty, with no data leaving Australia. The AI solution itself operates as a "closed model", retaining no client data. In short, we have a range of measures to ensure robust security of client data.

2. Quality in, quality out – The AI solution was purpose-built for document review in the context of legal proceedings, offering features such as pre- and post-prompt coding specific to the use case, grounding of review results by directly citing document excerpts, and enabling validation during second line human lawyer review through directly plugging in to the review interface. To further ensure accuracy, we conducted multiple rounds of prompt validation, comparing AI results against human review to confirm alignment with review instructions and acceptable thresholds.

3. Quality control – Rigorous quality control was embedded in the approach. Documents flagged as relevant and borderline by AI underwent a second line human review before production, while non-relevant documents were validated during pre-review prompt testing and post-review analysis. As part of this, second line reviewers were able to see the AI review rationale directly in the document viewer to assist their consideration. Any documents the AI could not process were reviewed manually to ensure completeness. This was coupled with our standard pre-production exception checks to ensure compliance with the agreed exchange protocol, and ensure consistency in terms of relevance decision making, privilege review and redactions.

4. Tracking is key – Comprehensive tracking was central to the workflow. Specifically:

• Pre-review prompt validation results were documented in a central spreadsheet, comparing AI outcomes with human review and noting acceptable thresholds for discrepancies where review outcomes did not align.

• All AI and human review was recorded in real time in the Relativity workspace, bolstered by clear audit trails of decisions and actions taken during the review directly connected to the related documents.

• Key decisions and outputs were also tracked in CaseTrace, Clayton Utz's custom application for holistic matter tracking and data driven insights within Relativity.

5. Cost – A cost-benefit analysis compared the AI-augmented workflow with a traditional review approach. In this case, the AI augmented approach reduced costs to one-quarter of the traditional method and completed the review in one-quarter of the time, inclusive of prompt preparation and quality control processes. When deployed, the AI reviewed 34,000 documents in under five hours at a rate of 6,800 documents per hour, compared to a human review rate of 60 documents per hour.

The successful review and disclosure

The implementation of the AI-augmented workflow allowed our clients to meet their supplementary disclosure obligations within a much shorter timeframe and at a much lower overall cost than would have been the case with a traditional human only review process. With carefully drafted prompts and quality control processes, the AI review results were accurate and consistent, resulting in less time spent at second line review when compared against a traditional second line review of multiple human reviewers conducting first line review.

RelativityOne's AI review functionality also allowed for a parallel review to be undertaken to locate and collate key documents relevant to the newly introduced issues in the proceeding. This represented a further truncation of search and review work which would be traditionally undertaken after or separate to the supplementary disclosure review and at additional cost. 

By leveraging AI, we met the tight deadlines without compromising quality or security. This approach not only delivered significant cost and time savings but also ensured a robust and defensible review process.