ESI Bytes

This episode puts Jeremy Pickens, Maura Grossman, and Robert Keeling in the same room to dig into a new study comparing generative AI review against continuous active learning, better known as TAR 2.0. Robert walks through the numbers his team at Redgrave published: 88% recall for generative AI against 64% for CAL, along with the precision tradeoffs that come with it. Maura pushes back on the framing and reworks the comparison herself, arguing the study raises more questions than it answers once you account for what happens when documents still need human eyes before they go out the door. Jeremy layers in his own read of the data and lands on a simple truth that keeps showing up throughout the conversation: proper sampling matters more than which tool you pick. We also get into whether prompts count as protected work product, how much the skill of the person writing the prompt actually shapes the results, and what it would take for the industry to adopt a real evaluation standard. And yes, rescue dog number four makes his debut.
Key Takeaways:• The headline numbers, 88% recall for generative AI versus 64% for TAR, don't tell the whole story on their own. Once Maura standardized the methods to compare apples to apples, generative AI performed competitively rather than as a clear winner, and the right tool depended heavily on whether the goal was producing without human review or reviewing every document before it went out the door.
• Sampling methodology is the real foundation everything else rests on. Jeremy made the point that if you sample correctly, the tool itself almost stops mattering. Robert's study did the sampling right, but both Jeremy and Maura agreed that's still far from the industry norm.
• Prompt writing is turning into its own skill set. Robert's expert prompt writer, who is also a software engineer, produced results a typical generalist attorney couldn't match. Maura connected this directly to her research on whether sophisticated prompts start crossing into protected work product territory.
• The reviewer's underlying job shapes the outcome more than people expect. Jeremy's small study found that an attorney doing internal investigations optimized for recall, while an attorney focused on document production optimized for precision, even without anyone planning the comparison that way.
• Robert flagged a real limitation in his own data: the recall calculation for generative AI didn't include the human review or privilege review hours that a full production workflow requires, which means actual cost comparisons need more than the raw numbers in the study.
Chapter Markers:00:00:00 - Introduction and guest panel: Jeremy Pickens, Maura Grossman, and Robert Keeling
00:03:48 - Robert introduces the generative AI review study and Redgrave's methodology
00:08:56 - The monkeys and cows analogy: why sampling correctly matters more than the tool
00:12:37 - The headline numbers: 88% recall for generative AI versus 64% for TAR
00:15:17 - Maura challenges the framing and reworks the comparison
00:19:14 - Standardizing the methods: apples-to-apples results
00:22:09 - Jeremy breaks down the cost tradeoffs of senior review versus contract review teams
00:28:08 - Robert responds and explains why his clients often skip human review at scale
00:32:46 - Does the prompt writer matter? The role of expertise in generative AI results
00:35:28 - Jeremy's study on how an attorney's day job shapes their prompting outcomes
00:39:35 - Are AI prompts protected work product?
00:45:31 - Sharing prompts with opposing counsel and the meet and confer process
00:48:12 - Closing thoughts: what still needs to be studied
00:52:09 - Lucky the rescue dog and closing remarks
generative AI review, TAR 2.0, continuous active learning, technology assisted review, eDiscovery, document review, recall and precision, predictive coding, generative AI prompts, prompt engineering, work product privilege, legal technology, AI in litigation, contract attorney review, sampling methodology, review workflow, machine learning classifier, Redgrave study, Jeremy Pickens, Maura Grossman, Robert Keeling, ESI Bytes podcast, legal AI research, discovery cost efficiency, production review, complex litigation review, AI validation methods

This episode puts Jeremy Pickens, Maura Grossman, and Robert Keeling in the same room to dig into a new study comparing generative AI review against continuous active learning, better known as TAR 2.0. Robert walks through the numbers his team at Redgrave published: 88% recall for generative AI against 64% for CAL, along with the precision tradeoffs that come with it. Maura pushes back on the framing and reworks the comparison herself, arguing the study raises more questions than it answers once you account for what happens when documents still need human eyes before they go out the door. Jeremy layers in his own read of the data and lands on a simple truth that keeps showing up throughout the conversation: proper sampling matters more than which tool you pick. We also get into whether prompts count as protected work product, how much the skill of the person writing the prompt actually shapes the results, and what it would take for the industry to adopt a real evaluation standard. And yes, rescue dog number four makes his debut.
Key Takeaways:• The headline numbers, 88% recall for generative AI versus 64% for TAR, don't tell the whole story on their own. Once Maura standardized the methods to compare apples to apples, generative AI performed competitively rather than as a clear winner, and the right tool depended heavily on whether the goal was producing without human review or reviewing every document before it went out the door.
• Sampling methodology is the real foundation everything else rests on. Jeremy made the point that if you sample correctly, the tool itself almost stops mattering. Robert's study did the sampling right, but both Jeremy and Maura agreed that's still far from the industry norm.
• Prompt writing is turning into its own skill set. Robert's expert prompt writer, who is also a software engineer, produced results a typical generalist attorney couldn't match. Maura connected this directly to her research on whether sophisticated prompts start crossing into protected work product territory.
• The reviewer's underlying job shapes the outcome more than people expect. Jeremy's small study found that an attorney doing internal investigations optimized for recall, while an attorney focused on document production optimized for precision, even without anyone planning the comparison that way.
• Robert flagged a real limitation in his own data: the recall calculation for generative AI didn't include the human review or privilege review hours that a full production workflow requires, which means actual cost comparisons need more than the raw numbers in the study.
Chapter Markers:00:00:00 - Introduction and guest panel: Jeremy Pickens, Maura Grossman, and Robert Keeling
00:03:48 - Robert introduces the generative AI review study and Redgrave's methodology
00:08:56 - The monkeys and cows analogy: why sampling correctly matters more than the tool
00:12:37 - The headline numbers: 88% recall for generative AI versus 64% for TAR
00:15:17 - Maura challenges the framing and reworks the comparison
00:19:14 - Standardizing the methods: apples-to-apples results
00:22:09 - Jeremy breaks down the cost tradeoffs of senior review versus contract review teams
00:28:08 - Robert responds and explains why his clients often skip human review at scale
00:32:46 - Does the prompt writer matter? The role of expertise in generative AI results
00:35:28 - Jeremy's study on how an attorney's day job shapes their prompting outcomes
00:39:35 - Are AI prompts protected work product?
00:45:31 - Sharing prompts with opposing counsel and the meet and confer process
00:48:12 - Closing thoughts: what still needs to be studied
00:52:09 - Lucky the rescue dog and closing remarks
generative AI review, TAR 2.0, continuous active learning, technology assisted review, eDiscovery, document review, recall and precision, predictive coding, generative AI prompts, prompt engineering, work product privilege, legal technology, AI in litigation, contract attorney review, sampling methodology, review workflow, machine learning classifier, Redgrave study, Jeremy Pickens, Maura Grossman, Robert Keeling, ESI Bytes podcast, legal AI research, discovery cost efficiency, production review, complex litigation review, AI validation methods

In this show Karl Schieneman, President and Founder of TestMyWitness has a conversation with Dr. Herb Roitblatt well known AI Data Scientist at Reliath AI and Legal Pioneer and Dr Stuart Hanscomb, Senior Lecturer at the University of Glasgow and author of Critical Thinking: The Basics on the limitations and challenges AI can present to lawyers and law students. We all recognize the potential benefits AI offers the legal community, but there is a downside we need to recognize and take steps to defend against. It is a fascinating conversation on an important topic for all lawyers.

This is a fun show as Karl Schieneman, President and Founder of TestMyWitness, interviews his good friend Tim Opsitnick, the new Executive Director of the influential Sedona Conference, which is a 3 decades old global think tank of lawyers and technologists who get together to move the law forward in a just way in areas where there is an intersection of law and technology. In addition to helping The Sedona Conference when it was formed, Tim brings many decades of experience as both a lawyer, businessman, and an entrepreneur. One of the best examples is his role in forming ESIBytes and supporting my efforts when the show first launched. Together, we brainstorm about what The Sedona Conference can look like going forward, as well as what ESIBytes 2.0 will look like. Maybe there can be a way to connect and help each other? We will explore this and other ideas.

In this show, Karl Schieneman, President and Founder of TestMyWitness has a conversation with Dr. Herb Roitblatt well known AI Data Scientist at Reliath AI and Legal Pioneer, and Dr Stuart Hanscomb, Senior Lecturer at the University of Glasgow and author of Critical Thinking: The Basics on the limitations and challenges AI can present to lawyers and law students. We all recognize the potential benefits AI offers the legal community, but there is a downside we need to recognize and take steps to defend against. It is a fascinating conversation on an important topic for all lawyers.

This is a fun show as Karl Schieneman, President and Founder of TestMyWitness, interviews his good friend Tim Opsitnick, the new Executive Director of the influential Sedona Conference, which is a 3 decades old global think tank of lawyers and technologists who get together to move the law forward in a just way in areas where there is an intersection of law and technology. In addition to helping The Sedona Conference when it was formed, Tim brings many decades of experience as both a lawyer, businessman, and an entrepreneur. One of the best examples is his role in forming ESIBytes and supporting my efforts when the show first launched. Together, we brainstorm about what The Sedona Conference can look like going forward, as well as what ESIBytes 2.0 will look like. Maybe there can be a way to connect and help each other? We will explore this and other ideas.