AI for Public Defense

• Name the exact pain point. Identify the specific problem the office is trying to solve – for example,intake triage, transcription backlog, evidence review, research delay, or case administration. State what the tool may assist with and what decisions it must not make.
• Map your data. List every category of data the office may upload, including body-camera video, jail calls, police reports, informant audio, strategy notes, client personally identifiable information, and mental-health information. Mark which categories are confidential, privileged, work product, subject to protective orders, or otherwise restricted. This Data Map supports the office’s analysis of its duties under Rule 1.6 and applicable jurisdictional rules.
• Identify users and affected people. Name who will use the tool, which clients and cases may be affected, and who will review the output.
• Define permitted and prohibited uses. Decide what data may not enter the system during a demo, evaluation, or pilot, and what output may never be relied on without human review.
• Record the baseline. Measure the current time, cost, error, coverage, or backlog for the selected workflow before introducing the tool.
• Assign owners. Name an operational owner, an ethics/legal reviewer, an IT/security reviewer, and a procurement contact.

• Confirm competence. Under Rule 1.1 and applicable technology-competence guidance, the responsible attorney should understand how the specific tool works, what it does with client data, and its limits – or obtain qualified help from IT, security, privacy, or other knowledgeable reviewers. 
• Resolve CJIS requirements when applicable. If the proposed use will process Criminal Justice Information, determine with the appropriate CJIS authority which CJIS Security Policy requirements apply and document how the office and provider will meet their respective responsibilities.
• Require written confidentiality protections. Use an NDA and a confidentiality/data-protection addendum reviewed by appropriate counsel. Address ownership, permitted use, human access, subprocessors, audit logs, incident notice, legal demands, return/export, deletion, and termination.
• Confirm no default training or secondary use. Require the vendor to state in writing whether prompts, uploads, outputs, usage data, metadata, feedback, or derivatives are used to train or improve any internal or third-party system. Negotiate an enforceable prohibition unless the office expressly approves otherwise.
• Create a prohibited-input list. Write down what may never enter generative-AI prompts or unapproved tools – for example, client identifiers, privileged strategy, sealed material, protected health information, or other restricted data.
• Map the full data chain. Identify hosting locations, models, cloud providers, human reviewers, and every subprocessor; confirm access controls, encryption, auditability, change notice, and responsibility for subprocessors.
• Review conflicts and adversarial use. Ask whether the vendor serves police, prosecutors, corrections, courts, intelligence, or other potentially adverse users, and evaluate separation, incentives, cross-customer learning, and access to defense usage data. 
• Test incident and exit readiness. Review breach notice, forensic cooperation, subpoena or warrant notice, data export, deletion including backups, service continuity, and offboarding.

PRO TIP: Loop in IT/security and ethics counsel early. Run a tabletop breach scenario before the pilot: who pauses use, pulls logs, contacts the vendor, assesses affected clients, and authorizes notification?

Review the CLJC library of existing AI tools and complete the Legal AI Vendor Pre-Demo Scorecard (PDF) before selecting a product.

Questions to ask:

• Does the vendor train on my data by default? If yes, can I opt out – and is that opt-out enforceable for the exact product and plan, or is the use a dealbreaker?
• Where is my data stored and processed, and who can access it? If Criminal Justice Information is involved, what responsibilities apply under the relevant CJIS review?
• Will the vendor sign an NDA and a confidentiality/data-protection addendum?
• Do I understand how this tool works well enough to explain its role, limits, and data handling when communication to a client, court, or other person is required?
• Is there a documented human-verification step before any output influences advice, a filing, or a consequential decision?
• Can I export my records and obtain verified deletion if I stop using the tool? What happens to backups and derived or aggregated data?
• Which models, cloud services, human reviewers, and subprocessors touch the data, and will I receive notice before they charge?
• Does the vendor serve law-enforcement, prosecution, correctional, or other adverse users How are defense content, metadata, usage data, staff, and product learning separated?

Decision Record: 

Keep the Data Map, completed scorecard, vendor responses and supporting evidence, signed addendum, risk and mitigation memo, pilot approval, and review notes. Treat any summary rating as a decision aid-not a substitute for counsel, security, procurement, or leadership judgement.

SUGGESTED STARTING POINT – NOT A STANDARD: Choose one high-volume use case and consider limiting the pilot to 5-10 attorneys, 50-100 cases, and about 90 days. Adjust these figures to office size, risk, available test data, personnel requirements, and the seriousness of potential error.

• Choose one bounded use case. For example, test body-camera transcription rather than introducing several unrelated functions at once.
• Start safely. Use synthetic, public, or otherwise properly approved test material before live client data.
• Set scope and stop rules in writing. Define users, case and data types, dates, excluded uses, required review, and events that pause or end the pilot.
• Name owners for questions and emergencies. Designate one pilot contact and one person with explicit kill-switch authority to pause use immediately.
• Create a short cheat sheet. Include approved prompt templates when relevant, prohibited inputs, verification steps, known failure modes, and a way to flag errors or incidents.
• Hold a comprehensive kickoff. A two-hour session is a reasonable planning default for a complex tool; cover demonstrations, permitted and prohibited uses, confidentiality, limits, human verification, escalation, and hands-on practice.
• Use a defined evaluation set and failure log. Establish expected answers or review criteria in advance, then record corrections, false negatives, unsupported citations, unprocessed files, incidents, access problems, and verification time.

Select and define only the metrics that fit the use case. Establish the baseline before launch and document how each measure will be calculated

• Efficiency: average attorney or staff hours saved per case after including time spent reviewing and correcting the output.
• Quality: the percentage of reviewed outputs meeting a pre-defined material-error threshold on a representative test set. Define ‘material error’  in advance and require human review for critical content.
• Coverage and recall: relevant items found, false negatives, low-confidence or unprocessed material, and performance across relevant accents, languages, file types, and conditions
• Adoption: active users or log-ins per week, completion of required training, appropriate use, and attorney or staff satisfaction.
• Harm check: whether any unsupported citation, factual assertion, omission, confidentiality incident, or biased output reached a client communication, filing, hearing, or consequential decision. 
• Security and control: access-control tests, audit logs, policy or subprocessor changes, incidents, deletion or export tests, and response time.
• Case outcomes – optional context only: a measure such as motion-grant rate may be tracked, but should not be presented as caused by the tool without a valid evaluation design.

REVIEW CADENCE: As a suggested schedule, review at weeks 4 and 8 and decide by week 12 whether to scale, revise, extend, or stop. An urgent safety, confidentiality, or accuracy issue should trigger immediate review.