The Black Box on Trial: Algorithmic Decisions, Due Process, and the Right to Contest

Friday, October 16, 2026

Overview

When forensic software helps convict someone, its developers have repeatedly invoked trade secrecy to keep defendants and their lawyers from learning how the technology works. In this panel from the 2026 BTLJ–BCLT Fall Symposium, trade secrecy and due process scholar Rebecca Wexler (Columbia Law School), AI regulatory lawyer Sean Quinn (Cooley), and AI auditing researcher Inioluwa Deborah Raji (UC Berkeley; Leadership Conference on Civil and Human Rights) join moderator Colleen Chien of the Berkeley Center for Law & Technology to break down trade secret claims against criminal defendants, third-party AI audits, and algorithmic notice requirements. Together they examine whether protective orders, audits, and notice can make the right to contest an algorithmic decision real for defense counsel, regulators, and the companies deploying these tools.

Instructor(s):
Sean Quinn, Partner, Cooley
Rebecca Wexler, Alfred W. Bressler Professor of Law, Columbia Law School
Inioluwa Deborah Raji, Academic Fellow at the Leadership Conference on Civil and Human Rights, UC Berkeley
Colleen Chien, William G. Simon Professor of Law, Co-Director, BCLT, UC Berkeley Law

5 Key Questions Answered

1. Why does Rebecca Wexler argue that trade secret claims should not let forensic-technology developers withhold how their tools work from criminal defendants?

Wexler explains that developers of forensic technologies have refused to tell defendants how their tools work, claiming intellectual property, even when the tools’ outputs are used to convict. She argues trade secret law protects commercial know-how from business competitors, not from defense attorneys vindicating due process, and that narrow criminal discovery plus protective orders already safeguard sensitive information.

“There have been repeated situations where developers of forensic technologies that are used to generate or analyze evidence against criminal defendants have refused to tell those accused persons or their attorneys details about how the technology works, even when outputs of the technology are used in the substantive guilt phase to convict the person of a crime. The rationale from the developers, they say, well, that’s my intellectual property, but let’s think a little bit about what kind of intellectual property that claim is. So, trade secret is a form of IP that’s designed to protect commercial know-how from business competitors, but the people who are trying to get access to the information in these circumstances aren’t business competitors; they’re attorneys trying to vindicate the due process rights of their clients.”

— Rebecca Wexler

“So, why would we recognize an intellectual property right in that context [defense counsel seeking access]? My position is that we should not, for a variety of reasons. One, the criminal legal system already has robust and, I think, sufficient protections for sensitive information, including commercially valuable information, because criminal subpoenas are very narrow. Criminal discovery is very narrow, and once information is ordered to be disclosed to an attorney, it can be disclosed under a protective order… And finally, I will say that recognizing a secrecy right in that kind of a context mistakes due process as a business competition.”

— Rebecca Wexler

2. Why does Sean Quinn argue that seals and protective orders don’t adequately protect companies in criminal cases, and how does Wexler respond?

Quinn says that, unlike civil fights between well-resourced companies, a leak of source code by a public defender or academic expert leaves the company with no real recourse, and the disclosure would repeat for every defendant the tool touches. Wexler counters that defense counsel face disciplinary sanctions and criminal contempt for violating a protective order — more than ordinary NDAs provide — while Quinn replies that deterrence does not compensate a company that has lost its “crown jewels.”

“Generally, when it’s two large companies fighting over some sort of IP issue, each of them has resources and can be enforced against if those secrets are disclosed. In the criminal context, when a company discloses its source code, which is the entire value of the company, to a public defender or to an expert who’s an academic, there’s no enforcement against those individuals or those entities. If they actually disclose those secrets, the company’s just out of luck. That information is out there. The cat’s out of the bag. The value of their company has just plummeted, because now everybody can recreate their product. So that’s a practical concern. Again, setting aside kind of the balancing of the rights, and that kind of gets to why the seal and protective order is, at least as the system is constructed right now, is really not sufficient.”

— Sean Quinn  

“I’ll make another tiny quibble with what Sean said. I don’t think you’re quite right that there is no remedy, or an insufficient remedy, if there were to be a leak. Now, yes, a public defender might be judgment-proof in that they have shallow pockets, but they’re subject to professional disciplinary sanctions and possible criminal charges for contempt of court if they violate a protective order that’s a court order. So to have criminal punishment on the line for a misappropriation is more protection than you usually get with the nondisclosure orders that companies often disclose under to swaths of employees, etc.”

— Rebecca Wexler  

 3. Can ex ante audits replace contesting algorithmic evidence in individual criminal cases?

Wexler calls auditing a “yes-and”: ex post litigation alone puts too much verification burden on under-resourced litigants, but individualized scrutiny remains necessary because as-applied errors — like a lab analyst’s swapped samples exposed on cross-examination — would never surface in an ex ante audit. She rejects total non-disclosure and points to secure-review precedents, such as classified material viewed in a SCIF, with the courts supplying the security measures.

“So, my response to the auditing is that it’s ‘yes and.’ Certainly, there are limitations to ex post litigation as a safeguard for accuracy… in part because it puts a burden on often under-resourced individual litigants to do a lot of the verification work, and that can’t be sufficient. At the same time, there is a necessity to do ex post individualized scrutiny, in part because things can go wrong in an individual case that you would never get from an ex ante audit. So this is just the foundational validation perspective versus the as-applied. You can have errors as applied, and, you know, we’ve seen that in the forensic context in individual cases. There was one example where somebody on the witness stand during cross-examination recognized, because of the cross-examination, and disclosed that this forensic lab analyst—that she had mistaken the crime scene sample with the defendant’s biological sample, swapped the samples. So that’s just something that an [ex ante] audit isn’t going to catch.”

— Rebecca Wexler  

“I think that what Deb laid out as an additional kind of safeguards, especially the security of the data environment, is really important. And I think that we could achieve that in individual criminal defense cases. We have precedents. There are examples for classified information or contraband possession like CSAM [child sexual abuse material], where individual counsel actually will go to an FBI and view material in a protected SCIF [Sensitive Compartmented Information Facility]. So the concern that I have is the solution of total nondisclosure. That just seems unreasonable to me. But putting a system into place that can, you know, address reasonable concerns about the security of information—let’s do it. Let’s figure out how to do it. And I think a government entity that is at play here are the courts. You know, they have some of these security measures available, so we should use them.”— Rebecca Wexler   

 4. What went wrong with the third-party audits under New York City’s Local Law 144, and what is “audit washing”?

Deb Raji says Local Law 144 anchored hiring-tool audits to the EEOC’s four-fifths rule of thumb, and the resulting audits — often by for-profit startups with little guidance on auditor conduct — proved performative rather than giving applicants evidence for recourse. She ties this to the NTIA’s warning about “audit washing,” where audits become a stamp of approval with no legal or accountability impact.

“But effectively, you know, once this law [Local Law 144] came into effect, everyone thought the third-party audits were going to collect the evidence required for any aggrieved kind of applicant to be able to have the avenue to get recourse or to fight legally a particular hiring result that they thought was unfair, if they felt like they were discriminated against by these systems. And it just wasn’t the case. The audits ended up being very performative, and in many cases, the third-party auditors, especially the for-profit startups—again, we’re operating in a world in which there’s very little guidance around auditor conduct—so we would have situations of, you know, providing consulting services, very Enron-like, you know, providing consulting services before coming in and then doing the formal audit that then gets reported to the city.”— Deb Raji   

“NTIA [National Telecommunications and Information Administration] also had this interesting report where they talked about this risk of audit washing… which is there’s this risk of, you know, having the audits or the third-party audits operate as…  a stamp of approval, without it actually playing any, or having any, legal impact or any accountability impact in terms of the behavior of these companies. And so I think that that’s something that we saw with Local Law 144, and something that we’re seeing in a couple of other cases as well. And it’s a real risk here when people don’t do the work of aligning sort of legal requirements or regulatory requirements with, you know, auditor requirements or sort of audit conduct.”— Deb Raji 

5. What does notice that an automated system is being used actually accomplish?

Quinn argues notice’s underappreciated value is internal: it forces companies to map which of their systems make automated decisions about individuals, though notice is probably of little value to users without a chance to opt out. Raji, acknowledging notice fatigue, says notice enables contestation and could let complaints accumulate as evidence, much like post-market surveillance for drugs and devices.

“On notice, from my perspective, I think it performs a function that people maybe don’t really think of as a function of notice in this context. But for companies, at least, and for lawyers dealing with this, it requires companies to actually map what an automated decision is to systems that they actually have, and actually take account of and understand which systems of theirs are making automated decisions about individuals. …[1 sentence omitted]… But a notice requirement at least requires the company to take stock of which of their systems are doing that, and then obviously provide notice to the user. Whether or not that notice is very valuable to the user, I think, is a separate question, and it’s probably not. And notice without an opportunity to then opt out does very little.”— Sean Quinn 

“I do think it [notice] does play interesting roles, though. One being that opportunity for contestation. If I’m notified that an AI system is being used, I can then, you know, file a complaint or participate in some accountability process. Another one in which we’ve been exploring with SB 53 through this incident monitoring, also at organizational levels, has been this idea of accumulation of evidence through notice. So, you notify someone that an AI system is being used as part of some consequential process, they have the opportunity to file a complaint or, you know, notify if they’ve experienced some kind of adverse event. And then, in the same way that we have sort of monitoring for post-market surveillance monitoring for things like vaccines or pharmaceuticals or medical devices, we could maybe have, you know, post-market surveillance monitoring for some of these alerts or complaints coming around AI systems and deployment.”— Deb Raji  

 

Summary generated from the recorded interview/transcript and reviewed by BCLT staff.

Keywords

trade secrets and due process • forensic technology source code disclosure • protective orders • graymail • zero-knowledge proofs • risk-limiting audits • Digital Services Act Article 40 • Local Law 144 • audit washing • SB 53 • can trade secrets block criminal defendants from examining algorithms • third-party AI audit data access

B-CLE Recordings (CLE: Free) | Youtube Recordings (Non-CLE) | Agenda | Resources | Speaker Bio(s) & Contact Info | Summaries

Download the transcript here!

Interview/Transcript

This interview/transcript was based on a conversation from the 7th Annual BTLJ-BCLT Fall Race & Technology Symposium: Being Human in the Age of AI about The Black Box on Trial: Algorithmic Decisions, Due Process, and the Right to Contest, on October 16, 2026, hosted by Berkeley Center for Law & Technology, UC Berkeley School of Law. The conversation was presented by trade secrecy and due process scholar Rebecca Wexler (Columbia Law School), AI regulatory lawyer Sean Quinn (Cooley), AI auditing researcher Inioluwa Deborah Raji (UC Berkeley; Leadership Conference on Civil and Human Rights), and moderator Colleen Chien of the Berkeley Center for Law & Technology.

Jude Abu Hijleh  0:25

Thank you, Panel Two, for that wonderful presentation. Hi, everyone. My name is Jude Abu Hijleh, and I’m a member of the BTLJ [Berkeley Technology Law Journal] Symposium team. Welcome to Panel Three: The Black Box on Trial, Algorithmic Decisions, Due Process, and the Right to Contest. As algorithms take over decisions that used to guarantee a hearing, they raise difficult questions about how the law should protect people’s right to understand and challenge the outcome. Today’s panel will explore several related but distinct concerns, from due process and anti-discrimination law to trade secret barriers and discovery and algorithmic systems. At the center of the discussion is a fundamental question: When the decision maker is a black box, what does the right to contest actually require, and who’s positioned to deliver it — courts, auditors, or legislators? Moderating today’s conversation is Colleen Chien, professor of law at UC Berkeley School of Law and faculty co-director of the Berkeley Center for Law and Technology. She is joined by Rebecca Wexler, Alfred W. Breyer Professor of Law at Columbia Law School, whose research focuses on trade secrecy and due process in criminal justice technology; Sean Quinn, partner at Cooley and Adjunct Professor at Georgetown University Law Center, who advises technology companies on AI regulatory strategy and defense; and Deb Raji, a PhD researcher at UC Berkeley and academic fellow at the Leadership Conference on Civil and Human Rights. Her work includes landmark audits of racial and gender bias in commercial facial recognition systems.

 

Colleen Chien  2:02

Thank you for those introductions. It’s my pleasure to be moderating today’s panel, and I’m going to turn first to Professor Wexler and ask her to just kick off today’s discussion with a brief background around what due process is and how it’s operating in this space. Rebecca, turning over to you.

 

Rebecca Wexler  2:16

Well, thank you so much for having me. I’m thrilled you’re doing this, and I’m thrilled to be a part of it. So one of the big questions that comes up whenever we’re handling AI and trade secret claims in the context of a rights-based system, as for example the criminal legal system, is whether the trade secret claims might interfere with some of the due process protections. So let me give you an example. There have been repeated situations where developers of forensic technologies that are used to generate or analyze evidence against criminal defendants have refused to tell those accused persons or their attorneys details about how the technology works, even when outputs of the technology are used in the substantive guilt phase to convict the person of a crime. The rationale from the developers, they say, well, that’s my intellectual property, but let’s think a little bit about what kind of intellectual property that claim is. So, trade secret is a form of IP that’s designed to protect commercial know-how from business competitors, but the people who are trying to get access to the information in these circumstances aren’t business competitors; they’re attorneys trying to vindicate the due process rights of their clients. So, why would we recognize an intellectual property right in that context? My position is that we should not, for a variety of reasons. One, the criminal legal system already has robust and, I think, sufficient protections for sensitive information, including commercially valuable information, because criminal subpoenas are very narrow. Criminal discovery is very narrow, and once information is ordered to be disclosed to an attorney, it can be disclosed under a protective order. Giving additional protection to withhold information when there’s no explicit or intended threat of misappropriation — right, what attorneys say they want to do is scrutinize, not use in a competitive business — overprotects intellectual property from even the perspective of substantive trade secret law, because you can’t generally get an injunction from somebody who doesn’t threaten to misappropriate your technology. And finally, I I will say that recognizing a secrecy right in that kind of a context mistakes due process as a business competition. I’m eager to hear what the rest of the panelists think.

 

Colleen Chien  5:18

All right. Well, that’s a great introduction, and I want to turn over to Sean because Rebecca very subversively in her framing remarks just presented a, you know, very principled challenge to the idea that I think a lot of companies might start with as a position to when they’re asked to reveal information about how an algorithm works. I’ve gotten — you know, my client has been accused and, you know, convicted based on, in part, based on its algorithm, and we just want to understand like what went into it, and, you know, could there be any mistakes in what you did? You would do that with any sort of witness, and it’s just very hard to do that when you’re hitting this wall. So what is your response? Is that a fair understanding of what’s going on, or can you actually tell us more from a company perspective, what’s at stake? And particularly, I guess, from my perspective as a complete novice in this area, I don’t understand why protective orders and seals and other things can’t be, you know, providing enough cover for companies to just provide this information, because it doesn’t seem adverse to their interest to just tell this person, tell the the victor, the person who’s been scored, what the basis for that is.

 

Sean Quinn  6:25

Yeah, yeah, absolutely. And and first of all, thanks very much for having me here. I I hear and take and frankly agree with Professor Wexler’s sort of principled position. And my my opposition is going to be more practical, and I wouldn’t even necessarily call it an opposition necessarily, but just to kind of give the company’s perspectives on what the risks from a, you know, a business perspective are, and set aside whether or not the balancing of those risks and defendants’ rights are is the right balancing as it’s set right now. So the I mean the first point I’ll make is that for a company who is faced with a request to disclose what is a trade secret, or what they treat as a trade trade secret, they have no option but to oppose that request, because part of the elements of maintaining a trade secret is that you essentially take every opportunity to defend that secret. And so failure to oppose that sort of request in the criminal context would threaten sort of the future ability to protect that trade secret.

 

Colleen Chien  7:29

That’s interesting because I guess I would have thought that it would still be considered — maybe is this like not a litigated question? — but that to disclose something under seal, you’re saying that that would break the trade secret or that information?

 

Sean Quinn  7:41

No, once the I mean once the the the company is then ordered to disclose it under seal, then they will have —

 

Colleen Chien  7:48

Oh, I see.

 

Sean Quinn  7:48

They can demonstrate that they took every reasonable measure to prevent the disclosure of the trade secret. So I’m just saying at the first at the outset, it’s really not a company’s choice whether or not to oppose that sort of request in the criminal context. So that’s that’s step one. Step two, I think, or argument two to think about, I think, are the the practical implications and what sort of protections the criminal system provide companies who do disclose these sort of things. So, one, I agree that sort of the import of this civil protection into the criminal system doesn’t make a lot of sense, and and one of the reasons because the companies disclose this information aren’t going to receive the same protections they would in a criminal context as they would in a civil context. For example, generally when it’s two large companies fighting over some sort of IP issue, each of them have resources and can be enforced against if those secrets are disclosed. In the criminal context, when a company discloses its source code, which is the entire value of the company, to a public defender or to an expert who’s an academic, there’s no enforcement against those individuals or those entities. If they actually disclose those secrets, the company’s just out of luck. That information is out there. The cat’s out of the bag. The value of their company has just plummeted, because now everybody can recreate their their product. So that’s a that’s a practical concern. Again, setting aside kind of the balancing of of the rights, and that kind of gets to why the the seal and protective order is, at least as the the system is constructed right now, is really not sufficient.

Because the other thing to think about here is scale. We’re not really talking about disclosure to one defendant. It’s disclosure to every single defendant who is subject to the use of that tool in a criminal process, and so that can be hundreds of sets of attorneys, hundreds of sets of experts across the country, and the practical.

 

Colleen Chien  9:50

Why? Why is that the case? I mean, like in principle, if they ask, like, yeah, you gave it up to that that court, you should give it here. But why is that? Is that a reality, or is that just like the possibility that it could go to every defendant?

 

Sean Quinn  10:02

No, no, that’s what I mean. That’s a reality. If If every defendant gets the opportunity to examine the the the code or the —

 

Colleen Chien  10:09

Oh, the brain motions. You’re saying all the all the public defenders are going to —

 

Sean Quinn  10:12

Every defendant across the country going to have that right, and and you are repeatedly disclosing essentially the crown jewels of your company to defendants and every partner with that case over and over. I mean, what I will say is, I absolutely — sorry, one other point is, I think an important question is what is what is the request? What are you actually asking for the company to disclose, and and what’s the utility of that disclosure? So most companies are going to oppose disclosing like the the the source code of whatever their algorithmic tool is, because that is the value of their company. That is a question of, like, you know, do does the the defendant and the their counsel have the the resources to make use of that source code, or what what they actually want, is it some sort of validation process, some sort of testing of, like, a kind of agreed-upon testing framework and results from that, that might be useful. So I I think you also need to like get to the nuance of does is the source code useful, because that’s what companies really want to protect, and is there a reasonable alternative to provide the sort of validation or the opportunity to challenge challenge the use of the tool that doesn’t involve disclosing the source code.

 

Colleen Chien  11:41

That’s great. I’m going to ask Rebecca and Deb to both respond, especially Deb. I think that you might have like an engineering solution to this, honestly. But before that, I want to put us in the position of being sympathetic to the companies, and in the way as professors — like, if somebody asks me, just give me the exam questions from the exam and tell me exactly how you calculated my grade? Like, I don’t want to do that, because I want to reuse that stuff, and I also don’t want to — I like I might have made a mistake. I worry, right? I mean, just being completely honest about a lot of times why professors are very, you know, kind of — they’re a little protective of their exams that they reuse or that they reuse questions on, or, you know, to reveal their entire process for how they how they exactly gave that person a participation score of eight and that person one of four or something, right? So, but what I’ve seen in Berkeley is Berkeley is very demanding place. Like The students are very demanding, and so what I’ve seen is that professors have gotten better at explaining their logic. They’ve actually explained, this is why I gave you this grade, and I’m going to actually take attendance every single time, and I’m going to like actually — they do things to actually make it more like — to make it the possible — make the transparency possible. So instead of accepting that there’s going to be no transparency, like I’m just going to use professor with privilege. Like there’s there’s a There’s a discussion. There’s just kind of why ANSI exists, because we want to have these interchanges to actually get to a solution. And maybe it is that we don’t reveal the source code because nobody really cares about that. They want to be able to audit it. So I want to turn to Rebecca first, and then I want Deb to to talk, because I think that we, you know, we have on this panel people who really could solve this problem potentially.

 

Rebecca Wexler  13:15

Sure. Yeah. Absolutely. Well, so I I will acknowledge. I just want to start out by acknowledging that there is this concern beyond scale, there’s a concern about what we call gray mail, where there’s a strategic value in trying to target sensitive information, kind of up the risk of continuing a prosecution, because if you could scare away the prosecutor from pursuing the prosecution or scare away the vendors from working with the prosecutors, that might be advantageous for an individual client. So that’s not a valuable, socially valuable part of this of this, but it is a valid dynamic. My my sort of rebuttal argument there from a policy perspective is to say gray mail is a concern with all kinds of sensitive information in the criminal legal system. So you can have gray mail strategy for classified information. You can have gray mail strategy for a complaining witness’s medical records, and so the system has safeguards built in to try to protect against that, and one of those safeguards is something Sean mentioned. What’s the relevance or value of the information? Rebecca,

 

Colleen Chien  14:30

Rebecca, can you just, for context, explain what graymail is? Because I think I’m understanding, but I’m not totally understanding it. I just want to —

 

Rebecca Wexler  14:35

Yeah, yeah. Well, it’s a little bit like, you know, the the the the word, I think, is supposed to evoke the concept of blackmail, but you’re trying — you know, but the graymail. So it’s not exactly blackmail, where you’re you’re threatening to disclose something if somebody doesn’t do something and it’s a criminal act. But it’s similar, because you’re trying to increase the risk to the opposing party. Could happen in civil litigation, criminal litigation, you know, in any context. Here, you’re trying to increase the risks to the opposing party of pursuing litigation by saying, well, if you continue to pursue this litigation, you’re going to have to put some sensitive information out there, and and that sensitive information will increase the cost to you of pursuing this prosecution. But that is a risk of a kind of abuse of the subpoena power with all sorts of information, not merely commercially sensitive information. So it’s a real risk, but it’s a risk that I believe the criminal legal system has established safeguards to account for. One of those safeguards is making sure that information is not discoverable unless it has proven relevance to the individual case. So, if developers were to actually say you can’t and shouldn’t be able to have this information because it is irrelevant, I would find that to be a very powerful argument for secrecy. Irrelevant information — I just taught relevance in my opening evidence law class last week — is not admissible in court. There’s nothing you can do with it. It’s irrelevant. You can’t use it. What I think concerns me is that actually developers don’t make the argument that it’s irrelevant, because it’s often not, and, you know, instead what they often say is this isn’t the most valuable information for you to have, or there’s an alternative, which we think should suffice. The tricky thing about that argument is, we don’t want to give third parties that have their own interests control over the litigation strategy of defense counsel. So, in our adversarial legal system, it’s the counsel who gets to choose what method they want to use to prove a case. I just taught another evidence case last week where there was a fight about whether you could stipulate your way out of your opponent’s chosen proof. You say, “I don’t want you to introduce some information I think will be prejudicial against my client. I’ll just concede the point. So you can’t introduce it. And the Supreme Court says we’re not going to let you do that, because it’s the attorney’s choice how to prove their case. So my main argument here has been, if the attorney can establish relevance, that should be the baseline. They should get it, and these other safeguards of the protective order, the sealing orders, should come into play. I’ll make another tiny quibble with what Sean said. I don’t think you’re quite right that there is no remedy, or an insufficient remedy, if there were to be a leak. Now, yes, a public defender might be judgment proof in that they have shallow pockets, but they’re subject to professional disciplinary sanctions and possible criminal charges for contempt of court if they violate a protective order that’s a court order. So to have criminal punishment on the line for a misappropriation is more protection than you usually get with the non-disclosure orders that companies often disclose under to swaths of employees, etc.

So a couple of other provocations there. I also have a possible technical fix. I’m eager to hear Deb’s technical fix, but I’ll just seed the conversation and say I worked with some computer scientists and suggested zero knowledge proofs as a way that could potentially solve the problem for both sides.

 

Sean Quinn  19:01

Just to respond —

 

Colleen Chien  19:03

But can you take a second to explain what a zero knowledge proof is for the audience?

 

Rebecca Wexler  19:07

Well, I would. I I want to make sure to get — and the other one to hog the the the the thing. And Deb may have a better definition of a zero knowledge proof. But basically, zero knowledge proof is a cryptographic, mathematically secure way of proving that the content of certain kind of information or possession of a capacity, without disclosing the actual information or capacity. So you could imagine a system where criminal defense counsel could query a developer’s — maybe a third party possesses the code, and there could be some kind of wrapper, and then a query could go, and they could get back a mathematically verifiable answer about, say, did your technology work this way or that way, without having to trust a human to certify that, and yet they wouldn’t have to view the actual source code.

 

Sean Quinn  20:04

I was just going to say quickly on the on the point of the value of protective orders and seals and whether they’re enforceable. I told — I agree and very much appreciate the deterrent effects of the criminal or professional sanctions against lawyers. But to the to the company who has lost essentially the entire value, there’s no there’s no compensation for that that’s likely going to be possible. So there’s there there’s deterrent, but there’s that’s not going to do a lot for the company who then has lost, you know, their the the crown jewels that they’ve that they’ve disclosed in the litigation.

 

Colleen Chien  20:44

Okay, I want to turn over to Deb.

 

Deborah Raji  20:47

Yeah, I was like, I, I, I don’t know if I have a solution to everything that was discussed. I also — it’s interesting to hear, Colleen, your point of, like, you know, please be sympathetic to the companies. I was like, I feel like there’s a lot of different stakeholders that are very sympathetic to these companies, but I think one of the things I wanted to flag is — I think Sean’s actually raising — so for context, I’m coming at this from the perspective of sort of third-party auditing of like deployed AI AI systems and trying to, you know, collect statistical evidence that can then be used as part of some accountability process. And so the work that we’ve done — we’ve sort of worked with — we’re not necessarily playing the role of public defender, per se, or prosecutor in the kind of the traditional legal sense. We’re kind of like third party, like technical consultants, and we might be working with like an enforcement agency at the state or federal level, or you might be working with, or you might be working with a public defender’s office, for example. But you’re not playing that direct legal role, and I think that does lead to different dynamics. One being that I think Sean does actually raise a good point around one of the concerns that we often deal with in terms of access is around competence to hold certain data requests. So, you know, to Rebecca’s point, you know, knowing the source code is not actually necessarily useful information, and there is an argument that can be made about the relevance of certain disclosures. But even after you decide that, like, oh, actually, no, like, this is the type of data that we need to do this investigation, there’s often a lot of concerns raised on the vendor, the vendor side, or the platform side, being like, do you have the capacity? Like, at Facebook, there is, you know, people working full time to set up, you know, privacy and security measures to make sure that the data is safe. If we give you this data, do you have the sort of like technical infrastructure capacity to hold that data? And I think, you know, to Sean, I think you were kind of alluding to, like, if we were to make the claim that we did have that capacity, we took that data, and that data was released, like, who is liable, and, you know, where does that put the platform or the the vendor? And so I just wanted to raise that. You know, beyond the sort of graymail of sort of malicious release of the data, there’s also just like a straight-up capacity question of those that want to play this third-party accountability role. Sometimes even when it’s a government, if we’re working with a government entity, sometimes they don’t even necessarily have that capacity in house to hold that data, and so that has always been a very serious tension point with respect to data access. I think one of the things that maybe Colleen was alluding to — I’m not sure — but I was very involved in the Digital Services Act [DSA] in the European Union. So that was kind of like this bill to sort of enable third-party accountability of online platforms. So think of like very large online platforms like Facebook and Twitter, and they have this Article 40 that is about data access. And a lot of us that have been doing this type of work have, you know, did our best to sort of shape the the character of this Article 40, and a lot of these tensions came up with Article 40 in California. Like The conversation has also kind of arisen around if we — you know, and even like this week, there’s been a lot of discussion about like AI auditing for, you know, not just like traditional online platforms, but also, you know, ChatGPT. Like, does that count as a very large online platform? These like large language models. If we want people to sort of do this third-party auditing of these systems, you know, how much access are they entitled to? I think a lot of the tensions that rose arose there might be also worth addressing or discussing. One being relevance. I think methodologically, the approach that people take is just give me as much data as possible.

 

Deborah Raji  24:44

But again, we have this capacity constraint where we actually don’t have the capacity always to handle, you know, the full scale of knowing every single thing about the platform. So what’s the middle ground? One thing that me and some colleagues have actually been working on is this idea of risk-limiting audits and applying that to DSA enforcement. So the way that that works statistically — this is kind of like the the maybe more like technical-oriented solution of for something like the DSA, and also kind of similarly inspired audit regimes in certain states — the way that what the the function of the audit is really as a verification mechanism for claims being made about the system by the vendor or by the platform. So, you know, Facebook might say like, oh, we removed X percent of posts based off of X percent of criteria according to, you know, DSA standards, and the role of the sort of third-party auditor is just to verify that claim: like is that true or is that not true? And so when that’s the kind of setup for the audit, there’s actually a really interesting opportunity where, in like traditional election spaces, there’s this idea of a risk-limiting audit, which is like a statistical approach. It’s like an online hypothesis testing approach where you can look at a subset. The kind of inspiration is from traditional electoral audits. So if you say like Bob won the election over Alice, and that’s the claim being made about a particular election, there’s a just statistical approach where you can look at a — like if you, you know, do a recount on like manual paper ballots to verify that claim, you don’t have to look at every single ballot. You can actually look at a subset of the ballots in order to sort of statistically verify, within some specific determined error bound, that like oh, the the claim being made about this election is true or false. And so we’re trying to see if we can bring that same approach to claims being made in terms of compliance with DSA guidelines. So if a if a platform says, oh, we you know, we removed X number of posts based off of X criteria, then we can look at a specific number of samples of posts to verify that claim. We don’t have to look at every single post, and then that allows us to sort of validate the claim with, you know, as little data as as possible, or as little data as like as as as relevant a sample as possible, without kind of overexerting our privileges and like using more data than than what would be like statistically required to validate the claim. And so that kind of approach and other methods are things that I think folks are exploring to say, like, you know, how can we actually define relevance, and how can we make sure that we can validate claims being made about these systems? Where a lot of the onus in making a correct claim is still on the vendor, on the platform, but when we go through the validation process, we still require some amount of data access, but we’re very deliberate and almost narrow in our requests, so that, one, it’s an amount of data that we can actually handle from like a security, privacy, compliance point of view. But then, two, we’re not kind of overexerting or seem to overexert our privileges as auditors. You know, in collaboration with, again, like some litigation team or or an enforcement agency or whoever. And so I think that that has been one approach that at least kind of the AI auditing community has has worked on in this sort of kind of audit compliance regime, where like the role of the auditor is really just to to validate an internal audit claim. I think another thing that has been discussed or or or talked a lot about in this space has been the government as a mediating entity. So like rather than giving direct access to sort of a third party, you know, what role could the government play as sort of this mediator for transparency or for access? You know, one example is like, you know, laws like SB [Senate Bill] 53 and other attempts in California. You know, very recent example is SB 288 13, I think, on the independent auditors.

 

Deborah Raji  29:01

But there’s been sort of this attempt to sort of write into legislation this idea of, okay, you know, could we have the vendors give — you know, make their claims to government, but also provide access to government, and then the government sort of vets the third parties that are given access through, you know, some kind of government mediation, and that kind of providing sort of sort of a gateway for for partial transparency or partial access, rather than giving it directly to this third party or requiring the vendor or the platform to give directly to the third party. And so, again, these are very kind of like like legal-enforcement-adjacent situations in which like the government is involved. I think for something like, you know, a lawsuit, or situations in which like the government does not want to be involved and it probably isn’t, it’s it’s it’s a it’s a more complicated conversation. But for sort of these kind of compliance audit cases, that’s been one path forward that some folks have been very excited about. Again, there’s still a capacity issue. I mean, California is probably one of the highest capacity states in terms of some of this stuff, and they’ve been really struggling even in terms of being able to host that data. But it’s a kind of promising avenue forward, and and so people are exploring a lot of these things in this sort of AI auditing space, because, yeah, the question of data access is definitely not a straightforward thing. I just wanted to also flag one minor additional issue, or just like two additional issues. So one is — another huge debate in the AI audit policy conversation has been the qualification of third-party auditors. I think for accounting, we have, you know, PCAOB [Public Company Accounting Oversight Board], I guess, that the public company kind of accounting oversight board, you know, GAAP [Generally Accepted Accounting Principles] principles, things like that. For For medical devices, there’s like MDSAP, which is like the Medical Device Single Audit Project, and things like that, where you’re effectively anointing certain organizations to be qualified audit organizations. And in the AI space, we do not have anything like that at all. And so, any organization can kind of claim to have this expertise, claim to have the capacity to be able to, you know, take advantage of certain third-party privileges. And so, yeah, we’ve had, you know, different degrees of challenges with that in the AI audit space, and it would be interesting to hear thoughts from this panel about that, or just what qualifies someone to gain access, especially if they’re not playing that, you know, direct public defender role. And then I think another thing I was going to flag as well was another issue we’ve had in the AI audit space, just from like a pragmatic level, has been corporate compliance. So like when, under DSA, the Digital Services Act, you know, when you make a data access request, you make it to a particular very large online platform, let’s say, you know, X or or Facebook, and the company will often respond to the request. Of all the instances that I’ve participated in or observed with other people, when the company responds to the request, they’ll often scope their response. So, you know, you might request like, hey, can I see all the political ads from Arkansas in the last X months or something, and then they’ll they’ll scope it to be like, oh, actually, can we only provide it from these particular counties or these particular dates? And so there’s been a lot of — and the same thing happened with Local Law 144 bias audits for AI employment algorithms. And so a lot of people in the audit space are really worried that when you when data access is kind of willfully constrained on the corporate side, there’s ways in which the data might be manipulated, or you might be presented with like a biased data sample when you make these requests, and there’s no way to verify what the true data sample looks like, and, you know, should the government intervene at that level? And so there’s conversations there about just like data integrity and data bias when you make these requests. Are you even getting information that’s reliable when you make these access requests?

 

Colleen Chien  33:10

This is great. I wanted to turn it over to Sean to ask about how practically auditing looks like from the client side and what it is. You know, are clients using these tools as like compliance measures and kind of check the box, or do you feel like they’re really using the auditing process to shape their product design and features, and really try to ensure that they are abiding by whatever sort of best practices are out there? So you can tell a little more, especially for students who might be interested in working in this space in the future. I mean, do law firms do this, or is it accounting firms, for people who want to be AI lawyers and, you know, want to see auditing as the thing that everybody does? Like One thing I I get in my AI law governance classes — I’m teaching a lot of sort of cases that are old older cases, but then law that’s applied to new facts. But a lot of times they’re saying, well, we just want to know how a corporate counsel deals with these issues, because they’re having to approve products all day long, and what does that audit look like? And there don’t seem to be there’s not a lot there. So how do they sort of bring the force of whatever normative values that they have to the job? So can you tell us a little more about that, and how students who want to do this — like what paths they might pursue? Is it transactional work, or is it something else in AI practice?

 

Sean Quinn  34:22

Yeah, yeah. I think there are — I mean, an audit team is is not in it. There are a bunch of different roles that you might play in an audit role, and so in an audit team. So, from the perspective of lawyers on the team, corporate counsel, or else generally, we are not doing the audit. We might be shaping the scope of the audit, describe like understanding some of the — or help advising the client on some of the ramifications of the audit. And then it also really depends on like what kind of audit you’re talking about. So, if you’re if you’re doing a self audit, then that is probably going to be attorney directed, so that you can apply available privileges to the outcome of that audit, and that’s going to be, you know, a company’s first preference. Obviously, that’s not always going to — I see Deb laughing, or everybody laughing — but so you’d rather, you know, write your own test and and and then apply it. That’s not always going to — the purpose of the audit, right? So there’s lots of other reasons you might do an audit for which a self audit is not going to suffice. So and this is where other types of lawyers might come in. So transactional lawyers might be involved in the sorts of audits that are useful in procurement processes. So, if you need ISO [International Organization for Standardization] to confirm that you are compliant with a particular standard, then that’s going to have to be a third-party audit, and it needs to be a third-party audit because it provides some sign of credibility and of validity of the outcome of the audit. And then, obviously, you’re not going to be able to do a self audit if it’s a regulatory requirement, or one like Deb was talking about under the DSA, where you might have either a regulator or civil society or an academic who has the right to audit certain data or algorithms. So, lots of different sort of — you know, not all audits are the same. The purpose of the audit is not the same, and and the role that a lawyer plays on that audit may also differ based on on the type of audit.

 

Colleen Chien  36:32

Rebecca, I want to turn to you, because I think one thing that struck me about Deb’s, I thought, quite elegant summing of what like sort of an engineering mindset or sort of problem-solving mindset might bring to this is that, you know, you can have, as worked on, three interventions, right? You have You have a, you know, adequate infrastructure — you know, nobody can really argue with that. You have risk-limiting audits, which sort of provide governance that this thing actually works — it’s, you know, kind of the claims they make are actually something we can verify. And then government as a mediating entity. But to kind of invoke Margot Kaminski’s sort of binary governance idea, that you have systemic protections and then you have individual rights: litigation is very contested, and that’s what makes a better outcome. Like, Do you think auditing — like, how do you see auditing go hand in hand with the contested number — nature of litigation — to ensure that this is like adequate? Do you think that public defenders would be satisfied, and maybe reluctant a little bit because they’re giving up that sort of thing that they can use to, you know, say, well, you know, give us the algorithm — we don’t want to do that, okay, let’s settle, or something? Like, Are they going to be satisfied with — or are they going to never satisfied with — the audit results that say, no, our claims are accurate, and we did the best we could, and this is like not perfect, but this is what it is? Sort of going back to Loomis and, like, well, here’s all the warnings of what we’ve done. So I guess I would like try to ask you about the like how we still use the contestation of litigation in in harmony with some of these audits. And I think the same question is like also, sometimes people want to fight in in a way, right? Like You could come up with this really great set of things that Deb has mentioned. How do we get that done? Like How do we actually — usually, often it’s litigation, right? It’s like a consent decree, or it’s it’s it’s laws being passed, but then with enforcement behind it. Like, How do we actually get uptake of these like things that would actually, in theory, resolve a problem? So I just want to turn that with you.

 

Rebecca Wexler  38:33

Yeah, thanks. Well, I I’m already learning so much from the other panelists, and it’s, you know, jotting down notes for kind of — I’d love to shift to a problem-solving perspective. So, my response to the auditing is that it’s yes-and. Certainly there are limitations to ex post litigation as a safeguard for accuracy, or accuracy — we’ll just, let’s just stick with that — in part because it puts a burden on often under-resourced individual litigants to to do a lot of the verification work, and that that can’t be sufficient. At the same time, there is a necessity to do ex post individualized scrutiny, in part because things can go wrong in an individual case that you would never get from an ex ante audit. So this is just the foundational validation perspective versus the as-applied. You can have errors as applied, and and you know, we’ve seen that in the forensic context in individual cases. There was one example where somebody on the witness stand during cross-examination recognized, because of the cross-examination, and disclosed that this forensic lab analyst — that she had mistaken the crime scene sample with the defendant’s biological sample, swapped the samples. So that’s just something that an ex ante audit isn’t going to catch, and we do have examples where the where the ex post. I think that what Deb laid out as additional — as kind of safeguards, especially the security of the data environment, is really important. And I think that we could achieve that in individual criminal defense cases. We have precedents. There are examples for classified information or contraband possession like CSAM [child sexual abuse material], where individual counsel actually will go to an FBI office and view material in a protected SCIF [Sensitive Compartmented Information Facility]. So the the the concern that I have is the solution of total non-disclosure. That just seems unreasonable to me. But putting a system into place that can, you know, address reasonable concerns about the security of information — let’s do it. Let’s Let’s figure out how to do it. And I I think a government entity that that is at play here are are the courts. You know, they have some of these security measures available, so we should use them.

 

Colleen Chien  40:59

Deborah, Sean, anybody want to have the last word on this topic before we move on to notice?

 

Deborah Raji  41:05

 No, I was just going to say — I actually, yeah, I appreciate Rebecca bringing up the point of like there is some precedent with CSAM and other kind of sensitive material that’s partic like — that’s been part of these investigations in the past, that we could look to as like an example of when we do have sensitive data. You know, there’s it’s not like there’s no precedent. Like There is some. I did want to point to something Sean had mentioned, which is, I think, the kind of definitive factor in these cases seems to be scale. And so thinking about how to take some of these existing measures — I agree with you that non-disclosure is like a kind of unacceptable resolution, but I think there there does need to be like innovation in in the norms of how we historically have explored this. You know, for the platform, a lot of the platform audit work in response to DSA, but also just in general with some of what was happening at the state level here, I think I think people were surprised at the fact that it was it was different at scale. It was We’re operating at a completely different scale than something like CSAM or historical kind of security vulnerabilities. Like There is something different about this technology that does require innovation on the infrastructure point. And, yeah, like very excited about working on that, and also talking more about — I know, Rebecca, you’ve been thinking a little bit about that too. So be interesting to just explore what that is. But it, I do think the scale does cause new technological challenges that that are required to address separately. I think another thing I wanted to raise was what, like Colleen, you had alluded to in your question to Rebecca around like legal compatibility, where sometimes these audits happen and it’s not clear how it feeds into actual accountability for the vendors or for the organizational user. I think Local Law 144 was like a universally traumatic experience for everyone in the AI auditing world. So for context, Local Law 144 was, you know, municipal law in New York City around sort of required third-party audits for employers that were using AI-enabled technology as part of their hiring process, and me and a couple folks were, you know, very engaged in the process of, you know, drafting the bill and seeing it pass into law. And I think a lot of people — there were many fights between many different stakeholders in the establishment of this law, and what they landed on was sort of anchoring the audit results to this like four-fifths rule heuristic from EEOC [Equal Employment Opportunity Commission] as to what constitutes sort of discrimination in like a hiring context. Again, that’s like a rule of thumb from EEOC. It’s not like a legal requirement. It’s not, you know — there’s it’s not necessarily like even compatible with like most mainstream definitions and understandings of things like disparate impact. And so when this law came into effect, there’s a really kind of scathing review of enforcement of this law by the New York State Comptroller, which I encourage folks to read if they’re interested in that. But effectively, you know, once this law came into effect, everyone thought the third-party audits were going to like collect the evidence required for any aggrieved kind of applicant to be able to have the avenue to get recourse, or to to fight legally a particular hiring result that they they thought was unfair, if they felt like they were discriminated by these these systems. And it just wasn’t the case. Like The the audits ended up being very performative, and in many cases the third-party auditors, especially the for-profit startups — again, we’re operating in a world in which there’s very little guidance around auditor conduct, so we would have situations of, you know, providing consulting services, very Enron-like, you know, providing consulting services before coming in and then doing the formal audit that then gets reported to the city. And so there was all of these like things going on that yield i

 

Colleen Chien  45:09

Great. Well, we’re running short on time, but I will ask for kind of a go-around-the-room around the question of notice and the role of notice. So, for example, in New York, we saw the passage around — and I think the students can relate to this. If you have not realized it, the prices that you see online on your phone are set a lot of times based on information about you personally. So, I did this exercise in my class last week. I haven’t actually crunched the data yet, but we did it. Anybody who wants to sort of do this as demonstrative, you take out your phone. If we all put up — take out our phones right now, and we’re all in the same location and all looked for, you know.

 

Rebecca Wexler  46:50

Sorry, Colleen. I’m gonna interrupt you because I just realized Sean sent a message. I thought everybody saw, but it was a direct message to me saying he has to leave in 30 seconds. So, okay, let me get that.

 

Colleen Chien  47:00

Oh yes, thank you.

 

Sean Quinn  47:01

Yeah, sorry. I do have to drop at one. I can stay on for a couple more minutes, but I have to drop for another meeting. Didn’t realize I just sent that to Rebecca. Sorry.

 

Colleen Chien  47:07

Oh, okay, no problem. So yes, I think then we — then we can — let’s let’s go ahead, and —

 

Sean Quinn  47:12

I can answer that. Yeah.

 

Colleen Chien  47:13

Okay. Yeah. So I guess the question with with that I was getting towards was, what’s what do you think the role of notice is? And then also — gonna — so I’m gonna ask you guys, like you know, how does notice get involved in this, and how do consumers start to take a role? Because we have like, for example, New York: your price was set using an algorithm. Now what do we do with that? Because we’ve talked about enforcement on the corporate side — like companies need to do things, and then, you know, they they will do things, but whether they do them well is another question. But consumers don’t have any reason to be performative. They’re like, I don’t understand this. They They have notice, but where does that go? So, what’s the role of notice in this whole ecosystem, and of consumer sort of people power actually being part of this movement? And then the other question is, if you were going to put resources into one place that you thought was like the most practical, whether it’s like the auditing infrastructure or it’s enforcement or something else, that you would think would most move the ball forward, what would it be? So, I’m going to start with Sean, and I’ll just ask these same two questions of every person.

 

Sean Quinn  48:04

Yeah, I notice, from my perspective, I think it performs a function that people maybe don’t really think of as as a function of notice in this context. But for companies, at least, and for lawyers dealing with this, it’s it it requires companies to actually map what an automated decision is to systems that they actually have, and actually take account of and understand which systems of theirs are making automated decisions about individuals. I know it sounds odd to say that companies don’t always appreciate that, or like what what an automated decision is, or or which of their systems are doing that. But a notice requirement at least requires the company to take stock of which of their systems are doing that, and then obviously provide notice to the user. Whether or not that notice is very valuable to the user, I think is a is a separate question, and it’s probably not. And notice without an opportunity to then opt out is does very little. So, yeah, from my view, which is, you know, obviously a lawyer helping companies figure out these regulations, like the first step is companies figuring out what an automated decision is and when are they making it. To your second question about kind of where the the best place for resources is, going back to our original question as well, around kind of how do you how would you audit audit algorithms in the criminal context and avoid disclosure that could be harmful to a company? I think just better standardized frameworks for appropriate assessment of these systems that companies can do ex ante, and then with some ability to challenge those audits later. But an actually useful and valid framework that companies can rely on to provide real transparency and real validation of their systems is useful both to the companies and to the people that want to understand and challenge the application of those systems.

 

Colleen Chien  50:03

Great. I think of the motto, like clarity is kindness. There’s more clarity — we could probably do a better — yeah. Rebecca or Deb, and thank you so much, Sean, for joining. We really appreciate it.

 

Sean Quinn  50:12

Yeah, absolutely. Nice to meet you both, or all of you

 

Deborah Raji  50:18

Yeah, I can. I can kind of follow up on that. I think I think the role of notice can play two roles. So like, one is like, Local [Law] 144 did have a notice requirement of like every job applicant in which an AI system is being used gets notified of the fact that an AI system is being used. But like, as we’ve seen with sort of like, you know, cookie notifications and privacy policy, like people get overwhelmed by all these pop-ups, and they like stop understanding like how to respond and how the effect of them responding might affect the quality of service or their opportunity for employment and all these things. And so, I think there’s a lot of confusion as to like how to actually do that well, and I don’t think that’s been a solved issue at all. I I do think it does play interesting roles, though. One being that opportunity for contestation. If I’m notified that an AI system is being used, I can then, you know, file a complaint or participate in some accountability process. Another one in which we’ve been exploring with SB 53 through this like incident monitoring, also at organizational levels, has been this idea of like accumulation of evidence through notice. So like, you notify someone that an AI system is being used as part of some consequential process, they have the opportunity to file a complaint or, you know, notify if they’ve experienced some kind of adverse event. And then in the same way that we have sort of monitoring for like like post-market surveillance monitoring for things like vaccines or pharmaceuticals or medical devices, like we could maybe have, you know, post-market surveillance monitoring for some of these alerts or complaints coming around AI systems and deployment. And so, I think that that’s been like an interesting policy opportunity. And in California, there’s been a lot of enthusiasm for that — just like notice as an opportunity to then collect feedback, kind of like aggregated, crowdsourced feedback on what’s going on with the system, and then that feedback then operating as some kind of evidence or some kind of information that that can then inform, you know, further action.

 

Colleen Chien  52:16

Great. And then you had like just one place where you would — I mean, I want to actually bring in your role at the the Leadership Conference on Civil and Human Rights. So if you were going to put your, you know, your energy just towards one thing — I know you’re working a lot of different projects — where do you think is like the highest value? I guess if we’re going to suspend not just your effort but like more in the ecosystem, like the three things you talked about, like where do you think the highest impact would be in terms of doing something, given. Yes.

 

Deborah Raji  52:44

Oh yeah. I I think a lot of it has been on the infrastructure point. So like building capacity for accountability processes through developing infrastructure. So even what I was talking about with notices, it’s like there’s a bunch of these things called harm discovery tools that help us collect information about people’s experiences with these systems. That’s been very helpful for civil society to then build narratives around what is practically happening on the ground, like who is actually at risk, and and what are those experiences that they’re having they’re having. And it’s been interesting, too, because a couple of these like civil society groups involved at the Leadership Conference, for example, have litigation capacity, like ACLU [American Civil Liberties Union] and NAACP [National Association for the Advancement of Colored People], and they’re they’re starting to think of like, okay, if we wanted to come up with some kind of like class action lawsuit, like this can also help us identify potential, you know, plaintiffs and things like that. And so that’s been interesting from just like an infrastructure level of like, okay, how do you build the database to collect that feedback? How do people get notified? And can civil society play a role in like raising awareness so that we can collect more stories of the types of harms people are experiencing? But other types of infrastructure are also really important. Transparency infrastructure, like I mentioned, to then like, once you make a data access request, to then receive that data and keep it safe, so that you’re qualified to hold that data. There’s also kind of, you know, performance analysis tools. Actually, I worked a lot with the Mozilla Foundation to fund a bunch of these like open source AI accountability tools and and support some of these kind of like infrastructure-type projects. But I think moving forward, a lot of my energy, coming more from the computer science side, of course, has been to just like figure out what these gaps are in terms of technical infrastructure, and then and then see if we can build these tools to sort of facilitate that auditing and accountability process and make it a more accessible process for a wider range of participants and stakeholders. So I think that’s been my focus for the last like year or so, and probably moving forward has been just kind of oriented around that. And I think from like a policy perspective, a lot of my focus has actually been on this sort of data access point. Trying to think through logistically, given the scale of these systems and the types of harms that people are already reporting, you know, what are what are the kind of privileges we can afford to give third-party auditors? How can we qualify and sort of set audit conduct standards so that we make sure that we have the right participant pool? How do we grow that third-party audit ecosystem, et cetera? Again, like it’s been especially relevant in California this like last couple weeks, but it’s been a big topic in the EU for for a couple years at this point. And so there’s a couple of those things that I’ve been, you know, dealing with for a while now, but a lot of it has to — yeah. I think one thing that might actually be interesting, as just like a follow-up — I like I should just talk to Rebecca about this, but also just to bring —

 

Rebecca Wexler  55:50

It’s hard to talk.

 

Deborah Raji  55:52

Yeah, something that actually I’m curious to get Rebecca’s take on is there’s this idea of evidence-based policy that’s been floating around in like AI policy spaces — I’m sure you might have encountered it. And like SB 53, for example, was meant to be like a a law to to collect evidence around AI harms and and and support evidence-based policy and AI harms. That’s like the role of the incident databases. That’s the role of the transparency reports. DSA is also positioned as one of these types of things, especially now that people are looking at ChatGPT and things as a very large online platform, potentially, this year. So I think there’s interesting use of the language of evidence and like, you know, third-party auditors operating to collect, you know, independently derived evidence from these companies, et cetera, and that language has just been floating around a lot. And I’m just curious, as someone that actually studies legal evidence, like you know, are are they onto something? Is there Is there a role for this kind of like statistical evidence, or is it just like people are actually really not understanding what that word means? And like, there’s there’s opportunity for things to be a little bit more fleshed out and a little bit more nuanced.

 

Colleen Chien  57:08

Well, now turning over to Deb to take us both the panel. You have the two questions around what’s the purpose of notice and how do you see it sort of moving us forward, and also where do you see sort of the most important investments being made, whether it be in enforcement or some of the things Deb has mentioned, or shoring up public defenders’ ability to do X, Y, or Z. And feel free to respond to Deb’s question as well, if you’d like to.

 

Rebecca Wexler  57:30

Okay, great. Well, so when I was thinking about notice, I actually was thinking about an example that’s been powerful and informative to me personally, which has been Colleen, your work on the Paper Prisons Initiative. And so there, the idea is, you know, built into our due process, we have this concept of notice and an opportunity to be heard before the government deprives you of rights. But there are all these examples that your work has helped to surface about people who have an entitlement to a restoration of rights, but because it’s restoration rather than deprivation, the due process baseline doesn’t kick in, and they often don’t know they have that opportunity. So, and that’s —

 

Colleen Chien  58:12

Can you explain that to the the crowd. I’m so —

 

Rebecca Wexler  58:13

Yeah. So the example —

 

Colleen Chien  58:15

Maybe so other people understand, what is this context that you’re talking about?

 

Rebecca Wexler  58:19

Yeah. So the example is, say, for instance, you had a right to have a conviction purged from your record. Imagine, say, it was a minor marijuana possession conviction or something like this that has been retroactively eliminated, and yet that’s still part of your record, and it can have ongoing harmful consequences in your life by making it difficult for you to get a professional license or something like that. You might have an entitlement in law — these more recent laws that said, “Well, we’re going to enable everybody with those types of convictions to have them removed from their record, but you have to take some affirmative step to do it. And a lot of times, there’s what Colleen and her partners in this work have identified as a second chance gap, where people have the entitlement, but they don’t know it, or they don’t have the resources to to close the gap to actually, you know, go ahead and and get that charge cleared from their record, as an example. So I think notice in a case like that is incredibly important, and something that needs a different legal mechanism than just the baseline due process rights, because those aren’t doing the job. But at the same time, I think it’s drastically insufficient, because another thing that that work has done — and in part of, I think, call— so Colleen, you’re more — think of you a little bit more about being open to the kind of AI for good, like let’s figure out how these technologies can can help us — is that there’s also the opportunity to just automate that clearance process. And so why don’t we go forward with that, rather than — why don’t we do yes-and again: the notice, and let’s use the technology to automate automate the restoration of rights. So, I think notice —

 

Colleen Chien  1:00:00

New York has actually passed a clean slate law. California has them as well, and so it’s it’s exciting. And we’re at the point now where we’re actually now in the public defender trying to notify people, and there’s a role for AI in that. But but there are 13 states under the clean slate initiative that have closed the gap, at least in terms of doing the clearance. But now there’s a question of actually knowing: if you don’t know you have now a clear record, you you can’t really act as if you are free. So we have a digital Juneteenth initiative, sort of around people getting notified that they actually have like freedom from their records, because there’s no obligation to notify. So we’re trying to figure out how to compel, or, you know, come up with affordable ways for public defenders to get information out to the ecosystem.

 

Rebecca Wexler  1:00:42

So, the clearance is actually happening automatically?

 

Colleen Chien  1:00:45

New York is going to have millions of New Yorkers who will have a cleaner record, and California the the freeman is even deeper, so it’s it’s quite exciting. But people don’t know about it.

 

Rebecca Wexler  1:00:55

So it’s they don’t get the benefit of the effects. Yeah, so, so on the other two points — I wanted — Deb had said something earlier about scale and the scale of the risk from disclosures and risk of harm from disclosures. I just wanted to point out that the scale argument also goes in the other direction. So, when we have a single automated technology that’s being used in the investigation or prosecution of potentially hundreds of thousands of individuals, one minor flaw in that technology can affect a large number of persons, and so that, you know, that scale argument can also be marshaled in favor of more auditability, more transparency. On the information-forcing policies, use of the word evidence — I think, you know, great. We do need information. We do. We do need information forcing, and evidence has a, you know, kind of cultural connotation. I think of it a little bit of like cost-benefit analysis in OMB [Office of Management and Budget] or something like this. But the uses of the word — it has a tradition in history. It has a tradition in journalism. It has a tradition in the sciences. It has a tradition in law, and the, you know, nuances are kind of fascinatingly different in those different schemes. But the the the overall idea that access to information can be important for knowledge production and accurate and fair outcomes, I think, spans across all of those domains.

 

Colleen Chien  1:02:31

Wonderful. Well, thank you so much, both of you, for sharing your brilliant insight. I learned so much, and I hope that the collaboration and the ideas on this panel turn into projects and papers and something that happens in rural. We’ve got sort of the leading people on the panel. So thank you so much for participating. Thank you to the audience for listening.

 

Transcribed by https://otter.ai