Transcript

Minderoo Centre for Technology and Democracy

Meredith Broussard, Confronting Race, Gender and Ability Bias in Tech

May 9 2023

Gina Neff:

I am delighted you are here. I'm Gina Neff and I'm the Executive Director of the Minderoo Centre for Technology and Democracy. Welcome to our latest event, Meredith Broussard, Confronting Race, Gender and Ability Bias in Tech. Now, before we get started, there's just a few housekeeping points here at Jesus College: in case of an alarm sounding, we ask you leave the room via the emergency exit. No test is planned for today.

The event will be recorded by Zoom. By attending the event, you are giving your consent to being included in the recording. If you would like captions for this evening's event, they're available through zoom. If you missed the link on our registration site on the desk in the back, our team are in the room and they're able to give you the link for your device.

The recording will be available on the Minderoo Centre for Technology & Democracy website shortly after the event. Afterwards, when we have our discussion, if you raise your hands, we'll bring a mic to you. We're using room audio for our recording. So we, we do ask that everyone is amplified with a microphone. A transcript will also be made available online after the event.

Now, I'm pleased to start. I am very pleased tonight to be joined by Meredith Broussard. Meredith is an associate professor at New York University. Her books include Artificial Unintelligence: How Computers Misunderstand the World, and she appears in the Emmy nominated documentary ‘Coded Bias’ on Netflix.

Her latest book, More than a Glitch: Confronting Race, Gender and Ability Bias in Tech, was released by MIT Press in March of this year. The word glitch implies an incidental error as easy to catch up as it is to identify, but what if racism, sexism, and ableism aren't just bugs, but they're mostly functional in machinery. What if they have been coded into the system itself? In this book, Meredith demonstrates how neutrality in tech is a myth and why algorithms need to be held accountable.

And I just want to take a minute to thank Meredith for joining. One of the things I truly admire about her work, and I recommend and commend her work to everyone, is she takes deep technical knowledge and translates it in a way that makes it incredibly accessible. She is such a rare combination of genius, genius and understanding of the inner workings of how our systems are being built and how they are designed. Genius too, and perhaps as importantly in telling the kinds of stories that people understand and resonate. That's what I think is so powerful about the book she's going to talk about tonight. How she makes concrete through peoples lived everyday experience some of these lofty principles that we talk about in AI ethics. She does the work of telling us exactly how they're impacting people's everyday lives.

So with no further ado, please join me in welcoming Meredith Broussard.

Meredith Broussard:

Hi everybody. Gina, thank you for that wonderful introduction. It is great to be here with all of you today at Jesus College. I am going to talk a little bit about some ideas from More than a Glitch. As Gina said, it came out about two months ago and what happens in the book is I offer a hype-free explanation of AI. What it is and isn't. I give some examples from justice, medicine, education and more about how discrimination works inside automated systems and I give some options for change. So the book is not entirely a bummer.  

I'd like to start with talking about what AI is and isn't, and that of course starts with the Terminator, because at some level many of us are thinking about the Terminator when we think about artificial intelligence, right. When you really examine. Like what is the image in your mind's eye. This is one of the images that features heavily. Also things from Star Wars, Star Trek, any of our favourite Hollywood examples. And these are all terrific, they are wonderfully creative. They inspire us to imagine other worlds, but it's really important to remember that that is imaginary AI. What is real about today's AI, today's artificial intelligence is it’s just math. It's very complicated, beautiful math. Another way of describing it is computational statistics on steroids, right? So there is no Terminator. There is no R2D2, although I’m a little sad about that, and there is no robot takeover. There is no existential risk that we need to be worried about right now. Instead, what we should focus on is not what's imaginary that AI, but we should focus on what's real and we should focus on the ways that AI is harming people. We should focus on present harms to real people instead of obsessing over imaginary problems in an imaginary future.

So what kinds of harms are we talking about? Well, there are a lot of them. And in order to predict the kinds of harms that people face with automated systems and AI systems, I think a really good frame is one that's offered by Ruha Benjamin in her book, Race After Technology. And that's the idea that automated systems discriminate by default, OK. So, for a very long time, there's been a perspective that has dominated our, you know, cultural discourse, which is something I call techno-chauvinism and techno-chauvinism is the idea that computational solutions are superior to others, that tech is objective. That technology is neutral or unbiased. And when you let go of that, when you let go of techno-chauvinism, it becomes easier to see how real world problems are embedded in our data traffic.

So let me back up for a sec and explain what is actually going on with the math today's AI systems. Alright, super easy explanation that you can take with you and use in those business meetings that you go into when people are like yeah we should we should definitely buy this new AI system. Yes, right. So what happens when you build an AI system is it's data-driven. You take a whole bunch of data, absolutely as much data as you can get your hands on, ideally for free. But you get as much data as you can get. And if you get it into the computer and you say computer: make me a model, and the computer says: OK. And it makes a model and the model shows the mathematical patterns in the data. That's it. That's AI. Right.

So then this model, this AI model you can use in order to make predictions in order to make decisions in order to generate new texts or new images or new whatever, right? That's generative AI. That's what you see. In things like Chat GPT or Bard or Dall-E, or Midjourney. All of the kind of buzzwords that people are talking about nowadays. All of this is data, either text or images or video or whatever. Take the data, feed it into the computer. Computer makes a model, models show mathematical patterns. Then you can use that model for new things, right? But let's think about the Data. Because the data that we're using to train AI systems is data about the real world and the real world has a bunch of problems.

Right, I'm a journalist. I'm a data journalist. I write a lot about the problems of the real world, and those problems include discrimination, racism, sexism, ableism, all kinds of structural inequality, and even if you don't think that these problems are in the data that you're using to train and add a system even if you think, well, it's just data, just scraped it from the web, how bad could it be? Well, the computer is so good at finding the mathematical patterns that we see over and over again that automated systems, AI systems, replicate the problems of the past.

I'm going to give you a couple of examples of this. It shows up in finance. Markup, which is a really terrific algorithmic accountability organisation in the United States did an investigation into mortgage approval algorithms, right? So when you are trying to buy a house you apply for a mortgage. This loan finances about 80% of your purchase usually and it turns out that the mortgage approval algorithms were denying borrowers of colour loans. Nationally loan applicants of colour were 40 to 80% more likely to be denied than their white counterparts, and in some metro areas this disparity was more than 250%. And a data scientist might look at this and say, hey, well, I don't understand what's going on. Uh, you know, it's just data. It's not racist, and the sociologist might look at this and say, oh, yeah, it's the data that's racist, right? So what's happening is: the system is being fed with data about who has been given loans in the past. Well, in the United States there's a very long history of financial discrimination against borrowers of colour. The US has a lot of residential segregation. There is also a practice or there was historically a practice called redlining. Where people would physically draw a red line around certain neighbourhoods and say all right people who are not white can live here and then white people are going to live in all the other places. So all of these things are part of, you know, part of the US's frankly shameful history. And these patterns are being picked up on by machine learning models by the AI models, right? So historical discrimination being repeated.

We also see it in policing. You are probably familiar with some of the problems in facial recognition technology. We know this thanks to a really fantastic paper and project by Joy Buolamwini, by Timnit Gebru, and their team. Called Gender Shades. The finding from Gender Shades and several other projects was that facial recognition is better and recognising men than women. It's better at recognising light skin than dark skin. And when we do an intersectional analysis, we discover that facial recognition is best of all at recognising men with light skin. It's worst of all at recognising women with dark skin and trans and non-binary folks tend to not be recognised by these symptoms at all. Now part of this is due to the training data. Because when you dig into it, when you look at the training data that was used to make most of the major facial recognition systems, which is one of the things that the team did in the project. Yes, they found that most of the training data was men with light skin. Right. And so data scientists might look at this and say well, you know the problem is the training data. What if we just put in more women? What if we just put in a greater range of skin tones? Would that make the facial recognition more accurate? And yes, that is absolutely true, that is a way that you can make the facial recognition systems more accurate. But we need to go a step further and think about justice. Because facial recognition systems, when they are used in policing, are disproportionately weaponised against communities of colour, against poor communities, against communities that are vulnerable and are already over policed and over surveilled. So a justice focused solution would be not to use facial recognition and policing at all.

Another way we see AI discrimination pop up is in medicine. This is a kidney and one of the one of the examples that I find particularly horrifying. It comes from a calculation that was used to calculate when people are eligible for a kidney transplant. OK, so you qualify for the kidney transplant list when your EGFR score is 28 or below. That means your kidneys have about 20% function and until two years ago white people and black people were calculated using different scores. So basically, if you were black, you got a multiplier in your EGFR score that was kind of hidden in the algorithm used to calculate your EGFR score and it meant that black patients qualified for the kidney transplant list later than everybody else. Black patients had to be sicker in order to qualify for kidney transplant. Now, this doesn't mean that you get a new kidney. This just means you'd be eligible to be on the meeting waiting list. Why is this? Well, there was a deeply racist assumption that black people's bodies work differently than other people's bodies, and we see this, we see this a lot, I'm sure if we have historians of science in the room, you're very familiar with the way that race, which is actually a social construct has been treated as a biological construct sometimes in history and a lot of problems have resulted from that. The same thing happens when data scientists or AI computer scientists making AI take the, take the patterns that they see in the medical world at face value and just unthinkingly implement them in automated systems. Right? Because there was this racist assumption about how black bodies work. And then, that got uncritically incorporated into this medical technology. Fortunately, due to activism by doctors, by patients, by and by interested folks, the kidney calculation in the US was changed two years ago and it resulted in lots and lots of people being eligible now for life saving medical technology.

We see AI discrimination come up in access to public benefits. You're seeing a pattern here, right? Access to public benefits. This is something that's actually not in the book. But you absolutely should check out anyway. It is a new algorithmic accountability investigation from Wired and Lighthouse Reports. It is about Rotterdam. The city of Rotterdam was using an algorithm in order to detect welfare fraud, and I say that with scare quotes, air quotes, because anytime that somebody is trying to detect fraud in public benefits or detect welfare fraud, what they're actually doing is they're trying to restrict access to public benefits. And so what they found was that this algorithm that was used in Rotterdam to try and detect welfare fraud was actually biassed based on gender, and based on ethnicity. Because mathematically the factor that was most predictive in the data about whether or not somebody would be a cheater on their benefits was a Dutch language ability, right? So anybody who was, say, a recent immigrant, who, you know, their command of Dutch was not there yet, they were being flagged as potential welfare cheats. Rotterdam to their enormous credit, realised that this was a problem and they said, all right, we're going to stop using this algorithm and this is in part why we have access to it. So the journalist at Lighthouse Reports and at Wired, managed to get access to the code, the data, the model and the documentation used to construct this algorithm. This is the first time that we've had access to all of these things, right.

So when you make an AI system, when you make a machine learning system, I told you before, you make the model, you use the training data and then people theoretically write down what happens in the documentation, and then if you know how to read code you can read the code and you can like find where the potential flaws are and people tend to create code like it's a big secret. It is not actually a big secret, it shouldn't be a big secret, but people do treat it like it's a big secret. And one of the things that algorithmic accountability reporters do is we go poking around in people's code. Traditionally, one of the functions of the press is to hold power accountable. In a world where algorithms are increasingly being used to make decisions on our behalf, that accountability function has to transfer to algorithms and they're creators. So we have algorithmic accountability reporters. Markup is my favourite example of this, but there is also some great stuff happening at the New York Times at ICIJ and at Lighthouse Reports.

And so what we can do with the tools of algorithmic accountability reporting is we can open up black boxes, we can investigate algorithms, we can evaluate them to find out if they are treating people fairly. Generally what you'll find is that they are not. Generally what you'll find is that algorithms or algorithmic systems are discriminating. Why? Well, training data, bias, problems of the real world reflected in our AI systems. Right. There are actually mathematical methods we can use to address this bias. It is not the end of the world to find bias, but. You have to be willing to just look for it, to admit that it's there, to admit there might be a problem with your automated system and techno-chauvinism prevents people from wanting to do that.

Let's talk a little bit more about techno-chauvinism and how it intersects with accessibility. Uh, because techno-chauvinism gets in the way of making the world truly more accessible. And I want to tell you a story about something that happened to a man named Richard Dan in Maryland. So Richard worked at an Apple Store in Maryland and he was doing great in his job at the Apple Store. Richard is deaf. He is fluent in ASL, American Sign Language and folks would travel for hours to get to his store to work with him because he could speak with them in ASL, he could speak with them in their own language. And this mattered a lot, especially when you're talking about technology, because it was really difficult to talk about technology in the first place. and talking about all the little fiddly things in a different language is really tough. Right. So Richard was a really great resource to the deaf community. And one of the things that Apple provided him with was an iPad with the notes app installed so that he could type back and forth with customers who are not deaf, which worked really well - until it didn't. Sometimes customers would refuse to work with them. Right.

And his manager said this was OK. His manager said ‘oh well, you know, people have a choice to, people have the right to choose who they want to work with’ and this did not, this does not feel right. The same manager later on started giving Richard a hard time. He said Richard, you're not keeping up in the team meetings. And Richard said, well, I am having a hard time keeping up the team meetings because it's a setup, kind of like we are having today, where there's one person in front of the room and then the rest of the team is down there and the accommodation that Richard had been offered was that one of his teammates was supposed to sit next to him and type what was happening in the meeting.

Now, having Scribe does work really well sometimes I've had scribes sit in on my classes and it works really well, but it does not work well when it is a coworker because the coworker needs to pay attention to the meeting too. The coworker also needs to learn, right? So the coworker is not necessarily doing a great job of transcribing and also listening and participating. Right. So and then they tried speech to text. Right, which is a technological accommodation, but the speech to text did not work that well, because if you've ever used speech to text, which I'm sure all of you have by now it does not get everything and also the acoustics are a problem. So if I am… if you're sitting in the back with a speaker or a microphone that's supposed to pick up on the speech at the front of the room, if there's no projection in the room, then the the microphone is going to pick up on the ambient noise as well as the speaker at the front of the room, and it's not going to work exactly the way that you expect. Right. And Richard explained this and said I really need a human interpreter because sometimes you just need a human interpreter. Technology for accessibility has given us- has worked beautifully. It does work beautifully. Until it doesn't. Right. When you hit. The outer limits of what the technology can do and you're still relying on that technology. Then it becomes marginalising. Sometimes we should use a human instead of a computer. It's not a big deal, right? Only techno-chauvinists are very excited about-  ‘no we need to use technology all the time’, and instead I think you should use the right tool for the task because sometimes that's a computer. Sometimes it's something simple like a book in the hands of a child sitting on a parent’s lap, right? It's not a competition.

So we need more nuance in the way that we talk about technology for accessibility. We also need to talk about the ways that new technology creates peril. Right. This is so one of the things that happens- is that designers of technology, everything has unconscious bias. We all have unconscious bias. We're all working on it every day and trying to become better people, but we are not yet perfect.

We can't see unconscious bias because it's unconscious, right? And unfortunately we embed our unconscious bias in our technological systems. When you have small and homogeneous groups of people creating technology, then the technology gets the collective unconscious bias of creators. And one way you see this is in a particular case of delivery robots in California. But I don't know if you've seen in this particular delivery robot before?

Oh you do. Alright? So you have a trial or these delivery robots here? Has anybody had a positive experience with them? Raise your hand.

Audience member:

I rescued one

Meredith Broussard:

Can we get a microphone? Can we hear this? Uh, because these robots, they often do things like fall over, right? Kind of like turtles, and they need to be rescued. Tell us was that it?

Audience member:

It just got stuck, it was stuck on the curb. But it was just rocking backwards and forwards and I just thought - you feel very protective for them. I don't know why he feels very animal-like. Anyway, so I rescued it, put it back on its little way and off it popped. It didn’t say thank you.

Meredith Broussard:

That does kind of stink. Yeah, it's an ungrateful robot. It's interesting to say it was stuck on the kerb. Because the story that I want to tell you, it has to do with kerb cuts. So kerb cuts are part of the kerb that dip dips down to the street, right. If you're in a wheelchair kerb cuts are really helpful. In the US kerb cuts were mandated as part of the Americans with Disabilities Act. I do not know the legal context over here. I would be curious to hear it.

Audience member:

It's called a dropped kerb.

Meredith Broussard:

It’s called a dropped kerb, OK? Fabulous. And was there a disability legislation that was - that made it happen? I don't mean to put you on the spot? I apologise.

Audience member:

It's clunky. Under the social model of disability, it should be. And we are signed up to the UN Convention on the Rights of Disabled. And our equality laws, the Equality Act 2010, there are some provisions that is based on a medical model of disability, which is a different catalogue official together. But there are, I mean, there are in our city, county councillors can be approached to help implement them where they lack, but it's a slow process. It's iterative I think.

Meredith Broussard:

All right. I mean as long as it's iterating toward like putting them in all over the place I'm for it. But OK, so dropped kerbs, they are here. And what happened with this particular dropped kerb is that the dropped curb is really useful for people in wheelchairs, but when they put them in, they realised, oh wait these are actually useful for everybody. And so in the US, we call it the curb cut effect here. I guess we called the dropped kerb effect. Dropped kerbs are useful if you are using a different kind of mobility aid. They're useful if you're wheeling a bicycle. If you're wheeling a stroller, if you are wheeling a dolly, they're enormously useful. So when you design for greater access when you design for disability, everybody benefits. And it turns out that robots also really like to use the drop kerbs, so this particular robot had been programmed to wait before crossing the street, and it was waiting in the dropped kerb. But something in a wheelchair was trying to cross the street and the robot was blocking their way. The robot was occupying the dropped kerb and there was oncoming traffic, which made an enormously dangerous situation for the person in a wheelchair. And why did this happen? Well, the designers of the robot were not thinking about sharing space with human beings, they were thinking about the needs of the robot and elevating those needs above the needs of the people who were using the space.

So what can we do? Maybe you're depressed now. I did promise you the book is not entirely a bummer. You ready? Number one thing you can do. Write this down. Buy my book.

Another thing we can do is we can emphasise AI reality. All this magical thinking around artificial intelligence and imaginary futures is so working against us. So we need to emphasise what is real and what is imaginary about AI. When you think that AI is mysterious, it makes you feel like you have no power against it. When you look at AI as a system that can be understood with broad strokes, then you're more empowered, and this especially matters when we're trying to make policy regulating AI.

We can also invest in public interest technology. This is a new field. If we have students in the room, public interest technology is a great place for getting a job in technology for social good. So we can build public technology in the public interest. And we can pay attention to responsible AI principles and responsible AI governance while we're building. And the book does have a number of primarily US based resources around this.

We can look for human problems inside our technology. We can expect that the human problems will occur. If you're technologists, you think, right, I'm not sure what the human problems are, that's OK you can collaborate with the social scientists and the humanists, the journalists, even, who know a lot about social problems. Nobody needs to be a Unicorn. Nobody needs to take this on all by themselves. Collaboration is key.

We can test technology for accessibility before releasing it. Right, testing for accessibility is too often something that happens after the fact. There's so many resources out there for learning more about accessibility, and we should take advantage of them.

And finally, algorithmic auditing, which I mentioned before, algorithmic auditing is the process of taking a look at our algorithms for your AI system and saying alright there’s probably an issue. I'm going to confront it. I'm going to have these hard conversations and these are hard. It is hard to have conversations at work about race. It is hard to have conversations at work about gender. It is hard to have conversations at work about disability, and it is even harder when you have those difficult conversations in the context of  a technological system that an organisation has invested thousands or hundreds or thousands or millions of dollars in procuring it, or building, right? Nobody wants to hear that their algorithm is racist, but unfortunately their algorithm is racist, and so we're going to have to normalise that. Is most likely racist, I should say.

How do you know what to look for? Well, you can learn from journalists who are on the algorithmic accountability beat. I mentioned the Markup, look at ProRepublic, look at ICIJ, and because I'm a professor, I got to give you a reading list, right? Here's a syllabus for learning more about disability, public interest technology, facial recognition and in fact, the reading list is so long that I need two slides.

Everybody take a a picture of it because I'm going to go to the next slide. Right. 54321. Next slide, take a picture of this one too. Great readings. I mentioned Ruha Benjamin’s Race After Technology, and you probably know about Safiya Umoja’s Algorithms of Oppression, and another vented plastic Cambridge connected book is Programmed Inequality by Mar Hicks. And of course Weapons of Math Destruction by Cathy O'Neil is one of the books that kicked off the entire fairness, accountability and transparency movement. And,if this is not enough resources for you there are even more resources in the back of the book and I am delighted to be here with you tonight and now we have time for some. Questions. So thank you so much.

Gina Neff:

Thank you, Meredith. That was great. We're going to take questions from all of you soon. So get your questions together. Be thinking about that. We have 2 microphones circulating through the audience. I want to bring in the story that you tell of being told: just believe this thing, it's magic. Pretend it's magic. It works. Recursive recursion, recursion. Just believe it, don't question it. You'll come to understand it. Watch it work.

And you might, I think very powerfully, magic works until it doesn't. I wonder, in kind of thinking about that, to what extent have people bought into this idea that these systems are somehow magical and we just need to trust everybody making them?

Meredith Broussard:

Yeah. So this story happened when I was an undergraduate, so I studied computer science at Harvard as an undergraduate, as did Mark Zuckerberg. He took you know exactly the same class from the same professor and what happened in this class is we were learning about recursion, which is a super cool computer science thing, but I did not understand it at the time and I went up to the professor at the end of class and I said alright, I just, I just don't get it. Like you do this thing and then do this other thing then like it stops but like why does it stop like, there's this thing called the base case and I just, I did not understand that. And he got frustrated talking to me about it. And he said just pretend it's magic. Just don't - stop asking me questions. He was kind of annoyed with me, I could tell. And I felt really insulted. But because I'm a rule follower, I did what he said. I just did the thing. I just used the, I just wrote the recursive code and it didn't work and I was like alright well, I'll figure this out eventually and eventually I did, I had a kind of more intuitive understanding of it, but it was not a kind thing to say and it occurred to me that this kind of work pedagogy is pretty typical of that generation of elite computer science education. And that is the kind of pedagogy that the elite computer scientists are being trained in. They're then going to Silicon Valley. They are then going to these big tech firms and telling you know the other people in the organisation oh just trust me, it just works. Don't ask too many questions because you know I'm smarter than you. I'm in a position of power. And that's not sufficient anymore because these algorithmic systems are harming people and often people use- they kind of boundary police technology. They make it so that other people are intimidated by the technology and so they won't ask too many questions. And you can make enormous profits that way. But I don't think it's getting us toward a better world.

Gina Neff:

That brings us to literally today's headlines. Everybody wants to talk now about generative AI, ChatGPT and the other tools that are available. We're having to wade through an enormous amount of hype and magic. What's one thing you wish could have cut through that message. Just one thing because we've got like a dozen people who have questions that we've got one thing.

Meredith Broussard:

All right. I really want people to understand that generative AI systems are not the reinvention of fire. It is not going to change everything. It is going to change a couple of things a little bit. But we are not at the dawn of a new era. It's just some new technology. Generative, you hear a lot,  a lot of hype about generative AI. We're definitely at a specific point in  the hype cycle, right now, but something like ChatGPT is basically a souped up version of the auto complete that you were already using in your gmail and if you remember a couple of years ago when autocomplete came out, people were like, oh my God, this is going to make writing so much easier. And you know it, it really has not set the world on fire. So that's what's going on with generative AI.

Gina Neff:

There's a phrase that's had to be blocked on auto complete an e-mail because it's very common. I love telling the story. It's very common, but used in the wrong context is disastrous. Can you guess what it is?

Meredith Broussard:

I desperately want to hear this now.

Gina Neff:

Yes. Sheila knows.

Audience member:

I love you.

Gina Neff:

I love you. Let it be known, Sheila. Sheila knew. The thing we say, some of us I learned not in every language, but we say in English quite. commonly, when we're signing affectionately to friends and family we might sign off, love so and so, but put in the wrong business context. We can't do that. So it's so common. In the email corpus language that engineers have had to specifically remove that phrase. So we've taken the love out literally.

Meredith Broussard:

Where is the love? It is not in the autocomplete

Gina Neff:

It's in the black box, the love is in the black box. With that, we have time for questions.

Audience Question:

Hello. That was so incredible. So enlightening. Thank you so much. My question is connected to how we should remove AI from policing and that got me thinking about technology within marginalised communities and if we're removing it from policing and to then try to come up with a more social solution that has to do with education, access to resources, how can technology exit the policing area and enter into the educational slash community building are?

Meredith Broussard:

That's a great question. One of the ways I think that- But OK, so I think there is an individual answer, there is a collective answer and there is a policy answer. I think that the policy answer is actually tied up in kind of larger geopolitical stuff because the EU is ahead of absolutely everybody else in implementing AI regulation. And one the concepts that is in the proposed EU regulation around artificial intelligence is the idea that AI will be classified at a high risk and low risk uses. So something like facial recognition used to unlock your phone, it would be low risk, right? Facial recognition on my phone doesn't work half the time anyway, but it doesn't really matter because there's a passcode. It's not a big deal. But facial recognition used by the police on a real time video feeds would be a high risk use because there would be a great risk of people, especially with darker skin being misidentified, being caught up in you know, surveillance strike notes unnecessarily, so that would be a high risk use, and that would be registered, monitored regularly, right. So that is an example of a policy proposal that I think could potentially work really well. Collectively I think we need to examine our faith in technological solutions, we need to admit that police are no better and no worse than anybody else at using technology. Most of us are pretty bad at using technology, honestly. And so the idea that we would give these incredibly sophisticated systems to police and kind of expect them to use them really well, it's not like it's not a smart thing to do. And then on the individual level, we need to, you know, to just examine our- we can examine our beliefs around technology, invest in advocating for policy solutions, because it's not just about individual people avoiding surveillance, I mean there are things you can do to avoid surveillance like you could wear complicated makeup. Or I think there's fashion now: anti surveillance fashion. Which I mean, I'm very excited about the potential for anti surveillance fashion, but it's maybe not practical because you know sometimes you need to just, like, wear your uniform and go to work. So I think that there are. A lot of options. That starts with examining the unquestioning of faith, we have in AI.

Audience Question:

Thank you, Mary. Do you have any fair examples of existing AI systems? Just to lift our moods a bit.

Meredith Broussard:

Do we have any examples of AI working well? Yeah, no.

OK, I do. I do have an example of algorithm auditing working well to uncover problems in AI systems. So there are a couple of examples. Of effective AI audits, Cathy O'Neil, who wrote Weapons of Math Destruction, has a consulting company called Orcaa. They are one of the few organisations doing algorithmic audits, and so you can hire Orcaa to do a bespoke audit on your algorithm to help figure out what are the ways that it may be problematic? What are the ways that might be violating the law? And so if people are interested in making algorithms incrementally better, I would recommend checking out algorithmic auditing and also checking out mathematical methods for uncovering bias, and then once you uncover the bias then sometimes you can use mathematical methods to remediate it, but sometimes it's simply not possible, and then you need to just do something else.

Audience Question:

The content that's being - I'm not too techie and I'm asking it as a disabled journalist - the content that's been scraped from the internet copying images that is used to train AI is largely unregulated, so it includes biases, disinformation and a lack of authentic diversity. AI trained on this content, will embed and perpetuate these biases and overtime this lack of diversity is reinforced. What needs to be done at this point to ensure diversity is considered when training AI?

Meredith Broussard:

Ohh, I'm so glad your absence because I am super interested in what people think is the training data for things like ChatGPT and what is actually in the training data for things like ChatGPT. So one, there was this amazing investigation by the Washington Post about two weeks ago where they looked at what's called C4, the clean crawled something corpus. What's the other C? Oh yeah, at any rate, there's this data set called the common crawl, and it has been assembled over years by making these things called web crawlers which go all over the web and scrape data and just deposit it into this data set and the common call is a collection of just crap scraped from the Internet. It includes things like the entire corpus of 4Chan. It includes things like you know, white supremacist hate sites. And the set, the subset ff the common crawl that the Washington Post examined is something called the Clean Common Crawl corpus. And when I say clean in terms of data, what does that suggest to you? You know what do you think when I say clean data?

Gina Neff:

Ready to be analysed.

Meredith Broussard:

Yeah, it's ready to be analysed. It doesn't actually mean data that has been purged of bad words, right? A lot of people think that clean data means ohh, they've taken out all the curse words and that's you know, it's clean data that's been purged of year pornography. No, it's just tidy, like it's just structured. It's just ready to be ingested, right. So there are all these differences in how computer scientists use language and how regular people use language, and it leads to a lot of misunderstandings and it's part of what people misunderstand about these data sets in general. But no, like the only way to meet- nobody is making generative AI systems that are based on, like, curated data sets. They're just grabbing everything they can.

Gina Neff:

We have a question in the balcony.

Audience question:

Thanks so much. And my name is Seyi from an organisation called Glitch. So this topic is super interesting to me. Going back to the topic of the appropriate use of AI at Glitch, we unapologetically focused on black women's experience of online abuse and so content moderation is not up to par. When it comes to protest, it's not until. When it comes to their own terms, terms and conditions along gender alone race, we have this special report coming out to look at how AI and moderation in its design is perpetuating something where one of our recommendations is we want to be able to. Make this better. Are we actually create our fantasy? It's not actually able to, but not actually able to make. It better we. Need to think of a more a cool solution to content moderation so that platform stays safe for black women.

I mean, I think that one of the things. We need to do so. First of all, I'm so excited to hear about your organisation. I love the synchronicity of you being glitched and me being a having a book about glitches. And I want to learn more about your organisation. I. I think that. Undermining the patriarchy is really something that is necessary in order to have better content moderation, Without doing that like I'm not sure what kinds of technological fixes we can. Have in place it is. Theoretically possible to to put in better content moderation, it is a matter of will and funding. One of the can I talk about is that an acceptable thing to? Talk about from this. Maybe not in graphic detail. Well, I one of the reasons that we don't have filters on, you know on browsers or in dating apps is because the people who are making the apps don't want them, right? Like, that was a that has been a deliberate choice by. Developers. And then it gets backed up with this this legal. Manoeuvring where people say. Oh, yeah, well, if we if we moderated more and then we'd have to moderate everything. But you know, honestly, they're moderating everything anyway, so it doesn't really hold much water. It's really just a matter of choice.

Gina Neff:

We probably have time for two. One, I'm just told we have.

Time for one more question. Let's go here.

Ohh, we'll take two quickly. The question so we'll get this question and. Then we'll get your question.

Audience question:

Hi this is not directly related to race, but since we're talking. About like the. Ethical and unbiased use of a UI and the possibilities where like you talked about how like datasets are like the source of all these AI how can we actually like? Look for more. Ethical sources of data that are more. Diverse in general, because there are a lot of arguments about how, for example, Art AI's generative AI is actually taking data with completely no consent from the artist and it's harming a lot of creative about creatives online. Like what do you say about that? How do you think? We can actually make good data sourcing, also ethical access, good data sourcing.

Gina Neff:

And we'll take the last.

Audience question:

Question and my point was just like saying. Mid journey is. Highly motivated to scenography those specific words that are not included in there. But I was just saying down to prompting in generative AI or mid Germany, it's all down to the user to make sure that their images are inclusive and as a tutor at Cambridge, I wanted to get some images about just time management and I wanted to make myself a generative image. If I use Cambridge University as a prompt, I won't get any women or black ethnic minorities, so it's up toeed to change the visual weight of my images. And get them. In there so. Use the technology. The amazing qualities of it, and it's up to us as individuals to understand and use as techniques.

Gina Neff:

Inequality and better images. Comments on this?

Meredith Broussard:

Yeah, it is is possible to make more inclusive datasets. It is possible to use the technologies to make your outputs more inclusive. I think one of the things. We need to demand that the input data is more diverse and we need to demand that the that the output is more diverse and we need to just centre diversity in our conversations around technology.

Gina Neff:

And on that cheery note, I want to thank. All of you for. Being here and Meredith I want. To thank you. For joining us tonight., it was a great event.