Critical AI and Design Justice: An Interview with Sasha Costanza-Chock

Sasha Costanza-Chock: I would trace my work on [design justice and community practice] to my long-standing involvement in the Allied Media Conference, which began in Detroit and has been going on for about two decades. It is a pretty phenomenal event where people come from many different types of media and cultural work—technologists and designers, community organizers, software developers and coders, activists and filmmakers—to share ideas, build networks, launch new initiatives, and skill-share theory and practice. Design justice, the Design Justice Network, and the Design Justice Network principles all grew out of that space.

Back in 2015, Una Lee, Jenny Lee, Wes Taylor, and a number of other folks organized a workshop called “Generating Principles for Design Justice.” That work actually built on a whole other prehistory: a hands-on practice space at the Allied Media Conference called the Future Design Lab. That evolved from what was called the Media a Go-Go Lab and the DiscoTechs, which came out of the Detroit Digital Justice Coalition. So, there's a long history of extraordinary tech, community-building, and community-organizing work that's rooted in Detroit but is now linked to social movement networks across the United States and increasingly internationally.

I recently met with the steering committee of the Design Justice Network, because we're planning to launch our new mission, vision, and intentions. Increasingly, we've felt the need or the urgency to clarify what the actual mission and vision of the network itself is, and there'll be more coming out about that on the network's website, designjustice.org. When we were meeting to talk through this, we were thinking about and sharing our personal stories of how we became involved. I hadn't thought of this for a couple of years but remembered that my first activity with the newly formed Design Justice Network was a sort of multihour workshop. I did a sort of futuring exercise called Newsjack, which is a modified version of a Mozilla tool that lets you remix websites. You can just click on any element in a webpage, edit it, and share the edited version. The workshop was for people to remix the front page of The New York Times or their favorite other news site, imagining that the principles of design justice had been widely adopted. We imagined it was ten, then twenty-five, and fifty years in the future, and people came up with fantastic future headlines of what the world would look like if design justice principles were widely adopted.

Since then, we've evolved from a set of principles to launching signatories: there's, I think, now around two thousand signatories to the principles. We launched formal membership about a year and a half ago. We now have close to five hundred members who are active in the network. Either they're working in what's called “local nodes,” which are configurations of people working in particular cities or regions, or we also have working groups, which are dedicated to elaborating the principles and applying them in different areas of design and different areas of life . . . so a lot has happened. We now have three network coordinators who facilitate and support all the activities of the network and a growing steering committee.

That's some of my story. For me, this is really rooted in a community of practice that comes through the Allied Media Conference and all of the theoretical work about what design justice might mean. For example, how does it relate to other histories of participatory design, human-centered design, and collaborative design? Where do design justice principles cohere with these histories, and where do they diverge from other ways of approaching design work? What is the relationship between the ideas and principles of design justice and various bodies of critical theory and scholarship? Now, that's work that I did in the book, but it really is rooted in and emerges from my work with that community of practice. I started this off by saying design justice is a framework for analysis—of how design distributes benefits and burdens between various groups of people—and explicitly focuses on how design can reproduce or challenge the “matrix of domination,” which is Patricia Collins's term for linked systems of white supremacy, heteropatriarchy, capitalism, ableism, settler colonialism, and other forms of structural inequality. In addition to being a framework for analysis, design justice is a growing community of practice.

Kate Henne: That's so helpful to hear, and the commitment to practice has come through strongly in the book. This is a rich illustration of a commitment to praxis, and it conveys how a text can reflect the critical values that you've just described. In this case, you're bringing theory meaningfully to bear on practice, acknowledging the various relational connections that necessitated the book's coming into being, providing grounded reflections to help us make sense of how matrices of domination can manifest, and inviting readers to become part of the larger project of remaking design. I'd love to hear more about you and how you approached that commitment to practice through the book.

SCC: In terms of how I tried to engage the actual practice or relationship between theoretical knowledge, lived experience, and practice-based learning through the book, one way was by trying to clearly and repeatedly talk about “where's this coming from?” Throughout the book, I tried both to reference and to talk about the different sources of knowledge that are active for me as I learn about, think about, and try to practice design justice; to cite not only scholars who are writing about design theory, but also activists and practitioners to open up citational practices; and to talk specifically about “here's this time and place where I learned this thing,” and “here's where this piece of knowledge from this book came into contact with a particular community's organizing process.”

So, one of the ways [to engage actual practice] is through referencing and citing. I think another is to try to make the text clear and useful to a wide range of people. It's true that the book is published with MIT Press, which is an academic press and not a trade press. That brings with it certain expectations. It's a book that is used for teaching in universities, so it's not necessarily written for a mass audience. But I have been very happy to find that the book has had a pretty wide circulation, and I think it's because, in part, I tried really hard to shift registers between very clear, plain-language descriptions of the different dynamics, processes, or points that I was trying to make, and then to talk about them with reference to the different bodies of work or scholars that I might be citing. Whenever I was talking about a particular concept, the “matrix of domination” or “affordances,” for instance, I tried to give as clear examples as I could in plain language, sometimes drawn from my own personal lived experience.

The book opens with a story of me as a nonbinary trans feminine person going through airport security. I give this very visceral description of what happens when sociotechnical systems that are configured around certain assumptions—in this case, of cisnormativity, which assumes that everybody has a gender, gender presentation, and body that match what they're assigned at birth—meet the messy and beautiful reality of humans. So, I talked through that example, which I think connects with readers. I try to do that sort of work repeatedly throughout the book—draw on my own experience. But the task of design justice didn't start with the book, and it doesn't stop there. The book itself is an artifact that emerges from this community of practice, and that community is growing in a very organic way.

I would also say that the Design Justice Network is not the only space where people are talking about design justice and doing design justice. There's a way in which the moment was just ripe for this conversation, and so there's a number of parallel communities, thinkers, and texts that have all emerged around the same time—trying to think through very similar questions, sometimes directly referencing the term design justice, and some of the concepts and language that we're using in the Design Justice Network; and sometimes with other reference points. For example, in parallel to the emergence of the Design Justice Network, the Design As Protest Collective emerged, which is a community of architects and planners who have created “design justice demands for the built environment.” They are calling for an end to architects and planners participating in carceral design. So, no more helping to build prisons and militarized borders. They're calling for the end of what's sometimes called “defensive architecture,” which is this school of thought that you can control people's activities in the built environment and reduce so-called criminality by making public space inhospitable to certain types of people and bodies, typically Black, brown, queer, and/or disabled bodies. They're saying we shouldn't participate in that. You can read about their work by looking for “Design Collective” or “design justice demands built environment.” [Editor's note: see https://www.dapcollective.com.] There's also a number of design shops, like Equity by Design or And Also Too, which is a design studio started by Una Lee. There are many, many, many examples, and I named some of them in the book, but of course, the space is growing and evolving very quickly.

So I think it's an important and interesting moment for the theory and practice of design justice. There is just a huge amount of ferment in the conversation around design justice in technologies broadly, and in artificial intelligence specifically.

Anand Sarwate: Thank you very much. You talk about the messiness and beauty of humans and where humans meet design. That's where the book, and a lot of the examples you draw on, find specificity, focus, and saliency.

My question is about how we need a parallel framework to push things down what I call the technology stack. Here is an example, just to contextualize a little bit: you have a computer vision system that's doing face recognition, and there's an obvious bias there. Those images are generated by cameras. Those cameras contain chips that do the imaging. Those chips are silicon devices that collect the photons that come into the camera. And so, when we talk about design, we talk about design at this low level as well. We use the same word, “design,” but it's not quite the same way in which you used “design” in the book.

So my question is: how do we extend this framework, or how do you see ways of extending the framework that you developed, beyond the part where technology meets humans? Because those technologies are developed in ways that make it murkier to consider which humans we're affecting. The intersectionality analysis becomes a little bit more challenging when I'm trying to figure out, “how do I layer the silicon of this device?” Right?

SCC: This is a really great question. So, I'll give you a snarky answer and then a real answer. The snarky answer is that you and your students should join the Design Justice Network and start a working group on design justice in chip design—then figure out what it would mean to apply the ten design justice principles!

AS: So, you think they're universal in that sense? They apply any time I use the word design? That's actually kind of the question: is it encompassing or an inspiration for parallel projects, maybe in other domains?

SCC: In the book I spend most of my time talking about the design domains that I'm most familiar with, and I'll talk a little bit more about that in a second. But the Design Justice Network principles are high-level enough that people in many different design domains have found them useful or fruitful in thinking about how to improve particular types of practices and processes within their own subfields. I'll give you one example that is very far afield from the past, and then I'll bring it back for a second example.

In the Design Justice Network, we do what's called member stories. We regularly feature different members of the network: we interview them, and they talk about how they're applying the design justice principles in their own domain. We had one great story from Leila Sidi, a luthier (a guitar maker) who uses the design justice principles in her guitar-making shop, which is called TunaTone Instruments. She first reached out to us a couple years ago and said, “I'm so inspired by the design justice principles. I'm using them to help me make guitars.” And we said, “What? How does that work? What's the relationship between the design justice principles and guitar making?” And so we met with her, and she ended up doing a member talk. Basically, we learned a lot from her about how there's a Big Guitar industry—Fender and all of them—and they have standardized dimensions, materials, and production processes around what sells the best in the markets that they find the most profitable. The body sizes and shapes of guitars were standardized around certain dimensions of guitarists’ bodies: mainly, relatively middle-class men in the global North who had disposable income and wanted to play guitars. So, a guitar is a scalable thing. You can make frets at pretty much any scale, and it'll still work, but it was standardized in a certain way. And so the only smaller guitars typically available on the market are kid's guitars, which means they use cheaper materials, they're less expensive, but they don't feel or sound as good. Leila uses the design justice principles to develop guitars for differently sized and scaled bodies. That was a really interesting lesson for us, and we had never thought that someone would be applying these to instrument-making, but they were useful and generative for her.

Another example is a set of nurse practitioners who reached out to us with stories about how, in medicine, nurses have a really high level of interaction with people receiving care in institutionalized care spaces, like hospitals, so they constantly innovate both medical devices and medical processes. But that is often devalued. It's not made as visible. And it's often doctors who are seen as the innovators in medical devices and practices. One community of nurses has been extremely inspired by the Design Justice Network principles, have joined, and have been using the principles to try and create more visibility for the types of design and innovation that are coming out of that community.

Those are two stories that speak to the question, “what does it look like for someone to apply or use these principles as a sounding board in a domain?” That's not one of the areas that I talk about very much in the book, but we have many examples of that—certainly in hardware design and development as well as other subdomains, like camera design and so on.

AS: I have a follow-up about this. My question is more about places where people consider themselves doing design. For instance, in apps it's hard to articulate that they even have a default user, because the technology creates bias. The way in which we designed the sensor device for cameras helps enable the bias, but when you're designing that you're not quite sure what the camera is going to be used for, so the impact is felt in particular applications. The principles speak to involving the community, but identifying that community is complicated. There could be a default identified community, as you talk about in the book—an expected community—but sometimes it's less clear that you're assuming a default user who is, say, a cis white male in the global North.

SCC: Yeah, sure. On the one hand, I think you could say there may be certain areas of fundamental or basic research where it's a much bigger stretch to try and understand how design justice principles would directly apply. Maybe that's the answer to the question you're asking. But let's say you've spent time thinking about how to design the chip and then you're going to move to production. Design justice would have a lot to tell us about how to think about the chip's design and how to scale up production of this particular chipset in a way that will meet the principles. If you did that, then you'd start thinking about the environmental impacts, you'd start thinking about energy costs, you'd start thinking about the labor practices and who's going to be involved in the factory, cleaning the factories where the chip is going to be produced, if there are harmful by-products of the production process. That could then actually become a pretty useful feedback loop with chip design.

Can we rethink how this is designed in a way that would help us? For example, that relates with which materials you might choose to use, and whether you can use materials that are less harmful in the ways that they are extracted from the earth and so on. So even when we're talking about building a fundamental component that is going to be deployed in many different systems—so we're not concerned with the end user or interface design, or any of those places where it's very obvious how bias and inequality might show up—I would still invite people doing that type of work to consider the relationship between the approach that they're taking and the footprint, the labor practices, the extraction that will need to happen and cannot be modified.

AS: Yeah, that's great, because we talk about environmental impact and these types of things in a generic sense, but we don't think about it in terms of community and harm to people. We don't use the matrix of domination as a frame when we talk about those impacts, so that's very interesting to think about.

SCC: Another place where this conversation might be really interesting is thinking about the cycle, or the speed, of replaceability. There were some interesting motions a few years ago, although I don't think it's gotten very far, to try to modularize mobile phones more. I forget the name of the project, but it was basically saying, “Well, if the main thing that we're swapping out at this point is the processor and the camera, why can't we build devices that are designed for that so that we know that we can just swap the processor and the camera next year?”

Instead, we've gone down a different path, and it's increasingly harmful. E-waste is a huge ecological problem. That's another axis by which we might enter the conversation of what design justice means in the hardware space.

Sabelo Mhlambi: So, this is quite fascinating. I think many people in the audience might be familiar with terms such as “AI for social good,” “data for good,” “tech for good,” and even this whole “ethics and AI,” which is supposedly about creating ethical frameworks for good. I found it really interesting that in the first chapter, you mentioned that sometimes trying to design for good—whether it's AI, technology, or even design—doesn't often go far enough. In fact, you have a quote where you say, “good intentions are not necessarily enough to ensure that the design process and practices become tools of liberation.” I'm wondering, can you go more into what's meant by that, and how we in the community can go beyond design for “good AI” to really creating tools for liberation?

SCC: Thank you. One of the principles of the Design Justice Network, principle three, is that we prioritize design's impact on the community over the intentions of the designer. This idea of impact over intentions is a meaningful way to think about accountability. It's a frame shift that can hopefully help people pause or think again about slippery conceptions of “design for good,” where intentions are usually prioritized over what actually comes out the other side—over impact on the community. I think that this is really informed by conversations in the network around racism and the way it operates. For example, when white or non-Black people of color, or really anyone, is called out on a racist speech act or an action, very often one of their first reactions will be defensiveness: “Well, I didn't mean it that way,” which is that idea of, “Well, my intentions were not racist.” And that's because people are concerned about not being boxed in as a racist. But the question isn't, “What was the intent?” The question is, “What was the impact?” If we can switch our mindset and say, “Well, you know, I didn't mean to cause harm with this word that I said, or with this thing that I built, or with this process that I put in place,” and instead flip around from a defensive posture and say, “Well, I didn't mean to do this harm, but people are telling me that they feel harmed. What do I do with that?”

I think design justice is a community of people who are interested in asking that question. Sitting with that. It's a difficult one, because none of us wants to think that we're doing harm. But—either because we haven't thought things through well, or because we're just operating within already very exploitative systems and institutions, or because something that we built was then taken and used in a way we never intended by someone else—we can do harm unintentionally.

There's just such a huge difference between defensiveness and denial and saying, “Well, I didn't mean to do that,” and, on the other hand, recognizing that harm has occurred and then figure out what can be done to address and redress that harm, and change things so harms don't continue to be multiplied in the future.

SM: Thank you for that response.

You also mentioned this idea of the Zapatista movement, which talks about the many worlds existing within one world, and that it had a profound impact because the idea of designing for many worlds is reflected in your title. So does this idea imply that marginalized communities can make their own worlds? Can those who are marginalized have the power to really create alternative roles? Do they have to negotiate this power away from those who currently dominate the main society, or can they find other ways to break free and be able to build these alternative worlds without necessarily having the participation of the existing power structures?

SCC: Oh, that's a really deep question. I think that both of those things need to happen. The idea of building a world where many worlds can fit . . . for me, that is inspired by the indigenous Zapatista movement in southern Mexico that rose up in the mid-nineties against the North American Free Trade Agreement. They felt that the agreement was going to deepen the hold of the capitalist system (via US-based firms) and attempt a militarized erasure of Indigenous peoples, communities, and lifeways by establishing the Mexican government as a proxy for the US. For example, if a superhighway was needed to carry goods from Mexico into the US, and it required the displacement of Indigenous communities and there was going to be resistance to that, the Mexican government was indicating its willingness to use whatever force it needed to make sure it took place—not in consultation with the communities, not with appropriate compensation for those willing to move. All of that is to say that that idea comes from that liberation struggle, and that idea was really picked up by Arturo Escobar in his book Designs for the Pluriverse, which was very influential for me, and I cite it a lot in Design Justice. He looks at not only the Zapatista movement but at broader currents of Indigenous thought from throughout the Americas. He talks about the idea of how, for hundreds of years, settler colonialism has attempted to commit genocide and ontologically destroy or erase the possibility of other ways of being in the world with a one-world system. And that project is ongoing. That project has also not been as successful as it likes to paint itself within the mediated and educational systems that it controls. Indigenous peoples are still here, still have their own languages, ways of knowing and knowledge systems, understandings of history and of space and of relation with one another, with other peoples, with nonhuman species, and with the earth and with ecosystems. And those are all, for Arturo Escobar, very valuable systems of knowledge that should not be erased, should be respected, and we should find a way to ensure that they're not destroyed—not in a “protected and put it in a museum” mode, but more like “these are living peoples who need to be respected as such.” I think that design justice is inspired by that view.

And so, to bring it back to the other part of your question, I think that not only is it possible for people to create and do design in other ways without engaging with hegemonic culture and institutions; I think that it's already happening and has been ongoing. At the same time, we've moved increasingly to a globalized system that touches everywhere. Climate change would be the core example in that no matter where you live in the world, this will impact you. That's because of processes that are transnational, transgenerational, and are dominated by certain institutions, actors, and nation-states. The former colonial powers have burned far more than their share of the global carbon budget. And so, the way out can't just be done piecemeal; piecemeal means engaging with international institutions and agreements between many actors. I think that both have to happen at once. But people also need to explore other ways of doing things, including design of systems and objects and so on, while also engaging with the transformation of dominant institutions. I think those two practices can relate to each other so that the alternative designs can provide direct inspiration for the transformation of mainstream institutions. And, of course, as mainstream institutions are shifted, they can either reallocate more resources toward the support of other ways of doing things or, at the very least, stop the active and ongoing destruction of alternatives.

SM: Speaking again of these alternative worlds, is there a place, or a time and place, to decide which worlds should be part of such a pluriverse? Where there are many universes, many worlds? Or do we just accept every single point of view that can exist and sort of give it prominence or give it a place in such a world?

AS: My question is related to that. I think it's about priority. If you have a particular project in mind or particular community in mind that you'd like to do work with to design something, the design justice principles are really very helpful and help you understand that. But one question is, “but how do I prioritize which communities to go to?” Obviously, you can ask “who's being hurt the most and let me help them,” but there's still a question of whether there's a principled way of thinking about that, in using the design justice principles that you've developed.

SCC: I get asked this question a fair amount. Of course, another question people often ask is, “what about when there's conflict within a community?” Community . . . it's a lovely term. But in fact, communities are made up of subgroups and individuals, and there's always conflict internally. There can be terrible, highly patriarchal systems reproduced within a small-town community that is otherwise respectful of its environment.

So, there's a couple of ways that I approach this. One is that it is actually this question of “I want to be involved in developing this new product or this new application or this new system; who might be harmed by this?” Then, there is a whole process of what in cybersecurity is called “red teaming.” You spend focused time thinking about what could possibly go wrong: what's the worst way that this could be used? Who might it impact negatively? That process might yield something pretty clear, like, “oh, this might really do harm.”

For instance, let's say we want to build an AI system to help us filter job applicants, because it takes a lot of time and energy to filter job applicants. You would start thinking about, “well, who historically have been most disadvantaged in job application processes?“ Well, it depends where you are located in the world, right? In the US, you can say there's a long history of racial discrimination in hiring processes. There's a long history of gender-based discrimination. People with disabilities are also often discriminated against. So, if we're going to be building the system, then I would say we need to do really deep research on how hiring discrimination has operated in the past. We'd ideally connect with already existing community-based and advocacy organizations that are working on this problem along these different axes and get them at the table as soon as we can in our design process. That way, from the jump, we're already thinking about a problem like hiring discrimination and maybe even making it a primary goal of ours to not reproduce this well-known type of harm.

Now, it may be that through this process, you'll learn that an AI system for classification of potential applicants is not the right thing to do at all. You may need to build a totally different type of system if what you're trying to do is eliminate intersectional discrimination in hiring processes. I don't want to get sidetracked since this is just an example, but in response to a question like, “Well, who do we bring in?” you can start by thinking about who might be harmed. Another way to think about this, which I like even more, is to say that design justice is not necessarily an “approach.” Human-centered design, design for good—the primary figure in those frameworks, in thinking about how you do design work, is the designer. The designer is positioned as the most privileged subject in the process, an expert who's presumably leading and running the process. If communities are going to be brought in, it's typically consultative.

As a designer, I'm running this process, and I want some community feedback, because I know that it will improve the utility of the design. So I'll pay some people to do a focus group, and I'll gather some feedback from them (maybe from my prototype), and then do a few iterations of that.

I think that design justice takes a different approach and really tries to say, “well, communities are often already building things and designing things, trying to make a way with no way, and trying to use the resources they do have to solve the problems they have.” What if we flipped things so that designers, who have certain types of expertise, could add to the pool of expertise—of lived experience and other types of knowledge that include what marginalized communities already know and do? Then the model is: what community do I believe is doing something important to help us create a habitable planet, or to overturn centuries of racism, or to transform so that we no longer are confined to the gender binary? And how can I put this specific skill set that I have at the service of some already existing, organized, community-led effort? That's why the subtitle of the book is “community-led practices,” rather than something like “community consultative practices” or “practices where you extract knowledge from a community to design the thing that you want to.”

KH: That's a great series of questions and a thoughtful set of reflections in response to them. I want to ask one more question, given that Sasha gave us a teaser about the Design Justice Network rethinking and rearticulating its vision—clarifying it in light of, perhaps, recent events or the network's enduring connections within a wider movement. In what directions do you see Design Justice going now? How are you refining the mission and vision at this time?

SCC: I talked about that a little at the beginning of the conversation, but we're about to launch a new mission, vision, and intentions for the network publicly. We're also going to invite all of the working groups and local nodes in the network, that are operating in different cities and regions around the US and world, to develop their own intentions and then share those back out.

In the pandemic, we've also spent a bunch of time developing what's called “care circles.” These are practices of care: recognizing that we are human beings and that we need space and time for care—which is not something that is usually prioritized in work environments.

The other thing that I would say is that so much is happening in the growing movement to create more equitable and accountable artificial intelligence. I left a faculty position at MIT to work as the director of research and design at the Algorithmic Justice League, the organization that was founded by Dr. Joy Buolamwini to create a movement for more equitable and accountable artificial intelligence. There's so much happening in that space. Just today, Dr. Timnit Gebru announced the launch of her new distributed AI research network DAIR on the one-year anniversary of being forced out of Google's so-called ethical AI research group. There's a really beautiful statement of intentions and approach, which is about working to reduce and minimize the harms of AI systems and do so in a way that is very akin to design justice principles, which are informed by and led by the people who are most likely to experience those harms.

There are also so many legislative proposals now, and they're happening at multiple levels. At the city level, we've got New York City announcing that they're going to be auditing and speaking up about employment systems. If you want to use an algorithmic hiring system in New York, that's going to be subject to audits there. There are some challenges with the way that they've set them up—they're not intersectional, they don't cover some areas that are important—but it's an important move for public oversight and accountability.

We then have state and federal proposals. We've got the Algorithmic Accountability Act that is about to be reintroduced by Senator Wyden's office (there was a 2019 version). A lot of civil society organizations and researchers had questions and critiques, and I think the new version is a lot better. And there are transnational processes. UNESCO released this framework, and there are a number of countries that have signed on or indicated that they will sign on (the US is not among them, of course). For more on that I would say Khari Johnson over at Wired has a great summary article, which is about the growing movement for AI accountability. So, the conversation is shifting. The film Coded Bias came out on Netflix this year, and I think that's a really great teaching tool and entry point for people to understand what we mean when we're talking about biased and harmful AI systems that are increasingly being deployed in many areas of life. Very importantly, people are doing something about it, pushing back on these systems, and taking different actions to create more accountability for these systems. And so I'm just really inspired by all the work that's happening in these different domains.