Technology at hand (photo: Edwin J. Torres/ Mayoral Photo Office)

For years, people have worried that one day robots would replace all of our jobs, but today, they already can decide if you get one. And even worse, they’re racist. That’s the dystopian reality of artificial intelligence tools screening job applicants, automating and amplifying racism and sexism. Racism runs rampant in human hiring, but AI is even worse. And thanks to a bill that the New York City Council just passed, the robots are likely here to stay. If signed or allowed to age into law, the legislation would give the green light to employers who use these biased tools and the tech companies that make millions by selling them.

In 2014, Amazon developed a new AI-powered tool to cut costs. But the system amplified Amazon’s own past bias, selecting men for technical roles and promotions. Candidates who went to women’s colleges were penalized because existing employees hadn’t attended those schools. Amazon was an early example, but it’s far from the only one. 

Globally, 55% of companies use AI hiring and evaluation software. Chances are that AI has reviewed your own applications, and you never even knew it. This civil rights catastrophe is what drove the City Council to take recent action, passing a new AI hiring bill. Sadly, the passed bill, if it becomes law by mayoral signature or inaction, is going to make things worse…a lot worse.

Months ago, advocates warned that the Council’s proposal was too weak to stop the tech companies selling racist and sexist technology. So the Council amended the bill, but made it even weaker. The result is civil rights in name only, a toothless requirement for companies to “audit” their algorithms without any enforcement to actually make sure they comply.

Algorithmic audits aren’t like the financial audits companies typically go through. Part of the reason is that the field is so new. Mandating an algorithmic audit today would have been like requiring financial audits before developing modern accounting rules. It can tell us a lot, or it can’t tell us almost nothing, and the choice is completely up to the auditor.

Even worse, on the rare occasion a company fails this weak audit, finding that software has “disparate impact”,  there are no consequences. The auditor only must make a “summary” of the audit available to the public. But AI is all in the details. A summary isn’t going to tell us what we need to analyze these tools for ourselves. The only way to understand AI is to look at every aspect of how it works, starting with the data used to train an algorithm, and continuing through every ramification of how the software is used in practice.

And what happens if a company ignores the audit requirements completely? Almost nothing. Rather than letting those who AI harms have their day in court, the law places the burden of enforcing all of this in the hands of the New York City Commission on Human Rights. The venerable agency does tremendous work for New Yorkers, but its staff was clear to the City Council that it doesn’t have the capacity to take on this new work. Rather than listening to the agency and advocates and letting workers sue companies that use biased AI, the Council kept enforcement with the Commission and avoided the possibility of real accountability.

The city should be blocking discriminatory software, not supporting it. But we are doing just that by allowing corporations to claim their discriminatory products earned our stamp of approval. Rather than blocking discrimination, this weak measure will encourage more companies to use biased AI. Vendors will tell companies that as long as they comply with these weak new rules, they’ll be safe to automate with AI. New York has the chance to become a leader in digital rights, but if it becomes law this bill will make us a leader in supporting discriminatory tech. 

We should applaud the City Council for acting on discriminatory AI, but this bill would worsen the problem. Any law that does not require deep transparency, implements a categorical ban on biased systems, and provides a private right of action will only allow terrible technology to automate racism and sexism into the fabric of New York City.

Mayor de Blasio must veto this bill and ask the Council to submit one that includes stronger protections. 

If this mayor and City Council refuse to act against the tech lobby, it will be up to their successors next year. Our current mayor may have one foot out the door, but biased AI is here to stay.

***
Albert Fox Cahn is the founder and executive director of the Surveillance Technology Oversight Project (S.T.O.P.), a New York-based civil rights and privacy group, and a visiting fellow at Yale Law School’s Information Society Project. Evan Enzer is a D.A.T.A. Law Fellow at S.T.O.P. and a graduate of the University of California, Berkeley, School of Law. On Twitter @FoxCahn & @EvanEnzer.