(photo: Rob Bennett/Mayoral Photography Office)
During the first weekend of March, the New York City School of Data - a network of advocates, activists, and professionals for an open data ecosystem - hosted a day of panels and workshop sessions in recognition of the international day celebrating open data and the fourth anniversary of the city passing its first open data law. City Council Speaker Melissa Mark-Viverito and Manhattan Borough President Gale Brewer, who have each promoted open data initiatives, delivered keynote remarks at the event.
“As someone who represents some of the most vulnerable, yet resilient New Yorkers in this city, I want to make data a priority,” Mark-Viverito said at the gathering of tech advocates, civic hackers, and public officials. “I want our work to tell a story, one that has rarely been told rightly before. I want our data to move people, organizations, and of course, governments to act differently.”
What Mark-Viverito was getting at - something discussed in several forms over the course of the event and on an ongoing basis - is the idea of using data about civic life, people’s needs, and government services to make things work better while ensuring equity.
Late last year, the City Council approved the final bills of a package amending and adding to the 2012 Open Data law, which created the Open Data Portal, but the legislative action has been met with mixed reviews and questions about the portal’s effectiveness remain.
The portal is intended to increase information available to the public about city services and government operations. According to the de Blasio administration, “New Yorkers can use this data to make informed decisions, become more engaged in their communities, solve tough problems, or turn their dreams into a reality.”
Data available includes the extent to which city school buildings are being used, taxi trip pickups and dropoffs, tree censuses, requests made to the city’s 311 help line, general city budget spending, and much more.
The two newest laws seek to ensure city agency compliance with the Open Data Law, particularly regarding timely release of data, and updates to the Open Data Portal with information released through Freedom of Information Law requests (often known as FOILs). Both are part of ongoing efforts to make the portal more useful. The more quickly data is released, the more useful it is.
On Nov. 30 of last year, Mayor Bill de Blasio signed into law the five prior bills from the open data package.
“In 2012 we set a new bar for transparency and civic engagement with passage of the most comprehensive open data law in the country,” said Anne Roest, Department of Information Technology and Telecommunications (DoITT) Commissioner. “To be as effective as it can for all New Yorkers, open data needs to be usable data – and these bills will help us achieve that goal.”
While “open” is important, “usable” is really the key word, according to data experts.
The open data law and portal are part of efforts to make government transparent and accountable. In theory, data is to be made easily available - and in a useful format - for interested parties, who can use it to help solve city problems. The portal currently offers 1400 data sets through dozens of city agencies and other entities.
As stated by the Mayor’s Office of Data Analytics, “The Open Data Law mandates full coverage of all City public data by 2018. The value is clear – every time a new data set is published on the NYC Open Data Portal, there are new opportunities for users to find insight.”
While there is a wealth of information available, and hundreds of data sets still to come online, it is not always clear who uses the open data portal and to what end. How many New Yorkers even know about the availability of so much data?
And, getting back to usability, there are also significant questions about the format in which data sets are often published.
Still, open government advocates continue to say that it is an essential feature and push to see it improved. Public, usable data, they say, is key to holding government accountable and opening up policy-making to a broader audience, including people outside of government able to help solve a wide variety of problems.
BetaNYC, a nonpartisan group comprised of civic hackers and technologists that has worked with government actors to improve the city’s embrace of civic tech and open data, features a digital project list currently being developed by the NYC tech community. HeatSeekNYC, “a web-enabled hardware platform to detect heating violations in NYC”, and NYC Bus Adherence, “tools for analyzing and visualizing MTA Bus performance data,” are two examples of data-based projects moving civic discourse.
Since the passage of the 2012 open data law, there have been other modifications. In 2014, de Blasio signed into law two transparency bills which require the City Record, a daily publication of city business, to be published on the open data portal in a machine-readable format; the City Charter, the Administrative Code, and the Rules of the City of New York to be published online; and the compilation of laws updated within 30 days of any change.
In July 2015 the de Blasio administration released a new plan for the portal, Open Data for All, which emphasizes community partnership and focuses on making data sets accessible and user-friendly for all New Yorkers.
“[Open Data for All] means it will be easier for people, even those with no programming experience – like myself – to find the information they want, and better ways to utilize that information,” said de Blasio.
De Blasio’s rhetoric shows a typical shift from remarks by his predecessor, Mayor Michael Bloomberg, who remarked that sharing data with the public “[catalyzes] the creativity, intellect, and enterprising spirit of computer programmers to build tools that help us all improve our lives.” The evolution of the portal is aimed at ensuring that every New Yorker, not only computer programmers, will be able to utilize and benefit from the information being made available.
The Mayor’s Office of Data Analytics (MODA) and DoITT led a citywide engagement tour, assessing the needs and priorities of New Yorkers, across user types and domain areas. Throughout the fall, Dr. Amen Ra Mashariki, Chief Operations Officer at MODA, met with CUNY students, community groups, civic leaders, and others. MODA intends for the engagement tour to culminate in a Open Data Summit.
“[The summit] will bring people together who we’ve engaged over the citywide tour and let them know what we’ve heard from them and then we will identify strategies,” Dr. Mashariki told Gotham Gazette in an interview late last year.
In October, Dr. Mashariki testified before the City Council Committee on Technology, reporting on the progress of the tour at that point and the portal in general. He announced that the Open Data Portal contained 1,386 data sets -- up from 1,268 in 2014. Still, City Limits reported late last year that datasets due to be published are missing deadlines and that there has been little enforcement of the law. Mashariki said it will not be the number of published data sets that will determine the success of the portal: “The ultimate success [will be] in the number of New Yorkers who use open data in their daily lives. And that’s not just the tech-savvy New Yorkers – it’s all New Yorkers, in all five boroughs.”
At the “summit,” MODA and DoITT will identify strategies seeking to build an open data ecosystem prioritizing expanded access to data sets, high data quality, and enhanced portal usability. “Increased access to data is critical for open government and transparency in the digital age,” said Minerva Tantoco, the city’s Chief Technology Officer.
Technical and community engagement challenges remain as the open data portal continues to be refined and expanded. It is one often lower-profile way in which the de Blasio administration’s reputation for government transparency and effectiveness will be formed.
Technical challenges for the open data portal
“There are currently many challenges,” said Ben Wellington, visiting assistant professor in the City & Regional Planning program at The Pratt Institute and creator of the increasingly popular data science and policy blog I Quant NY. “One of them is the wide use of PDF documents. PDF is inherently difficult to analyze, so when an agency is forced to release data but doesn’t necessarily want to have it widely used, PDF is an opportune way to do that. “
PDF documents, unlike an Excel spreadsheet, are not in machine-readable format, which Wellington and others argue creates a long-term problem for civic technologists who end up spending a large amount of their time extracting data rather than analyzing it.
“It’s just not a good use of advocates’ time,” Wellington told Gotham Gazette. Wellington has become known for using government data to show trends, point out problems, and identify potential solutions. He delivered a popular TED talk in 2014 using big data to give insight for the “worst” places to park in New York City. Wellington’s blog features many articles crunching numbers to illuminate topics from “Trump’s Unpaid Bills” to pay raises for City Council members.
City Council Member and Chair of the Committee on Technology James Vacca has concerns regarding another aspect of city data - like with quality, the timely release of data from city agencies leaves much to be desired. Vacca sponsored a bill passed by the City Council and signed into law by Mayor de Blasio, which mandates a city office or agency examine the compliance of mayoral agencies posting public data sets under the open data law.
“Some agencies are not as diligent as they should be when it comes to posting information, there has to be good coordination among the city agencies,” Vacca explained. “Right now, there really is no enforcement mechanism if agencies don’t comply. We are dependent upon cooperation, which is great, but we need an enforcement mechanism.“
Wellington suggests that a position should be created in every city agency - an “open data liaison” - to act as a point of contact for the public and be responsible for any inquiries about information released by the agency. “Today, there’s no way for the public to understand who’s responsible for any dataset,” Wellington noted, which can only further obfuscate any process of data clarification, sourcing or transparency.
The New York City Transparency Working Group (NYCTWG), a collection of civic technologists, data advocates, and good government groups, has argued that FOIL (Freedom of Information Law) requests should be published online through a centralized tracking system akin to the open data portal. Earlier this month, the de Blasio administration launched “the OpenRECORDS portal.” It is “designed to streamline the process of submitting, tracking, and responding to Freedom of Information Law (FOIL) records requests as we work toward becoming a more transparent and effective government,” the mayor’s press office said in an announcement.
Along with Vacca’s bill, another also approved by the City Council and signed by the mayor requires agencies to individually review all data released through FOIL requests and determine if the information should be posted on the open data portal. The thinking goes that if one person or group is interested in a certain data set, others probably are, and that if the information is being released publicly in one sense, why not make it generally available.
In 2013, then-Public Advocate Bill de Blasio released a transparency report card in which he called for a mandate to publish online the most commonly-sought information through FOIL requests, asserting “proactive disclosure will save time and resources by posting minutes, public schedules and license data online for easy access.”
As mayor, it took de Blasio some time, testing the patience of open data advocates, but he has now launched a systemized and centralized portal for FOIL requests.
Community engagement and outreach
Noel Hidalgo, co-founder and executive director of BetaNYC, believes that Open Data for All is helping to facilitate more public involvement in the open data portal and wants open data to be used regularly to solve civic problems.
“We’re at the point [with open data] where it’s not just about making an app but about making a culture, an understanding of how to use data for improving New York,” Hidalgo told Gotham Gazette.
Last year, Mayor de Blasio announced a public-private partnership to launch Computer Science for All, a computer science education program for every city public school student. As a new generation of students learn web design and coding technologies, Hidalgo hopes it will translate to increased interest and participation in civic hacking.
“We fundamentally believe that there is an opportunity to enhance digital literacy through civic education and we would love to see that embedded within the computer science program,” Hidalgo said on behalf of BetaNYC.
One essential question for open data is whether it is being used beyond the Ben Wellingtons and Noel Hidalgos of the world. In other words, are community activists and those without advanced data science training using the city data?
Juan Camilo Osorio, Director of Research at the NYC Environmental Justice Alliance (NYC-EJA), says, like Hidalgo, that a cultural shift is necessary to understand the capacity for individuals to be involved in an open data ecosystem. Osorio wants to challenge the model that community organizations function only for public outreach. “These organizations are part of community-based planning and can do research provided with the right technical and financial resources,” he said.
Osorio thinks the open data portal has great potential to be used by community organizers - not just computer programmers or data analysts.
Moreover, he believes the city has a bigger responsibility beyond publishing information on the portal: “It’s not just about opening access, [in order to use it] you still have to know what is the right agency that would have the data you need and then you will also need to have the basic capacity to process that data. The city should going a few steps further providing the tools and training to learn how to work with that information.”
City Council Member Ben Kallos, a software developer and long-time advocate for government transparency who now chairs the Council’s governmental operations committee, agrees that free trainings should be offered for New Yorkers interested in learning how to use the open data portal. He suggests partnering with the city’s three public library systems (New York Public Library, Brooklyn Public Library and Queens Public Library) to train librarians who can teach patrons to use the open data portal as a research resource. Featuring the open data portal on library websites and other logical online research centers would be helpful in expanding usership and public awareness.
Kallos acknowledged that New Yorkers must have basic access in order to use the open data portal.
“I want to make sure that every low-income New Yorker has access to free and affordable broadband and low-cost computers. That would mean everyone in NYCHA should have free broadband and that anyone who is low-income should have an affordable internet plan,” Kallos told Gotham Gazette. “In order to have a modern government, we need to make sure that everyone can connect.”
Dr. Mashariki, of the Mayor’s Office of Data Analytics, certainly sees the portal as a long-term work in progress. “There’s no day I foresee where we stop and say you know, the portal is perfect and it’s at its most usable and we’ve expanded to it the point where we can’t expand it anymore,” he said. “This is always going to be a strategy that we’re going to have to adapt and adopt as needed to ensure new uses and new capabilities.”
***
by Kaela Sanborn-Hum for Gotham Gazette
@GothamGazette
