Council Member James Vacca (right)

At an oversight hearing on Wednesday of the City Council’s Committee on Technology, good government and civic technology experts praised a pair of bills aimed at updating the city’s 2012 open data law, but raised concerns that the proposals could have unforeseen ramifications.

The city’s current law mandates that city agencies publish non-confidential agency data on a public web portal for all New Yorkers to see and easily access. The open data portal is now home to over 1,600 data sets, with hundreds more data sets scheduled to be released in the coming years. The two “Open Data 2.0” proposals, sponsored by Council Member James Vacca, chair of the committee, and supported by Manhattan Borough President Gale Brewer, are intended to bolster open data reporting standards as well as improve certain areas of the current policy. “Access to government information ensures public institutions can be held accountable,” said Vacca, in his opening statement. “In New York City, the importance of public access to data is reflected in both law and practice.” He noted that New York City had been the first municipality in America to pass a law requiring public data be posted online in machine readable formats.

The hearing was a bittersweet occasion for Vacca, who is term-limited and will be leaving the City Council in December after 12 years. It was his final oversight hearing on open data policy, a project he has long championed.

The city’s open data policy is seen as a model not just for the United States, but the entire world, Brewer noted in her opening statement at the hearing. “I just met with elected officials in Spain... And they are looking at our law,” Brewer said. To illustrate one of the uses of open data, Brewer will be joining civic technology group BetaNYC to host a “311 Data Jam” on Saturday, employing the city’s dataset on the 311 helpline. It’s the third such event hosted by BetaNYC, and they will be introducing “BoardStat,” a tool for community boards to analyze visualized 311 data in order to discover community trends.

The bulk of Wednesday’s hearing covered Intro. 1707, which includes provisions requiring reviews of the “Technical Standards Manual,” which is the guideline for how agencies manage and present their datasets every two years, to ensure the datasets remain up to date.

“These standards are designed to make open data more usable to the maximum number of users,” said Albert Webber, director of Open Data for the Department of Information Technology and Telecommunications (DoITT). “We see the TSM as a living document, meant to keep up with advances in technology, data availability, and the passage of local laws that impact data disclosure. An official periodic review of this document would ensure that open data disclosure stays up-to-date, relevant, and true to the most current practices.”

The bill codifies a requirement for city agencies to appoint an “open data coordinator” to ensure agency compliance with the open data law. It also changes the deadline for agencies’ compliance plans from July to September, to avoid conflicting with the city’s budget season. The bill would also require that data analytics on public usage of the data portal be collected and studied.

Criticisms of the legislation stemmed not from opposition to any one plank, but rather from unintended consequences that certain provisions of Intro. 1707 could have in areas such as agency reporting, accountability, and protection of individual privacy.

One provision in the proposal raised eyebrows among good government groups and civic technologists. Noel Hidalgo, executive director of Beta NYC, and John Kaehny, executive director of Reinvent Albany, pointed out that the bill extends the open data compliance deadline for city agencies from 2018 to 2021, a full nine years after the law went into effect in 2012. Many datasets from numerous city agencies are currently scheduled for release on the current deadline, December 31, 2018.

Kaehny said allowing agencies to hold off on publishing data is akin to a high school student saying they’ll turn in all of their homework on the last day, a promise that can’t be seen as reliable. He made similar comments to Gotham Gazette in July after the city released its annual update to the open data plan. Kaehny referred to the extension to 2021 as a “homework extension” that will never be turned in.

Vacca pressed Webber on these concerns, and Webber insisted that the administration was committed to releasing the data sets on time. “Putting a deadline is always helpful,” he told the Council member. “But no, we do not have concerns that pushing it to 2021 will delay the release of some datasets.” In his testimony, Webber stated that the administration would “like to discuss how to improve the language to ensure that submissions meant to be disclosed by December 31, 2018 will still be held to that deadline.”

Another aspect of Intro. 1707 that received some critique was a provision calling for the collection of data analytics on visitors to the portal. Concerns were raised that recording data on unique visitors to the portal, especially those exploring data that could be construed as sensitive, might violate an individual’s privacy. The language of this provision in the bill was viewed as vague.

“Pageviews, great,” said Sumana Harihareswara, a programmer from Astoria who testified at the hearing. “Numbers of pageviews, great. Unique users? I don’t want to be identified uniquely in a public place for having looked up a certain piece of data. So number of unique users is perhaps what this ought to say.”

Those who testified otherwise generally praised the proposed biannual review of the TSM, the change in the annual compliance deadline from July to September, and codifying into law the agency requirement of an open data coordinator.

The bill also saw some critique for what it lacked, such as a “private right of action,” the ability of the public to sue agencies failing to perform their duties. Kaehny noted that the only course of action without the private right of action is to “name and shame” agencies considered to be “laggard” in regards to their dataset releases. According to Kaehny, open data is one of the few pieces of legislation in New York City that does not include a private right of action, making it highly unusual, especially for such a major city law.

Brewer echoed the call for a private right of action, emphasizing that future administrations may not be as committed to transparency. “I believe we must provide a private right-of-action to protect the New York City municipal open data apparatus from a future administration that does not wish to operate it in good faith as the current and previous administrations have done,” she said. She cited the Trump administration’s moves immediately post-inauguration to delete data that had been posted online under the previous administration; there is no federal law that establishes similar data protections as New York City’s policy.

Kaehny claimed that even naming and shaming is a difficult task, given that agency compliance with the law is not compiled in a single spreadsheet on the portal but rather in four separate spreadsheets. Compiling all relevant information about published and unpublished datasets into one spreadsheet was one of Reinvent Albany’s key recommendations to the committee. “What we’re seeing is fog when it comes to understanding exactly how well agencies are doing,” Kaehny said.

The administration also had suggestions for improving the bill. Echoing testimony from the advocates, DoITT’s Webber expressed dismay at the law’s lack of any licensing requirements for the data, hoping that “permissive licenses,” licenses with few, if any, redistributive restrictions, could be added to the program.

“Our mission to encourage public engagement with open data in the long term makes the ability to license very important,” Webber said. “Structured properly, licenses could allow us to formally make the city’s datasets free and open for public use in perpetuity, codifying our existing practice and ensuring that New Yorkers have access to open data for the long haul.”

Also under consideration at the hearing was Intro. 1528, an amendment to an existing open data law that would require agencies that process Freedom of Information Law requests involving non-confidential data to report and publish the names of the public datasets, rather than simply the metrics of the datasets. The measure received overall support from those testifying before the committee.