Mayor de Blasio at Thursday's bill signing (photo: @CitizensUnionNY)
Since 1873 the City Record has been providing a snapshot of daily city business. The Record includes schedules of public hearings, tracks procurement notices, lists sales or purchases of City-owned land, and keeps tabs on City employee salary changes.
It is a wealth of information, but it wasn't until 2011 when the City began to publish the daily versions online in PDF form. Reading it daily became easier, but searching was difficult. Readers have been able to search each individual PDF for terms, but had to know the date a desired item was published to open to the right document. Only procurement notices have had a separate search function.
On Thursday, August 7th, Mayor Bill de Blasio signed new legislation that will start the process of creating a new, data-friendly online portal for the City Record. The bill, introduced by Council Member Ben Kallos, requires the City Record be published in a machine-readable format and be fully searchable. In addition, the administration will, for the first time, formally partner with the civic tech community to ensure the backlog of City Records are in the same format.
"Now that the City Record is going online that public information will be truly public and easy for residents to access," Kallos said at the bill signing August 7. "This was a document that was previously only for insiders, now it is for every New Yorker."
De Blasio said the legislation was a way to "shine a little more sunlight on the inner workings of city government."
By requiring that the data is published in a new format, the legislation brings the outdated City Record Online portal into the 21st century. Going forward, it should be a much more desirable front-end experience for heavy users like journalists, those in real estate and the variety of other industries interested in the City's procurement process, as well as curious citizens. But it also offers the City a chance to benefit greatly from its own historical data.
There are roughly 4,000 digital copies of the City Record dating back to January 1998 currently stored on CDs in file cabinets. Those rich datasets, which could help the City better understand its day-to-day operations, is now doing nothing more than collecting dust.
"A smart city should be using this information to learn and be more efficient," Jeff Merritt, senior advisor to Mayor de Blasio said by phone on Wednesday.
To make this potential into reality, the City has entered into a public-private partnership with leaders in the city's civic technology community including BetaNYC, Socrata (which built the Open Data Portal), Sunlight Foundation, Ontodia, and DevBootcamp to enlist members' help opening up the old records. The effort is also being supported by the good government group Citizens Union, which advocated for and applauded this transparency effort by the de Blasio administration.
While communicating with the City's Department of Information Technology and Telecommunications (DoITT), these civic-tech organizations will build digital scrapers to pull the data from the PDFs and put it into machine-readable format that can be sorted and analyzed.
Noel Hidalgo, co-founder and executive director of BetaNYC, said the organizations were chosen due in large part to their expertise with scraping data. Over the coming weeks the group will devise a game plan to tackle the project, but did not have a projected deadline.
"It will be a challenge, but in some ways it is a rewarding challenge because if you can unlock this [data], we can learn a lot from it," Merritt said. "I think in a lot of ways we can learn a lot about our inefficiencies."
Merritt said one of the areas of interest is looking at procurements and expenses to understand the benefits and/or drawbacks of different types of requests for proposals (RFPs). As the fallout from both the CityTime and the new 911 system contracts are still being felt, historical data could be used to improve procurement processes.
Merritt did not have a timetable for the project either, but said as the data becomes available, it can be put on the open data portal. Within a year, when the new portal is built, the archived data will be uploaded and will be searchable alongside the data from new editions of the City Record.
"Hopefully we can put the government in the people's pocket," Hidalgo said.
Teaming Up With The Civic Tech Community
Thursday's announcement marks the first time Mayor de Blasio has publicly teamed up with the civic hacking community. While he has expressed his support for the tech community in several speeches, de Blasio is now utilizing the community's talent to help solve the City's problems.
De Blasio has only been in office for seven months and his time has been occupied with pre-k, the budget, negotiating union contracts, and other major initiatives. There has, however, been work happening on the tech front behind the scenes.
In one example, a new government portal for Department of Records and Information Services (DORIS) is currently in the works which would make accessing records online easier and in an open-data format. The project will use open source software and is on GitHub to allow developers to both use the system and offer suggestions for improvements.
A month ago, DORIS Commissioner Pauline Toole invited members of the New York City Transparency Working Group, which includes many of the same technologists and advocates helping with the City Record project, into City Hall to get feedback on the yet-to-be-released DORIS portal. Merritt said that feedback will be used to improve the platform before it is released next month.
The City Record project marks the first time members of the civic technology community are being embraced by this administration not just as technical advisors, but as a formal partners.
"With previous administration we were wearing the hats of agitators and activist and at times be allowed to collaborate around legislation," Hidalgo said after the bill signing August 7. "With these two bills, this is the first time that we are seen as collaborators."
This collaboration will provide a test case for de Blasio's direct interaction with the civic technology community, one Hidalgo feels is perfectly matched.
"I think this is one of the best test situations we could have teed up because it really strikes at the core of our values within the civic technology community and the values of a progressive mayoral administration," Hidalgo said.
***
by Kristen Meriwether, Gotham Gazette
@MeriwetherK
***
Note: Gotham Gazette is an independent publication of Citizens Union's sister organization, Citizens Union Foundation
Correction: this article has been amended to reflect that DORIS Commissioner Pauline Toole invited advocates to City Hall, not Mr. Merritt as previously stated.
