(maps.nyc.gov/crime)
With the advent of fantasy football, at the click of a button you can find out how many times Eli Manning has thrown for a touchdown against the Dallas Cowboys in each game of his entire career (spoiler: a lot). This data could help you decide if it was a good idea to start him or another quarterback to ensure a win in your fantasy league on Sunday.
While using data in sports can be fun, what about using big data to help citizens make important decisions? Would you move into a new building if you knew crime had been rising for months in the surrounding area? Would you send your child to a school if performance indicators showed it wasn't getting results? Would you let your kids bike down a certain road if you knew there had been multiple crashes on it?
For decades, government has been keeping data on crime, schools, and traffic safety (and much more), but only recently has there been a push to open that data to the public. On March 7, 2012 then-Mayor Michael Bloomberg signed Local Law 11, also known as the "Open Data Law," which requires the City to make its data available to citizen access.
It has been more than two years since the passage of the bill. New York City is not quite up to the speed of fantasy football - but it's getting there.
On July 15 of this year, the Department of Information Technology and Telecommunications (DoITT) released its first update to the City's open data plan under Mayor Bill de Blasio. It shows what datasets each city agency has actually added to the open data portal versus what they were scheduled to add, and updates the timeline for the next four years worth of releases. The report, required by law annually, was sent to the Mayor's Office and the City Council on time and details that there are now 1277 open City data sets.
"With the Open Data Plan, the public has an important accountability tool to measure the project's success, and provide feedback on what data is most useful and important for release," said Rachael Fauss, Director of Public Policy for Citizens Union, by email July 15.
The original plan for the open data portal, released in September of 2013, called for 94 data sets to be released that year. The updated plan notes only 61 datasets were put out. Officials with DoITT said that as they have gone through the data, some sets originally slated to be open could not become so because they were actually narrative reports. In other cases, data was deemed unable to be released because it would violate privacy laws. In total, 27 datasets were scrubbed from the original plan.
The updated plan provides for a more even release of datasets between now and the end of 2018, when the plan will be complete and is set to bring the total number of published datasets to over 1400:
With more data flowing to the portal, eyes are beginning to turn to how the data can and should be used. Anyone can grab data off the portal and use it to create an application, a map, or any other presentation or visual representation. In some instances it can be used to create community awareness, such as crash data put out by the NYPD as a part of Vision Zero.
In other cases New Yorkers can use City data to create successful businesses, such as SiteCompli. Founded in 2009 by Jason Griffith and Ross Goldenberg, SiteCompli is a service that helps owners and managers of New York City real estate stay on top of their compliance with local laws. SiteCompli clients include First Service Residential, Douglas Elliman Property Management, Tishman Speyer, and Brookfield Properties, companies serving over 1 billion square feet of space and over 500,000 apartment units across the city.
The company started prior to the City's open data being published, but now at least some of the data SiteCompli relies on is on the open portal, preventing the company from having to find it via other avenues. As more data comes online, SiteCompli should have to rely less on multiple agency websites or physically going to certain departments for data and more on the open data portal.
In an email, Goldenberg said there is a lot of data the real estate community would be particularly interested in that is not currently published. But, he added, simply publishing more data is only half the picture.
"We encourage City stakeholders to think carefully about the quality of the data and frequency of updates, not just the quantity," Goldenberg said. "For the data to be useful, it's crucially important that it is as accurate and as recent as possible."
Officials at DoITT said they are working to automate the updating of more datasets to prevent creating a backlog in the future. To date, 95 datasets have been automated.
In the recently adopted fiscal year 2015 budget, DoITT was provided an additional $515,000 in funding for five new positions: three developers (Senior Integration Developer, Senior Java Developer, Java Developer); a business intelligence analyst; and a technical project manager. These positions will expand the team, help with the quality of the data, and allow more automation, a labor intensive process.
"The city's completion of the Open Data Plan and provision of funding for five full-time staff will help ensure its future success," Fauss said. "This kind of commitment is crucial at this juncture as the effort is entering its third year with a new administration."
Manhattan Borough President Gale Brewer, who championed the passage of the bill while serving in the City Council, said during a phone interview that she is excited to see the funding stream in the budget. She said it shows a dedication by the administration and will allow for those hired to focus on the open data portal.
"Two years down the road, I am happy, I am pleased. But the site needs work," Brewer said by phone July 14. "That is what this funding stream and the permanent staff will be able to focus on."
She added, "You don't want to do all this work and then not have people use it in a way that makes sense."
The searchability function of the open data site, built by software company Socrata, has been an issue since the portal's inception, a problem DoITT is aware of and working on.
"The search functionality of the portal can be improved, and we're committed to addressing that as we work to improve the overall user experience," DoITT spokesperson Nick Sbordone said via email July 15. "We want this data to be as accessible as it can."
Advocates are hopeful. "We understand the complexity and novelty of the City's open data efforts, and we appreciate the tremendous amount of hard work that city officials have put into making the law a success," reads a statement from the New York City Transparency Working Group. "The next major step for NYC's open data program is to fix vexing usability problems with the open data portal, including significant issues with searching and finding data. These issues have hindered public use, and when overcome, will put the Open Data Law on the road to becoming an extraordinary success."
***
by Kristen Meriwether, Gotham Gazette
@MeriwetherK
***
Note: Gotham Gazette is an independent publication of Citizens Union Foundation, the sister organization of Citizens Union
