As Americans find more and more information a computer click away, advocates across the country are pushing governments at all levels to make their data available to the public. Despite recent strides to improve public access to government data, New York City still lags behind other jurisdictions in terms of how much data it makes available, how frequently and in what forms.

The City Council and Bloomberg administration currently are discussing legislation to create a single portal for city government data. Depending upon what emerges form this debate, New Yorkers may soon have much more information at their fingertips.

The Public's Data

As taxpayers, we basically own government information, and at least in theory, most government information is available to the public. Getting access to it, though, can prove difficult.

According to New York state's Committee on Open Government, almost all records of state and city agencies are available upon request. This would include such data sets as payroll records payroll records, geographic data geographic data, public library locations and hours, civilian police complaints and demographic information by community board. Exceptions exist, though. Records can remain confidential when, for example, disclosing the information would violate someone's privacy rights.

While many agencies post requested data and reports online as a matter of policy, some will only disclose in the face of a Freedom of Information Law, or FOIL request. (For more information on how to file requests, see the Committee on Open Government).

The FOIL process can be time consuming. Agencies are required to respond within five days of a written request but may take more time to complete the request under certain circumstances. At times, the person requesting information has to go to court to get the agency to act.

Many city agencies do not have a good track record for releasing data in a timely manner, according to a recent survey by amNewYork. Often they charge for providing paper records, although if the data is available electronically, the agency must provide it in electronic form and in some cases may be required to convert electronic information to a "more desirable format."

The only way to force an agency to comply with a FOIL request is to take legal action -- something beyond the means of many private citizens and small organizations. The New York Civil Liberties Union often seeks city data and has found it difficult to monitor FOIL requests.

"We like to pay pretty close attention to what we're getting and following up when agencies fail to supply information, but there have been instances where we've been unable to keep up with the myriad of delays and woefully late responses," said Donna Lieberman, executive director of the NYCLU. "We've had more luck than some organizations, though, because they know we'll sue them."

Finding the Data

A lot of data, though, is available without going through the FOIL process. One place to look is the city's website, nyc.gov. Some agencies provide data and information in a consolidated place on their websites, though some scatter information in different locations. The Metropolitan Transportation Authority, for example, has created a single data pagefor developers of mobile and web-based applications that gives access to information and allows users to sign up for updates.

While opening up government data and technological infrastructure to spur private sector growth is not a new concept (some credit Ronald Reagan with the creation of FourSquare through his release of government GPS technology), New York City has created a number of initiatives that opened up of city data. In October 2009 the city launched a competition called Big Apps that gives prizes to developers of software applications that allow the public to interact with city data either on the web, a desktop computer or a mobile handheld device. To help in creating the applications, the administration launched the website DataMine, which includes nearly 400 datasets from city agencies including both raw and geographic data.

The Department of Information Technology and Telecommunications, also known as DoITT – or Do It, coordinates the site, though individual agencies determine the amount and frequency of data released, and participation varies. The Department of Park and Recreation has 28 files of raw data on the site, ranging from calendars of events to directories of barbequing areas and playgrounds. The Department of Transportation contributed 16 data files and the Department of Finance 15 files. The Department of Education, certainly of great interest to many New Yorkers, contributed four data sets: job titles, the 2009 and 2010 school surveys, and school point locations. The police department, which has come under fire for not releasing data, did not participate in the program.

Not all data is updated frequently on DataMine, however, as there is no citywide policy on release of data through a central website.

Other city data, such as 311 information, is available elsewhere online. Local Law 47 of 2005 requires 311 to provide monthly reports to the City Council, the public advocate, community boards and the public. NYCStat, a website of the Mayors Office of Operations, has information regarding agency performance, 311 reports, the city's use of federal stimulus funds and more.

Apps for the Apple

The MTA's release of data and the Big Apps competition have resulted in a number of applications or "apps" for mobile devices and on the internet to help the public interact with city data. The city has official applications, including an app to use iPhones to submit 311 complaints.

Private developers have used government data and information to create other applications. ExitStrategy, which costs $4.99, helps the user find the exit locations at subway stations, while cultureNOW, a free app, allows users to learn more about art pieces in lower Manhattan while standing in front of them, part of "museums without walls" movement.

Government data also helps organizations and individuals keep a watch on government. This is why, for example, the NYCLU filed a number of lawsuits against the New York Police Department to get statistics on stop and frisks. The group found the city's policy on issuing this information inconsistent.

"The stop and frisk data is emblematic," said Lieberman. "On the one hand, the police department was giving access to the electronic database to the RAND Corp. and academic institutions, but was withholding it from the NYCLU. The power of the data was clearly what was motivating the department to make it selectively unavailable."

Advocates and parents also eagerly seek out education data, ranging from data on English language learners to information on graduation rates and failing schools.

A National Movement

New York, though, does not have an overall policy for releasing such information, even though the Bloomberg administration prides itself on being data driven. "All you get is what agencies choose to give you, and not in a timely manner. It makes for lopsided information and flawed public policy," Lieberman said. "For a democracy to thrive, the people have to have information about what our government is doing and the facts with which to hold our leaders accountable ... Without transparency and information, the public can't hold politicians accountable."

Governments across the country are working to develop websites and policies for open data. In 2009, the federal government established data.gov and set a framework for agencies to supply data to the website. Today, the site has 389,829 raw and geospatial datasets, and has spurred the creation of 1,063 government apps and 263 privately developed apps. The most popular dataset on the site is data on earthquakes worldwide, which is updated every seven days.

Data.gov also lists state and local open data sites within the U.S. as well as international sites from other countries. According to the site, there are 29 states and 11 cities that have developed websites devoted to open data. DataMine is included on the list.

At a local level, a number of other major cities have adopted comprehensive data policies. Baltimore and San Francisco have adopted open data policies through executive action, while in Portland the legislature required the release of data by city agencies. The policies generally contain specific deadlines and targets for the release of data, as well as standards for the data itself.

Opening Up New York

In New York, earlier this year, the Mayor's Office of Media and Entertainment released its Roadmap for the Digital City, which "outlines a path to build on New York City's successes and establish it as the world's top-ranked digital city, based on indices of internet access, open government, citizen engagement, and digital industry growth."

While much of the city's past work on open data has focused on developers, the Roadmap for the Digital City looks at the general public. As part of the plan, the city will overhaul its website, nyc.gov, and create an "NYC Platform" for data. While the details have not all been released, DoITT says it will "unveil data visualization tools" to allow the members of the public to create their own charts and tables of city government data.

The City Council also has been engaged in the debate over how to open up city data. In 2009, Councilmember Gale Brewer first introduced open data legislation, now Intro 29 of 2010, which would require the city to provide all public data produced or retained by the city in a centralized online repository. Under the bill, the city also would have to create an open data policy, publish data in formats that allow for automated processing – putting it in Excel, say, instead of a PDF -- and require deadlines for release of data.

Councilmember Dan Garodnick is sponsoring legislation that would require DoITT to publish a listing of city-owned datasets, known as the public data directory (something that the Commission on Public Information and Communication, a defunct agency, is required to do under the City Charter).

Brewer notes that many see the Big Apps competition as focusing on gimmicks, such as finding parking spaces and restaurants. "The Big Apps competition is great for a certain group of people, but there are other New Yorkers who want help with their daily lives, like finding affordable apartments and places to take their kids after school," she said.

She believes that opening up more city databases as proposed in her bill will allow researchers and private social service agencies to use the data to deliver services and address basic needs. "Knowledge of government is so low that people will be shocked and pleasantly surprised by the data -- things will come out that no one could imagine," said Brewer.

Behind the scenes, other groups have urged City Council and the Bloomberg administration to develop an open data policy and announce public targets and tracking of data releases. Civic groups and technology experts advocating for more open government, recently joined together to form the NYC Transparency Working Group. Citizens Union, the sister organization to Gotham Gazette, is a member, along with groups such as the NYCLU, New York Public Interest Research Group, Reinvent Albany and Open Plans. They seek simple advances such as making information posted on city websites downloadable in usable formats like CSV and have been involved in discussions with the City Council and city agencies regarding Intro 29.

Negotiation around the legislation may continue through the summer and early fall before an agreement is reached, with much of the conversation concerning how much data to release and how fast. One concern is making sure that city agencies see the value of making data public and do not feel threatened by its release. "We need to make sure agencies are comfortable doing this -- they are one important audience," said Brewer. "When the world doesn't collapse, more agencies will be comfortable releasing data."

Rachael Fauss is research and policy manager for Citizens Union Foundation, which publishes Gotham Gazette.