NYC Taxi (nyc.gov)
NEW YORK—It all started with a tweet. "It's #metricmonday again! Today's #infographic shows data on yellow taxis on the road by day and time. #nyctaxi," tweeted the Taxi and Limousine Commission (TLC) from @nyctaxi on March 10 at 12:18 p.m.
The TLC had no idea that its seemingly innocent tweet would lead to an unintended revelation of medallion and license data, bringing into question how city agencies handle the FOIL process, especially for large open data requests.
Chris Whong, open data advocate and co-founder of BetaNYC, saw the tweet and tweeted back, "is the data available?" Whong got a response from TLC shorty after saying the TLC trip records were available via Freedom of Information Law (FOIL) request and included a link to submit the request.
Whong filed the FOIL request by email and received a response within hours. The email informed him that to receive the data he would need to bring an unopened 200 GB external hard drive to the TLC offices where officials would provide the data.
Within a few days time, Whong made the hard drive purchase, dropped it off to the TLC, and picked up the data the following morning.
"Overall, I have to say I was impressed with the TLC's responsiveness, professionalism, and the fact that they allow email correspondence for this sort of thing in the first place." Whong posted on his blog, "Despite the annoyance of two in-person trips and the expense of a brand new hard drive, I was able to go from viewing the FOIL information page to possessing the data in just 2 business days."
The fare and trip data, totaling nearly 20 GB, includes full 2013 info for the following: medallion and hack license numbers, pick up dates and times, trip times in seconds, and the latitude and longitude coordinates for pick-up and drop-off locations. Whong posted screen shots of some of the data and promised a graphic visualization of the data soon.
Within days, civic hackers began to comment on Whong's blog, asking him to post the data to a public space so they could also hack the data. On June 16, Whong posted two files available for download. Two days later, Andrés Monroy, a contact Whong met through social media, offered to host smaller chunks of the data to make downloading easier.
Word spread quickly through the civic hacking community that there was a new data set available and it became a hot commodity, with excitement over what seemed an open data victory.
But then things got interesting.
Vijay Pandurangan, CEO of Mitro and an internet security expert, saw the buzz on Twitter about the taxi data. He noticed a comment on Reddit that questioned how the same driver was making so many trips.
Pandurangan looked at the data and saw the medallion numbers listed in the datasets were not the actual medallion numbers. The TLC used an algorithm to encrypt the numbers, a process known as hashing. Each taxi was given a unique identifier so the data could be analyzed, but the idea was that the medallion number, and thus the driver, would not be identifiable. Pandurangan recognized the type of hash the TLC used (MD5) and spent a little bit of time researching it.
"I realized if you use a standard algorithm for generating a hash, it is the value for zero," Pandurangan said by phone Monday, June 23. "When I saw that, I knew they were probably hashing the license numbers directly which meant you could very easily go backwards and find all of the license numbers from the data, even though they didn't want you to be able to do that."
Taxi license numbers are six or seven digit numbers starting with the number 5, with a total of roughly two million possible numbers. The medallion numbers have three structured patterns: one number, one letter, two numbers or two letters three number or three letters, three numbers. All said, there are roughly 24 million possible numbers for licenses and medallions.
Because the numbers are so structured, Pandurangan said he was able to, in laymen's terms, 'crack the code' within a few hours of looking at the data. He could take the taxi license numbers and look them up online, thus identifying the driver.
"They took the taxi cab number directly but because there is a lot of structure in...the medallion numbers...you could compute this function for every single number," Pandurangan said.
Pandurangan wrote a blog post on June 21 documenting his experience, a post which went viral in the tech community. Motherboard, a Vice tech blog, posted a follow-up story, as did Ars Technica.
Over the phone, Pandurangan said TLC had good intentions by trying to anonymize the data, but he wasn't sure if proper protocol was followed.
The TLC acknowledged issues with the process and acknowledges changes will be made moving forward. "We will take a look at how the anonymization was compromised, and make whatever improvements we see as being necessary to the process," TLC spokesman Allan Fromberg emailed on Monday.
When asked if the FOIL request was reviewed by the legal department before the data was handed over, Fromberg did not answer.
Pandurangan said the process is very complicated and with the powerful computers today, it is much easier to work backwards through millions of possible combinations of numbers. As a simple solution, he suggested the TLC not use original medallion or license numbers the next time they release data.
In a city that is just beginning to open up more of its data, this is a valuable lesson, but one that does not have to be a deterrent to continued open data expansion. Mark Headd, former Chief Data Officer for the City of Philadelphia, wrote in a blog post that the incident shows an example of how FOIL requests for large data sets should be treated more like open data requests with an opportunity for review and feedback, not just a request to be checked off as filled.
"Because FOIA is viewed as a one and done task, there is no opportunity to iteratively release data – if the release of NYC taxi trip data had been viewed as a process (particularly a collaborative one), the Taxi & Limousine Commission could have opted to be conservative in their initial release and then enhanced future releases based on actual feedback from real consumers of the data," Headd wrote.
Pandurangan argues this instance should be used as a lesson to learn from, not a reason to keep data closed.
"I hope it doesn't prompt them to not release data," he said of the City. 'It allows people to build all kinds of interesting maps and tools and to figure out how the city works. I think it is valuable to do that."
***
by Kristen Meriwether, Gotham Gazette
@MeriwetherK
