Enforcing Compliance, Improving Policy

The original Open Data Law required City agencies to self-submit compliance plans laying out a timeline for publishing public datasets. Last year, this requirement was updated after a package of amendments to the Open Data Law was passed; agency compliance plans now include additional legal mandates regarding Freedom of Information Law requests, response time for data requests, and update schedules (which ensure data doesn’t go stale). Together, these statutory measures form a framework for locating data that is eligible for publication on the Open Data Portal by the end of 2018, as required by the Open Data Law.

In addition to strengthening New York City’s already globally leading open data policy, the updated legislation requires that data from all agencies — not just select high performers — is accessible, understandable, and usable for everyone. Ensuring the City’s compliance with the Open Data laws and policy is the foundation of a strong and just Open Data for All. The following legal mandates comprise this framework.

Data Retention and Archiving

Local Law 106 of 2015 mandates the preservation of the New York City historical record as represented by the City’s official data, meaning that no records may be permanently removed from the Portal. This requirement helps ensure that while data is kept up-to-date, historical information is properly retained and archived for analysis at a later date. For example, the Department of Sanitation’s PlowNYC map allows New Yorkers to see where and when their streets are cleared during a snowstorm, in close to real-time (data is updated every 15 minutes during storms). The web map, however, is only active during snow events; when a winter storm ends, the web display is cleared of information. Since last November, all PlowNYC Data has been captured on the Open Data Portal. Now, all records are retained, meaning that users can analyze plow data long after the snow has melted.

Metadata Standards

Many datasets released on the Open Data Portal are raw data, and may never have surfaced outside of an agency. Local Law 107 of 2015 requires every dataset to have a “data dictionary,” written in accessible language. Data dictionaries help define the different columns of data and describe how the data was collected. In December, the Open Data team hosted a session with Reinvent Albany and a group of high-frequency Open Data users to collaborate on a new data dictionary template, which agencies can use to provide the most useful contextual information. Many agencies are now using this template, including the Department of Health and Mental Hygiene, which recently published neighborhood air quality data from the New York City Community Air Survey. In addition, thanks to a product-wide update from the City’s vendor Socrata, each dataset now has a “primer,” a unique landing page with information imported from data dictionaries to guide users before they dive into the data.

Geospatial Standards

Local Law 108 of 2015 requires that every dataset containing street addresses follow a standard format containing geospatial coordinates and political boundaries. These new standards ensure that every agency is writing addresses in the same way, making it easier for users to make maps with City data. In September, the Open Data team invited the public to contribute ideas for these address standards; guided by this feedback, the new standards require agencies to include geospatial fields most frequently captured by City agencies, information that is in highest demand from public users, as well as attributes that will have the biggest impact on citywide operations once they are standardized. This spring, the City Council published its Participatory Budgeting Dataset, which fully meets the standards and is easily indexed with other geospatial datasets. This dataset includes details on all projects funded through Participatory Budgeting, a democratic process in which Council Members choose to allocate at least $1 million from their budget through an open, community-wide decision-making process; residents propose projects, and then vote on which to fund. The Participatory Budgeting Dataset helps New Yorkers track those projects as they are implemented. Additionally, the Open Data team also released a tracker to check whether eligible datasets have been “geocoded” — the technical process of matching coordinates to addresses — to meet the standards.

Public Requests for Data

Local Law 109 of 2015 guarantees timely and thorough responses to all public requests for new datasets. This means that when users make requests for new data, Open Data Coordinators at the relevant City agencies assess those requests to determine whether the data can be made available to the public. The Department of Buildings (DOB) is one of the most popular data publishers in the City: DOB’s Job Applications Filings dataset is the most accessed on Open Data. Many users have requested access to DOB’s Certificate of Occupancy dataset, which details the legal use and occupancy of new buildings. The Agency made the release of this dataset a priority, and it was published earlier this summer. Additionally, through DOB NOW – the Agency’s new digital system that modernizes its paper-based processes – DOB continues to be responsive to the public’s demand for data by publishing eligible datasets on the Open Data Portal. Across this and all City agencies, the status of new dataset requests can also be tracked on the Open Data website.

Keeping Data Fresh

Local Law 110 of 2015 requires all data published on agency websites to be included and kept up-to-date on the Open Data Portal. This ensures that all published data is centrally discoverable and can be analyzed. Earlier this month, the Department of Transportation (DOT) published its real-time traffic speed dataset, which contains records on traffic speed collected from the DOT’s traffic speed detectors. While these records were previously available in XML data format on the DOT website, they are newly available in an easy-to-use tabular format and viewable on the Open Data Portal. This automated dataset is updated every five minutes — making it the Open Data team’s closest to real-time updates.

FOIL Responses and New Data Releases

LLocal Law 7 of 2016 requires agencies to review Freedom of Information Law requests containing data to determine whether they contained new public datasets that could be published on Open Data. This is part of a broader effort to tie the way agencies respond to these requests more closely to the way they publish datasets. To this end, this year the Open Data team required that Open Data Coordinators work directly with their agency records officers and general counsels to assemble their dataset inventories and compliance reports. Open Data is successful only when it becomes a common language at an agency—from its analytics teams to its legal division to its commissioner’s office.

Examination and Verification

Local Law 8 of 2016 requires MODA to examine three mayoral agencies each year to verify that all public datasets have been disclosed in their publishing plans. Last year, MODA conducted the process with Department of Sanitation, Department of Correction, and Department of Housing Preservation and Development. MODA found all three agencies to be fully compliant with the requirements laid out by the Open Data Law, and also issued recommendations on how to improve citywide compliance with the Open Data Law. These included the recommendation that agencies proactively look for ways to publish newly collected data (that is not subject to privacy restrictions) as Open Data. For instance, the contract procuring CitiBike’s data infrastructure included a provision for making the data collected public; because this Open Data requirement was built into the technology itself, the ride data published by CitiBike is easy to access, easy to use, and in high demand. The process is being improved for this year’s examination and verification of the Department of Buildings, the Department of Environmental Protection, and the Fire Department.

Next:
6. Compliance Plan