Office of Management and Budget Integrated Data Collection (IDC) Requirements

Office of Management and Budget Integrated Data Collection (IDC) Requirements

May 2016 IDC Requirements for Open Data Progress, Use, and Impact

Status: Updated

  • Agencies will continue to be assessed on their Open Data Policy progress. The Reports Metrics Dashboard can be found here: https://catalog.data.gov/report/metrics-dashboard.
  • Agencies will continue to provide a public version of their Enterprise Data Inventory at [agency].gov/data.json and include their data.json dataset with the title “Enterprise Data Inventory.” If necessary, agencies will coordinate with their Freedom of Information Act (FOIA) offices to redact specific metadata text for “non-public” datasets only if the text is subject to a FOIA exemption. Agencies shall use the instructions provided by OMB at https://project-open-data.cio.gov/redactions/ to conduct these redactions. If redactions are present, Agencies must submit a non-redacted Enterprise Data Inventory JSON file to OMB in the appropriate agency folder here: https://community.max.gov/x/8YamK.
  • Agencies will ensure that their customer engagement process includes a transparent two-way feedback mechanism. A link to an e-mail address or form by itself is not sufficient. For examples of sufficient tools, please see https://project-open-data.cio.gov/engagement.
  • Agencies will continue to enrich their metadata by:

Question on the Enterprise Data Inventory

To what extent is your agency’s Enterprise Data Inventory (EDI) complete? Select one of the following options:

  • To a very little extent (<25%)
  • To some extent (25-50%)
  • To a great extent (50-75%)
  • To a very great extent (>75%)
  • Complete (100%)

Completeness Steps What steps have you taken to ensure your Enterprise Data Inventory is complete?

Questions on Public Engagement

Public Feedback Mechanisms Describe your two-way public feedback mechanisms.

Feedback Received Provide five examples from this quarter of feedback received and what action your agency took in response (e.g. data quality improvements, access to previously unavailable data, or additional data formats).

Engagement Efforts Describe any examples of outreach efforts from this quarter (e.g. open data roundtables, hackathons, datapoolzas, etc.) beyond standard agency communication and engagement processes (i.e. blogs, social media).

Questions on Use and Impact

Internal Efficiency Describe how you are using your agency’s enterprise data inventory to improve internal efficiency or service delivery for your agency. Provide examples illustrating usage.

Data Governance Describe data governance within your agency (i.e. the policies, procedures, structure, roles and responsibilities which outline and enforce rules of engagement, decision rights, and accountabilities for the effective management of information assets). Provide links to any documents or materials.

Interagency Data Sharing Describe interagency data sharing processes within your agency. (Do you centrally track interagency data sharing requests? If so can you provide the total number of interagency data sharing agreements?)

Public Usage (Optional) Provide any highlights from this quarter in which the public used open data from your agency.

Feedback (Optional) Is there any additional information on your Enterprise Data Inventory, Public Data Listing, Public, Privacy and Security, Human Capital, and Use & Impact progress that you would like to share with OFCIO?

February 2016 IDC Requirements for Open Data Progress, Use, and Impact

Status: Updated

  • Agencies will continue to be assessed on their Open Data Policy progress. The Report Metrics Dashboard is available at: https://catalog.data.gov/report/metrics-dashboard.
  • Agencies will continue to provide a public version of their Enterprise Data Inventory at [agency].gov/data.json and include their data.json dataset with the title “Enterprise Data Inventory.” If necessary, agencies will coordinate with their Freedom of Information Act (FOIA) offices to redact specific metadata text for “non-public” datasets only if the text is subject to a FOIA exemption. Agencies shall use the instructions provided by OMB at https://project-opendata.cio.gov/redactions/ to conduct these redactions. If redactions are present, Agencies must submit a non-redacted Enterprise Data Inventory JSON file to OMB in the appropriate agency folder here: https://community.max.gov/x/8YamK.
  • Agencies will ensure that their customer engagement process includes a transparent two-way feedback mechanism. A link to an e-mail address or form by itself is not sufficient. For examples of sufficient tools, please see https://project-open-data.cio.gov/engagement.
  • Agencies will continue to enrich their metadata by:
    • Ensuring all public domain and open license information is properly documented. See more details at: https://project-open-data.cio.gov/open-licenses/.
    • Ensuring all public, restricted public, and non-public APIs are properly documented. See more details at https://project-open-data.cio.gov/v1.1/api/.
    • Ensuring all appropriate data assets are grouped as collections. See more details at https://project-open-data.cio.gov/v1.1/collections/.

Questions on Use and Impact

Public Datasets Ranking Identify at least 3 datasets that are valuable to the public and rank them in order of value. Explain why these datasets are valuable and how they are being used (or how they might be used) by the public.

Internal Datasets Ranking Identify at least 3 datasets that are valuable to your own agency (or among other agencies) and rank them in order of value. Explain why these datasets are valuable and how they are being used (or how they might be used) by your agency or other agencies.

Question on the Enterprise Data Inventory

To what extent is your agency’s Enterprise Data Inventory (EDI) complete? Select one of the following options:

  • To a very great extent (>75%)
  • To a great extent (50-75%)
  • To some extent (25-50%)
  • To a very little extent (<25%)

Completeness Steps What steps have you taken to ensure your Enterprise Data Inventory is complete?

Questions on Public Engagement

Public Feedback Mechanisms Describe your two-way public feedback mechanisms.

Feedback Received Provide five examples from this quarter of feedback received and what action your agency took in response (e.g. data quality improvements, access to previously unavailable data, or additional data formats).

Agency Points of Contact (relevant section)

Agencies need to report and update the contact person for the following responsibilities

  • Primary Open Data Contact (ODP)
  • ODP: Communicating Strategic Value
  • ODP: Public Data Release
  • ODP: Private/Public Applications and Services
  • ODP: Scaling Best Practices
  • ODP: Data Privacy and Confidentiality
  • ODP: Security Risk Assessment
  • Chief Data Officer (If applicable)

November 2015 IDC Requirements for Open Data Progress, Use, and Impact

Status: Same as August 2015 IDC Requirements

August 2015 IDC Requirements for Open Data Progress, Use, and Impact -

Status: Updated (bold indicates updated requirements)

  • Agencies will continue to be assessed on their Enterprise Data Inventory (EDI), Public Data Listing (PDL), Customer Feedback Process, and Data Publication Process. Additional information is available here: https://community.max.gov/x/tBWSK.
  • OMB greatly appreciates agencies’ cooperation in responding to OMB’s guidance to publish their EDI, with FOIA exemption-related redactions as necessary, as a downloadable dataset in their PDL. We believe this further makes the U.S. the most transparent government in the world. To further enhance this transparency, effective May 31, 2015, agencies are required to do the following:
    • Include all “non-public” data assets in their PDL, in addition to the “public” and “restricted public” data assets that have long been required. If necessary, agencies shall coordinate with their Freedom of Information Act (FOIA) offices to redact specific metadata text only if the text is subject to a FOIA exemption. Agencies shall use the instructions provided by OMB at https://project-open-data.cio.gov/redactions/ to conduct these redactions. Essentially, this reframes the PDL to be a public version of the full EDI. Note: Because the only difference between an agency’s full EDI and the agency’s PDL will be the existence of any needed redactions, agencies no longer need to submit an EDI to OMB unless their PDL contains any redactions. Non-redacted EDIs, if applicable, should be submitted to the appropriate agency folder here: https://community.max.gov/x/8YamK.
    • If a metadata record in an agency’s PDL contains one or more redactions pursuant to a FOIA exemption, the agency shall explain the need for the redaction in the “rights” metadata field. This is in addition to the existing requirement to use the “rights” field to explain the non-public access level. Please see https://project-open-data.cio.gov/redactions/ for instructions and examples. These explanations should not themselves contain redactions.
  • Agencies are already required by the Open Data Policy to have a process to engage with customers to help facilitate and prioritize data release. Each agency shall ensure that their customer engagement process includes a transparent two-way feedback mechanism. A link to an e-mail address is not sufficient. (https://catalog.data.gov/report/metrics-dashboard), as denoted with a green star. Data.gov is developing a “Help Desk” tool that will help fulfill this requirement. As Held Desk development progresses, updates will be provided on the Open Data Listserv.
  • To ensure the legal reuse of federal open data is conspicuous agencies must include an explanation for all datasets that do not include a Public Domain URL in the license field by filling out the rights field. See license and non-license (i.e. Public Domain) URL examples and more details at: https://project-open-data.cio.gov/open-licenses/
  • Each agency shall ensure their EDI and PDL contains all of the agency’s public, restricted public, and non-public APIs. Agencies shall use the guidance provided at http://18f.github.io/API-All-the-X/pages/apis_in_data_catalogs (with more details at https://project-open-data.cio.gov/v1.1/api/) to help ensure their APIs are included in these catalogs and that each API is identifiable by the use of the format field set to “API” in the metadata record’s distribution object.
  • Each agency shall enrich their EDI and PDL by ensuring all data assets include the individual datasets within by using the identifier and isPartof fields. See examples and more details at: https://project-open-data.cio.gov/v1.1/collections/

Questions on Use and Impact

Provide at least 5 quarterly examples of open maturity based on public feedback, and include the following information.

Requirement Description
Dataset Name The name of the dataset
URL The URL for the dataset
Date of Feedback Date public feedback on dataset was received
Public Feedback Received What public feedback was received? Public feedback examples include: FOIA requests, two-way public feedback mechanism requests, Data.gov Help Desk requests, emails, news reports, academic papers, and citizen or advocacy blogposts.
Improvements Made What improvements were made to the dataset to make it more open? Examples include: machine-readable formats, real-time delivery, worldwide public domain dedication, bulk downloads, free access to previously charged for data, public access to previously Restricted Public data, and data quality improvements.
Provide quarterly summary information from your two-way public feedback mechanisms. Provide qualitative responses to each of the four questions: 1. What are some of the primary uses of your agency’s data? 2. What have been the primary channels for the users to learn about your data? (e.g. email, agency-sponsored events such as datapaloozas and hackathons, non-agency sponsored events, etc.) 3. What are suggestions from key users on improving the usability of your data? 4. What are suggestions from key uses on additional agency data resources to release?

Question on the Enterprise Data Inventory

Question Options
To what extent is your agency’s Enterprise Data Inventory (EDI) complete? Select one of the following options:To a very great extent, To a great extent, To some extent, To a very little extent