DCAT-US Schema v1.1 (Project Open Data Metadata Schema)

Introduction

This section contains guidance to support the use of the Project Open Data metadata to list agency datasets and application programming interfaces (APIs) as hosted at agency.gov/data. Additional technical information about the schema can be found on the Metadata Resources page.

Standard Metadata Vocabulary

Metadata is structured information that describes, explains, locates, or otherwise makes it easier to retrieve, use, or manage an information resource (NISO 2004, ISBN: 1-880124-62-9). The challenge is to define and name standard metadata fields so that a data consumer has sufficient information to process and understand the described data. The more information that can be conveyed in a standardized regular format, the more valuable data becomes. Metadata can range from basic to advanced, from allowing one to discover the mere fact that a certain data asset exists and is about a general subject all the way to providing detailed information documenting the structure, processing history, quality, relationships, and other properties of a dataset. Making metadata machine readable greatly increases its utility, but requires more detailed standardization, defining not only field names, but also how information is encoded in the metadata fields.

Establishing a common vocabulary is the key to communication. The metadata schema specified in this memorandum is based on DCAT, a hierarchical vocabulary specific to datasets. This specification defines three types of metadata elements: Required, Required-if (conditionally required), and Expanded fields. These elements were selected to represent information that is most often looked for on the web. To assist users of other metadata standards, field mappings to equivalent elements in other standards are provided.

What to Document – Datasets and Web APIs

A dataset is an identifiable collection of structured data objects unified by some criteria (authorship, subject, scope, spatial or temporal extent…). A catalog is a collection of descriptions of datasets; each description is a metadata record. The intention of a data catalog is to facilitate data access by users who are first interested in a particular kind of data, and upon finding a fit-for-purpose dataset, will next want to know how to get the data.

A Web API (Application Programming Interface) allows computer programs to dynamically query a dataset using the World Wide Web. For example, a dataset of farmers markets may be made available for download as a single file (e.g., a CSV), or may be made available to developers through a Web API, such that a computer program could use a ZIP Code to retrieve a list of farmers markets in the ZIP Code area.

The catalog file for each agency should list all of the agency’s datasets that can be made public, regardless of whether they are distributed by a file download or a Web API. Please also see the extended guidance on documenting Web APIs in your data.json files.

Metadata File Format – JSON

The Implementation Guidance available as a part of Project Open Data describes Agency requirements for the development of metadata as per the Open Data Policy. A quick primer on the file format involved:

JSON is a lightweight data-exchange format that is very easy to read, parse and generate. Based on a subset of the JavaScript programming language, JSON is a text format that is optimized for data interchange. JSON is built on two structures: (1) a collection of name/value pairs and (2) an ordered list of values.

Where optional fields are included in a catalog file but are unpopulated, they may be represented by a null value. They should not be represented by an empty string ("").

When a record has an accessURL or downloadURL, they should be contained as objects within a distribution. Any object may be described by title, description, format, or mediaType, though when an object contains downloadURL, it must be accompanied by mediaType.

The Project Open Data schema is case sensitive. The schema uses a camel case convention where the first letter of some words within a field are capitalized (usually all words but the first one). While it may seem subtle which characters are uppercase and lowercase, it is necessary to follow the exact same casing as defined in the schema documented here. For example:

Correct: contactPoint
Incorrect: ContactPoint
Incorrect: contactpoint
incorrect: CONTACTPOINT

Links to downloadable examples of metadata files developed in this and other formats are in the metadata resources. Agencies can validate their metadata using this validator.

Catalog Fields

These fields describe the entire Public Data Listing catalog file. Publishers can also use the describedBy field to reference the default JSON Schema file used to define the schema (https://project-open-data.cio.gov/v1.1/schema/catalog.json) or they may refer to their own JSON Schema file if they have extended the schema with additional schema definitions. Similarly, @context can be used to reference the default JSON-LD Context used to define the schema (https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld) or publishers can refer to their own if they have extended the schema with additional linked data vocabularies. See the Catalog section under Further Metadata Field Guidance for more details.

Dataset Fields

See the Further Metadata Field Guidance section to learn more about the use of each element, including the range of valid entries where appropriate. Consult the field mappings to find the equivalent v1.0, DCAT, Schema.org, and CKAN fields.

Dataset Distribution Fields

Within a dataset, distribution is used to aggregate the metadata specific to a dataset’s resources (accessURL and downloadURL), which may be described using the following fields. Each distribution should contain one accessURL or downloadURL. A downloadURL should always be accompanied by mediaType.

Extending the Schema

“Extensional” and/or domain specific metadata can easily be added using other vocabularies even if it is not a term (entity/property) that will get indexed by the major search engines - it could still be indexed by other custom search engines and by Data.gov. Publishers are encouraged to extend their metadata descriptions using elements from the “Expanded Fields” list shown below, or from any well-known vocabulary (including Dublin Core, Schema.org, FGDC, ISO 19115, and NIEM) as long as they are properly assigned. It’s also recommended that these extensions be defined through the describedBy and @context fields at the top of the Catalog metadata.

Further Metadata Field Guidance

Additional details for each field are provided here broken down into sections for the overarching Catalog, each dataset, and each dataset’s distribution. Consult the field mappings to find the equivalent v1.0, DCAT, Schema.org, and CKAN fields.

• Key
  • Required
  • Required if Applicable
  • Expanded (optional)

• Catalog
  • @context
  • @id
  • @type
  • conformsTo
  • describedBy
  • dataset
    • @type
    • accessLevel
    • accrualPeriodicity
    • bureauCode
    • conformsTo
    • contactPoint
      • @type
      • fn
      • hasEmail
    • dataQuality
    • describedBy
    • describedByType
    • description
    • distribution
      • @type
      • accessURL
      • conformsTo
      • downloadURL
      • describedBy
      • describedByType
      • description
      • format
      • mediaType
      • title
    • identifier
    • isPartOf
    • issued
    • keyword
    • landingPage
    • language
    • license
    • modified
    • primaryITInvestmentUII
    • programCode
    • publisher
      • @type
      • name
      • subOrganizationOf
    • references
    • rights
    • spatial
    • systemOfRecords
    • temporal
    • theme
    • title

Catalog Fields

Dataset Fields

            "contactPoint": {
                "@type": "vcard:Contact",
                "fn": "Jane Doe",
                "hasEmail": "mailto:jane.doe@agency.gov"
            }

"distribution": [
                 {
                     "@type": "dcat:Distribution",
                     "description": "Vegetable data as a CSV file",
                     "downloadURL": "http://www.agency.gov/vegetables/listofvegetables.csv",
                     "format": "CSV",
                     "mediaType": "text/csv",
                     "title": "listofvegetables.csv"
                 }, 
                 {
                     "@type": "dcat:Distribution",
                     "conformsTo": "http://www.agency.gov/vegetables-data-standard/",
                     "describedBy": "http://www.agency.gov/vegetables/schema.xsd",
                     "describedByType": "text/xml",
                     "description": "Vegetable data as an XML file",
                     "downloadURL": "http://www.agency.gov/vegetables/listofvegetables.xml",
                     "format": "XML",
                     "mediaType": "text/xml",
                     "title": "listofvegetables.xml"
                 },
                 {
                     "@type": "dcat:Distribution",
                     "description": "Vegetable data as a zipped CSV file with attached data dictionary",
                     "downloadURL": "http://www.agency.gov/vegetables/vegetables-all.zip",
                     "format": "Zipped CSV",
                     "mediaType": "application/zip",
                     "title": "vegetables-all.zip"
                 },
                 {
                     "@type": "dcat:Distribution",
                     "accessURL": "http://www.agency.gov/api/vegetables/",
                     "description": "A fully queryable REST API with JSON and XML output",
                     "format": "API",
                     "title": "Vegetables REST API"
                 }
                ]

"publisher": {
  "@type": "org:Organization",
  "name": "Widget Services",
  "subOrganizationOf": {
    "@type": "org:Organization",
    "name": "Office of Citizen Services and Innovative Technologies",
    "subOrganizationOf": {
      "@type": "org:Organization",
      "name": "General Services Administration",
      "subOrganizationOf": {
        "@type": "org:Organization",
        "name": "U.S. Government"
      }
    }
  }
}

Federal Government Fields

^USG — Fields specific to the U.S. Federal Government have been denoted with the ^USG superscript. The Project Open Data schema has been developed as part of a U.S Federal Government open data policy. However, every attempt has been made to align the schema with existing international standards and to provide opportunities for re-use and interoperability with state and local government as well as non-profits, academic institutions, and businesses. There are however some fields that have been introduced specifically for use by the U.S. Federal Government and have special meaning in that context. These fields are: bureauCode, programCode, dataQuality, primaryITInvestmentUII, and systemOfRecords. Non-federal data publishers are encouraged to make use of this schema, but these fields should not be seen as required and may not be relevant for those entities.

Rationale for Metadata Nomenclature

We sought to be platform-independent and to align as much as possible with existing open standards.

To that end, our JSON key names are directly drawn from DCAT, with a few exceptions.

We added the accessLevel field to help easily sort datasets into our three existing categories: public, restricted public, and non-public. This field means an agency can run a basic filter against its enterprise data catalog to generate a public-facing list of datasets that are, or could one day be, made publicly available (or, in the case of restricted data, available under certain conditions). This field also makes it easy for anyone to generate a list of datasets that could be made available but have not yet been released by filtering accessLevel to public and accessURL to blank.

We added the rights field (formerly accessLevelComment) for data stewards to explain how to access restricted public datasets, and for agencies to have a place to record (even if only internally) the reason for not releasing a non-public dataset.

We added the systemOfRecords field for data stewards to optionally link to a relevant System of Records Notice URL. A System of Records is a group of any records under the control of any agency from which information is retrieved by the name of the individual or by some identifying number, symbol, or other identifier assigned to the individual.

We added the bureauCode field to ensure every dataset is connected in a standard way with an agency bureau.

We added the programCode field to ensure that when applicable, every dataset is connected in a standard way with an agency program office.

We added the dataQuality to indicate whether or not the data meets an agency’s Information Quality Guidelines.

Additional Information

• Metadata Resources (including starter template and sample files)
• DCAT