ERDDAP > RESTful Web Services
ERDDAP is both:
- A web application
– a web page with a form that humans with browsers can use (in this case, to get data, graphs, or information about datasets).
- A RESTful web service
– a URL that computer programs can use (in this case, to get data, graphs, and information about datasets).
For every ERDDAP web page with a form that you as a human with a browser can use, there is a
corresponding ERDDAP web service that is designed to be easy for computer programs and scripts to use.
For example, humans can use this URL to do a Full Text Search for interesting datasets:
http://erddap.exploratorium.edu:8080/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=temperature
By changing the file extension in the URL from .html to .json (or .csv, or .htmlTable, or .jsonlCSV1, or .xhtml, ...):
http://erddap.exploratorium.edu:8080/erddap/search/index.json?page=1&itemsPerPage=1000&searchFor=temperature
we get a URL that a computer program or JavaScript script can use to get the same
information in a more computer-program-friendly format like
JSON
.
There are many features in ERDDAP that can be used by computer programs or scripts
that you write. You can use them to build other web applications or web services on
top of ERDDAP, making ERDDAP do most of the work!
So if you have an idea for a better interface to the data that ERDDAP serves or a web
page that needs an easy way to access data, we encourage you to build your own
web application, web service, or web page and use ERDDAP as the foundation.
Your system can get data, graphs, and other information from ERD's ERDDAP or from
other ERDDAP installations, or you can
set up your own ERDDAP server,
which can be
publicly accessible or just privately accessible.
Requests for user-interface information from ERDDAP (for example, search results)
use the web's universal standard for requests:
URLs
sent via
HTTP GET
.
This is the same mechanism that your browser uses when you fill out a form
on a web page and click on
Submit
To use HTTP GET, you generate a specially formed URL (which may include a query)
and send it with HTTP GET. You can form these URLs by hand and enter them in
the address text field of your browser (for example,
http://erddap.exploratorium.edu:8080/erddap/search/index.json?page=1&itemsPerPage=1000&searchFor=temperature)
Or, you can write a computer program or web page script to create a URL, send it,
and get the response. URLs via HTTP GET were chosen because
- They are simple to use.
- They work well.
- They are universally supported (in browsers, computer languages, operating system
tools, etc).
- They are a foundation of
Representational State Transfer (REST)
and
Resource Oriented Architecture (ROA)
- They facilitate using the World Wide Web as a big distributed application,
for example via
mashups
AJAX applications
- They are stateless
as is ERDDAP, which makes the system simpler.
- A URL completely define a given request, so you can bookmark it in your browser,
write it in your notes, email it to a friend, etc.
In URLs, some characters are not allowed (for example, spaces) and other characters
have special meanings (for example, '&' separates key=value pairs in a query).
When you fill out a form on a web page and click on Submit, your browser automatically
percent encodes
the special characters in the URL (for example, space becomes %20), for example,
http://erddap.exploratorium.edu:8080/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=temperature%20wind%20speed
But if your computer program or script generates the URLs, it probably needs to do the percent
encoding itself. If so, then probably all characters other than A-Za-z0-9_-!.~'()*
in the query's values (the parts after the '=' signs) need to be encoded
as %HH, where HH is the 2 digit hexadecimal value of the character, for example, space becomes %20.
Characters above #127 must be converted to UTF-8 bytes, then each UTF-8 byte must be percent encoded
(ask a programmer for help). Programming languages have tools to do this (for example, see Java's
java.net.URLEncoder
and JavaScript's
encodeURIComponent()
) and there are
websites that percent encode/decode for you
Although humans using browsers want to receive user-interface results (for example,
search results) as HTML documents, computer programs often prefer to get results in
simple, easily parsed, less verbose documents. ERDDAP can return user-interface
results as a table of data in these common, computer-program friendly, file types:
In every results table format (except .jsonlKVP, where column names are on every row):
- Each column has a column name and one type of information.
- The first row of the table has the column names.
- Subsequent rows have the information you requested.
The content in these plain file types is also slightly different from the .html
response — it is intentionally bare-boned so that it is easier for a computer
program to work with.
A Consistent Data Structure for the Responses
All of the user-interface services described on this page can return a table of
data in any of the common file formats listed above. Hopefully, you can write
just one procedure to parse a table of data in one of the formats. Then you can
re-use that procedure to parse the response from any of these services. This
should make it easier to deal with ERDDAP.
.csv and .tsv Details
- If a datum in a .csv file has internal double quotes or commas, ERDDAP follows the
.csv specification strictly: it puts double quotes around the datum and doubles
the internal double quotes.
- Special characters in a .csv or .tsv file are encoded like
JSON
backslash-encoded characters:
\n (newline), \\ (backslash), \f (formfeed), \t (tab), \r (carriage return)
or with the \uhhhh syntax.
jsonp
Requests for .json files may now include an optional
jsonp
request by
adding "&.jsonp=functionName" to the end of the query. Basically, this tells
ERDDAP to add "functionName(" to the beginning of the response and ")" to the
end of the response. The first character of functionName must be an ISO 8859 letter or "_".
Each optional subsequent character must be an ISO 8859 letter, "_", a digit, or ".".
If originally there was no query, leave off the "&" in your query.
griddap and tabledap Offer Different File Types
The file types listed above are file types ERDDAP can use to respond to
user-interface types of requests (for example, search requests). ERDDAP supports
a different set of file types for scientific data (for example, satellite and buoy
data) requests (see the
griddap and
tabledap
documentation).
ERDDAP doesn't offer results stored in compressed (e.g., .zip or .gzip) files.
Instead, ERDDAP looks for
accept-encoding
in the HTTP GET request header sent
by the client. If a supported compression type (gzip, x-gzip, or deflate) is found
in the accept-encoding list, ERDDAP includes "content‑encoding" in the HTTP response
header and compresses the data as it transmits it.
It is up to the client program to look for
content-encoding and decompress the data accordingly.
Requesting compression is optional, but compressed responses are often 3-10 times faster,
so this is a big time savings if you are downloading lots of large files.
(Note that there is no benefit
to requesting compressed .png files since the files' contents are already compressed.)
- By default, browsers and OPeNDAP clients always request compressed data and
decompress the returned data.
- With curl, add --compressed to the command line to tell curl to
request a compressed response and automatically decompress it.
- With other client software, you have explicitly set this up.
Here is a
Java example.
Here is a
Python example
(although you should either handle deflate'd responses or not request deflate).
ERDDAP has these URL access points for computer programs:
- To get the list of the main resource access URLs, use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- To get the current list of all datasets, use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- To get metadata for a specific data set
(the list of variables and their attributes), use
http://erddap.exploratorium.edu:8080/erddap/info/datasetID/index.fileType
for example,
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- To get the results of full text searches for datasets
(using "searchFor=wind%20speed" as the example), use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
(Your program or script may need to
percent-encode
the value in the query.)
Or, use the
OpenSearch 1.1
standard to do a full text search for datasets.
- To get the results of advanced searches for datasets
(using "searchFor=wind%20speed" as the example), use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
But experiment with
Advanced Search
in a browser to figure out all of the optional parameters.
(Your program or script may need to
percent-encode
the value in the query.)
- To get the list of categoryAttributes
(for example, institution, long_name, standard_name), use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- To get the list of categories for a specific categoryAttribute
(using "standard_name" as the example), use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- To get the list of datasets in a specific category
(using "standard_name=time" as the example), use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- To get the current list of all datasets available via a specific protocol,
- For griddap: use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- For tabledap: use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- For WMS: use
.csv,
.htmlTable,
.itx,
.json,
.jsonlCSV1,
.jsonlCSV,
.jsonlKVP,
.mat,
.nc,
.nccsv,
.tsv, or
.xhtml.
- Griddap and tabledap have many web services that you can use.
- The Data Access Forms are just simple web pages to generate URLs which
request data (for example, satellite and buoy data). The data can be in any of
several common file formats. Your program can generate these URLs directly.
For more information, see the
griddap documentation and
tabledap documentation.
- The Make A Graph pages are just simple web pages to generate URLs which
request graphs of a subset of the data. The graphs can be in any of several
common file formats. Your program can generate these URLs directly. For
more information, see the
griddap documentation and
tabledap documentation.
- To get a dataset's structure, including variable names and data types,
use a standard OPeNDAP
.dds
resquest. For example,
http://erddap.exploratorium.edu:8080/erddap/griddap/jplMURSST.dds (gridded data) or
http://erddap.exploratorium.edu:8080/erddap/tabledap/pmelTaoDySst.dds (tabular data).
- To get a dataset's metadata, use a standard OPeNDAP
.das
resquest.
For example,
http://erddap.exploratorium.edu:8080/erddap/griddap/jplMURSST.das (gridded data) or
http://erddap.exploratorium.edu:8080/erddap/tabledap/pmelTaoDySst.das (tabular data).
- ERDDAP has a special tabular dataset called
allDatasets
which has data about all of the datasets currently available in
this ERDDAP. There is a row for each dataset. There are columns with
different types of information (for example, datasetID, title, summary,
institution, license, Data Access Form URL, Make A Graph URL).
Because this is a tabledap dataset,
you can use a tabledap data request to request
specific columns and rows which match the constraints, and you can
get the response in whichever response file type you prefer,
for example, .html, .xhtml, .csv, .json, .jsonlCSV1, .jsonlCSV, or .jsonlKVP.
-
ERDDAP's other protocols also have web services that you can use.
See
- ERDDAP offers
RSS subscriptions,
so that your computer program can find out if a dataset has changed.
- ERDDAP offers
email/URL subscriptions,
which notify your computer program whenever a dataset changes.
- ERDDAP offers several converters as web pages and as web services:
- ERDDAP has a system to keep track of
Out-Of-Date Datasets.
See the Options at the bottom of that web page.
If you have suggestions for additional links, contact
erd dot data at noaa dot gov.
As described above, since Java programs can access data available on the web, you can
write a Java program that accesses data from any publicly accessible ERDDAP installation.
Or, since ERDDAP is an all-open source program, you can also set up your own copy of
ERDDAP on your own server (publicly accessible or not) to serve your own data. Your Java
programs can get data from that copy of ERDDAP. See
Set Up Your Own ERDDAP.
Many ERDDAP installations don't have authentication enabled and thus
don't provide any way for users to login, nor do they have any private datasets.
Some ERDDAP installations do have authentication enabled.
Currently, ERDDAP only supports authentication via Google-managed email accounts,
which includes email accounts at NOAA and many universities.
If an ERDDAP has authentication enabled, anyone with a Google-managed email account
can log in, but they will only have access to the private datasets
that the ERDDAP administrator has explicitly authorized them to access.
For instructions on logging into ERDDAP from a browser or via a script, see
Access to Private Datasets in ERDDAP.
If you want to use a new feature on a remote ERDDAP, you can find out if the new
feature is available by sending a request to determine the ERDDAP's version
number, for example,
http://erddap.exploratorium.edu:8080/erddap/version
ERDDAP will send a text response with the ERDDAP version number of that ERDDAP.
For example:
ERDDAP_version=2.22
If you get an
HTTP 404 Not-Found error message, treat the ERDDAP as version
1.22 or lower.
Or, you can request the version_string, which may have additional information.
For example,
http://erddap.exploratorium.edu:8080/erddap/version_string
ERDDAP will send a text response with the ERDDAP version_string of that ERDDAP.
It will be a floating point number (the version number)
with an optional suffix of '_' plus additional ASCII text (no spaces or control characters).
For example:
ERDDAP_version_string=2.22_JohnsFork
If you get an HTTP 404 Not-Found error message, treat the ERDDAP as version
1.80 or lower.