Create a Dataset with the API#
Here you will learn some datasets API basics by using the API to create a dataset, add data to it, and get that data using the dataset’s the auto-generated REST API.
Prerequisite
IEX Cloud Apperate account - Create one here.
Creating a Dataset#
Here you will create a dataset as specified in a JSON file.
Create a
.json
file that has the following content. Replaceyou@company.com
,Your Company
, andYour Dataset
, with your own values.Important
The _system prefix (case-insensitive) is reserved for Apperate system tables and columns. You must not prefix dataset IDs or dataset property names with _system.
{ "dataset": { "email": "you@company.com", "schema": { "properties": { "close": { "type": "number" }, "date": { "type": "string", "format": "date" }, "high": { "type": "number" }, "low": { "type": "number" }, "open": { "type": "number" }, "symbol": { "type": "string" }, "volume": { "type": "integer" } }, "required": [ "symbol", "date" ], "type": "object" }, "source": "Your Company", "columnMapping": { "key": "symbol", "date": "date" }, "smartLinkAttributes": [ { "attributeName": "symbol", "type": "equity" } ], "title": "SAMPLE AAPL DATASET YOU", "datasetId": "SAMPLE_AAPL_DATASET_YOU" } }
Create a dataset from the
.json
file by running aPOST /datasets/:workspace
request as described in Create a dataset. For example, run the following command, replacing theWORKSPACE
,SECRET_TOKEN
, andFILE
values with your own.Sample Request
curl -H "Content-Type: application/json" \ -X POST "https://api.iex.cloud/v1/datasets/WORKSPACE?token=SECRET_TOKEN" \ --data-binary @FILE.json
Sample Response
{"success":true,"message":"Dataset has been created","datasetId":"YOUR_DATASET"}
Verify the dataset by running a
GET /datasets/:workspace
request as described in List datasets. For example,curl -X GET https://api.iex.cloud/v1/datasets/WORKSPACE?token=SECRET_TOKEN
Let’s add data to the dataset.
Adding Data to Your Dataset#
Here you will specify data in a CSV file and submit the file in your request to create a dataset
Note
You can specify data in text files that use CSV, JSON, and JSONL formats. The product supports CSV files that use the following common data delimiters: comma (,), tab, or pipe (|) characters.
Here are the data file ingestion steps:
Specify the data in a CSV. Here’s a CSV data file.
Data
close,date,high,low,open,symbol,volume 27.72,2016-11-09,27.83,27.0125,27.47,AAPL,236705444 27.9475,2016-11-25,27.9675,27.7375,27.7825,AAPL,45903688 27.8925,2016-11-28,28.1163,27.8475,27.8575,AAPL,108775932
Tip
Create the file using the following Example Command for your operating system.
Example Command
echo " close,date,high,low,open,symbol,volume 27.72,2016-11-09,27.83,27.0125,27.47,AAPL,236705444 27.9475,2016-11-25,27.9675,27.7375,27.7825,AAPL,45903688 27.8925,2016-11-28,28.1163,27.8475,27.8575,AAPL,108775932 " \ >> aapl
( echo close,date,high,low,open,symbol,volume echo 27.72,2016-11-09,27.83,27.0125,27.47,AAPL,236705444 echo 27.9475,2016-11-25,27.9675,27.7375,27.7825,AAPL,45903688 echo 27.8925,2016-11-28,28.1163,27.8475,27.8575,AAPL,108775932 )>aapl
Ingest the data to your dataset using a
POST /data/:workspace/:id
request as described in Ingest data. For example, use the following command, replacing theWORKSPACE
,YOUR_DATASET
, andSECRET_TOKEN
values with your own.Sample Request
curl -H "Content-Type: application/json" \ -X POST "https://api.iex.cloud/v1/data/WORKSPACE/YOUR_DATASET?token=SECRET_TOKEN" \ --data-binary @aapl
Sample Response
{ "success": true, "message": "Data upload of 579B for YOUR_DATASET completed, jobId: 887b948762ff4b5c889112afb21ea463 has been created", "jobId": "887b948762ff4b5c889112afb21ea463", "jobUrl": "/v1/jobs/WORKSPACE/ingest/887b948762ff4b5c889112afb21ea463" }
Validate your dataset’s record count using a
GET /datasets/:workspace/:id
request as described in Get a dataset. For example, use the following command with your values.Sample Request
curl -X GET https:/api.iex.cloud/v1/datasets/WORKSPACE/YOUR_DATASET?token=TOKEN
Sample Response
{ "columnMapping": { " date": "date", "key": "symbol" }, "smartLinkAttributes": [ { "attributeName": "symbol", "type": "equity" } ], "date": 1650569968000, "updated": 1650569968000, "datasetId": " YOUR_DATASET ", "schema": { "properties": { "close": { " type": "number" }, "date": { " format": "date", "type": "string" }, "high": { " type": "number" }, "low": { " type": "number" }, "open": { " type": "number" }, "symbol": { " type": "string" }, "volume": { " type": "integer" } }, "required": [ " symbol", "date" ], " type": "object" }, "description": "", "parentDatasetId": null, "keys": 0, "records": 3 }
The records value
"records":3
matches the number of records in your data file. Your data is ready for apps to access!
Lastly check out your dataset’s auto-generated RESTful API and use it.
Using Your Dataset#
A RESTful API endpoint was automatically created for your dataset.
Visit your API docs in your browser at the following URL, replacing the dataset name.
URL:
https://iexcloud.io/docs/datasets/YOUR_DATASET
As a test, get your last record by pasting the following URL into your browser, replacing
WORKSPACE
,YOUR_DATASET
, andTOKEN
.URL:
https://api.iex.cloud/v1/data/WORKSPACE/YOUR_DATASET?last=1&token=TOKEN
Sample Response
[ { "close": 47.8925, "date": "2017-11-28", "high": 28.1163, "low": 27.8475, "open": 27.8575, "symbol": "AAPL", "volume": 108775932 } ]
Congratulations on making your data available using the datasets API!
What’s Next#
Learn more about the Apperate APIs at Use Apperate’s APIs.