Create a Dataset with the API#

Here you will learn some datasets API basics by using the API to create a dataset, add data to it, and get that data using the dataset’s the auto-generated REST API.

Prerequisite

IEX Cloud Apperate account - Create one here.

Creating a Dataset#

Here you will create a dataset as specified in a JSON file.

Create a .json file that has the following content. Replace you@company.com, Your Company, and Your Dataset, with your own values.

Important

The _system prefix (case-insensitive) is reserved for Apperate system tables and columns. You must not prefix dataset IDs or dataset property names with _system.

{
    "datasetId": "YOUR_DATASET",
    "title": "Your Dataset",
    "dataset": {
        "schema": {
            "properties": {
                "close": {
                    "type": "number"
                },
                "date": {
                    "type": "string",
                    "format": "date"
                },
                "high": {
                    "type": "number"
                },
                "low": {
                    "type": "number"
                },
                "open": {
                    "type": "number"
                },
                "category": {
                    "type": "string"
                },
                "symbol": {
                    "type": "string"
                },
                "volume": {
                    "type": "integer"
                }
            },
            "required": [
                "symbol",
                "category",
                "date"
            ],
            "type": "object"
        },
        "columnMapping": {
            "key": "symbol",
            "subkey": "category",
            "date": "date"
        },
        "smartLinkAttributes": [
            {
                "attributeName": "symbol",
                "type": "equity"
            }
        ]
    }
}

Create a dataset from the .json file by running a POST /datasets/:workspace request as described in Create a dataset. For example, run the following command, replacing the WORKSPACE, SECRET_TOKEN, and FILE values with your own.

Sample Request
```
curl -H "Content-Type: application/json" \
 -X POST "https://api.iex.cloud/v1/datasets/WORKSPACE?token=SECRET_TOKEN" \
 -d @FILE.json
```
Sample Response
```
{"success":true,"message":"Dataset has been created","datasetId":"YOUR_DATASET"}
```
Verify the dataset by running a GET /datasets/:workspace request as described in List datasets. For example,
```
curl -X GET https://api.iex.cloud/v1/datasets/WORKSPACE?token=SECRET_TOKEN
```

Let’s add data to the dataset.

Adding Data to Your Dataset#

Here you will specify data in a CSV file and submit the file in your request to create a dataset

Note

You can specify data in text files that use CSV, JSON, and JSONL formats. The product supports CSV files that use the following common data delimiters: comma (,), tab, or pipe (|) characters.

Here are the data file ingestion steps:

Specify the data in a CSV. Here’s a CSV data file.

Data

close,date,high,low,open,category,symbol,volume
72,2016-11-09,27.83,27.0125,27.47,tech,AAPL,236705444
9475,2016-11-25,27.9675,27.7375,27.7825,tech,AAPL,45903688
8925,2016-11-28,28.1163,27.8475,27.8575,tech,AAPL,108775932

Tip

Create the file using the following Example Command for your operating system.

Example Command

Linux/MacOS

echo "close,date,high,low,open,category,symbol,volume
27.72,2016-11-09,27.83,27.0125,27.47,tech,AAPL,236705444
27.9475,2016-11-25,27.9675,27.7375,27.7825,tech,AAPL,45903688
27.8925,2016-11-28,28.1163,27.8475,27.8575,tech,AAPL,108775932
" \
>> aapl

Windows

(
echo close,date,high,low,open,category,symbol,volume
echo 27.72,2016-11-09,27.83,27.0125,27.47,tech,AAPL,236705444
echo 27.9475,2016-11-25,27.9675,27.7375,27.7825,tech,AAPL,45903688
echo 27.8925,2016-11-28,28.1163,27.8475,27.8575,tech,AAPL,108775932
)>aapl

Ingest the data to your dataset using a POST /data/:workspace/:id request as described in POST /data. For example, use the following command, replacing the WORKSPACE, YOUR_DATASET, and SECRET_TOKEN values with your own.

Sample Request

curl -H "Content-Type: application/json" \
    -X POST "https://api.iex.cloud/v1/data/WORKSPACE/YOUR_DATASET?token=SECRET_TOKEN" \
    -d @aapl

Sample Response

{
    "success": true,
    "message": "Data upload of 579B for YOUR_DATASET completed, jobId: 887b948762ff4b5c889112afb21ea463 has been created",
    "jobId": "887b948762ff4b5c889112afb21ea463",
    "jobUrl": "/v1/jobs/WORKSPACE/ingest/887b948762ff4b5c889112afb21ea463"
}

Validate your dataset’s record count using a GET /datasets/:workspace/:id request as described in Get a dataset. For example, use the following command with your values.

Sample Request

curl -X GET https:/api.iex.cloud/v1/datasets/WORKSPACE/YOUR_DATASET?token=TOKEN

Sample Response

{
  "columnMapping": {
    "date": "date",
    "key": "symbol",
    "subkey": "category"
  },
  "datasetId": "YOUR_DATASET",
  "date": 1689188903000,
  "description": "",
  "keys": 0,
  "parentDatasetId": null,
 "records": 3,
  "schema": {
    "properties": {
      "category": {
        "type": "string"
      },
      "close": {
          "type": "number"
      },
      "date": {
        "format": "date",
        "type": "string"
      },
      "high": {
          "type": "number"
      },
      "low": {
          "type": "number"
      },
      "open": {
          "type": "number"
            },
      "symbol": {
        "type": "string"
      },
      "volume": {
        "type": "integer"
      }
    },
    "required": [
      "symbol",
      "category",
      "date"
    ],
    "type": "object"
  },
  "smartLinkAttributes": [
    {
      "attributeName": "symbol",
      "type": "equity"
    }
  ],
  "updated": 1689188903000
}

The records value "records":3 matches the number of records in your data file. Your data is ready for apps to access!

Lastly check out your dataset’s auto-generated RESTful API and use it.

Using Your Dataset#

A RESTful API endpoint was automatically created for your dataset.

Visit your API docs in your browser at the following URL, replacing the dataset name.

URL: https://iexcloud.io/docs/datasets/YOUR_DATASET

As a test, get your last record by pasting the following URL into your browser, replacing WORKSPACE, YOUR_DATASET, and TOKEN.

URL: https://api.iex.cloud/v1/data/WORKSPACE/YOUR_DATASET?last=1&token=TOKEN

Sample Response

[
    {
        "close": 47.8925,
        "date": "2017-11-28",
        "high": 28.1163,
        "low": 27.8475,
        "open": 27.8575,
        "category": "tech",
        "symbol": "AAPL",
        "volume": 108775932
    }
]

Congratulations on making your data available using the datasets API!

What’s Next#

Learn more about the Apperate APIs at Use Apperate’s APIs.