Create a Dataset with the API#

Here you will learn some datasets API basics by using the API to create a dataset, add data to it, and get that data using the dataset’s the auto-generated REST API.

Prerequisite

IEX Cloud Apperate account - Create one here.

Creating a Dataset#

Here you will create a dataset as specified in a JSON file.

  1. Create a .json file that has the following content. Replace you@company.com, Your Company, and Your Dataset, with your own values.

    Important

    The _system prefix (case-insensitive) is reserved for Apperate system tables and columns. You must not prefix dataset IDs or dataset property names with _system.

    {
        "datasetId": "YOUR_DATASET",
        "title": "Your Dataset",
        "dataset": {
            "schema": {
                "properties": {
                    "close": {
                        "type": "number"
                    },
                    "date": {
                        "type": "string",
                        "format": "date"
                    },
                    "high": {
                        "type": "number"
                    },
                    "low": {
                        "type": "number"
                    },
                    "open": {
                        "type": "number"
                    },
                    "category": {
                        "type": "string"
                    },
                    "symbol": {
                        "type": "string"
                    },
                    "volume": {
                        "type": "integer"
                    }
                },
                "required": [
                    "symbol",
                    "category",
                    "date"
                ],
                "type": "object"
            },
            "columnMapping": {
                "key": "symbol",
                "subkey": "category",
                "date": "date"
            },
            "smartLinkAttributes": [
                {
                    "attributeName": "symbol",
                    "type": "equity"
                }
            ]
        }
    }
    
  2. Create a dataset from the .json file by running a POST /datasets/:workspace request as described in Create a dataset. For example, run the following command, replacing the WORKSPACE, SECRET_TOKEN, and FILE values with your own.

    Sample Request

    curl -H "Content-Type: application/json" \
     -X POST "https://api.iex.cloud/v1/datasets/WORKSPACE?token=SECRET_TOKEN" \
     -d @FILE.json
    

    Sample Response

    {"success":true,"message":"Dataset has been created","datasetId":"YOUR_DATASET"}
    
  3. Verify the dataset by running a GET /datasets/:workspace request as described in List datasets. For example,

    curl -X GET https://api.iex.cloud/v1/datasets/WORKSPACE?token=SECRET_TOKEN
    

Let’s add data to the dataset.

Adding Data to Your Dataset#

Here you will specify data in a CSV file and submit the file in your request to create a dataset

Note

You can specify data in text files that use CSV, JSON, and JSONL formats. The product supports CSV files that use the following common data delimiters: comma (,), tab, or pipe (|) characters.

Here are the data file ingestion steps:

  1. Specify the data in a CSV. Here’s a CSV data file.

    Data

    close,date,high,low,open,category,symbol,volume
    27.72,2016-11-09,27.83,27.0125,27.47,tech,AAPL,236705444
    27.9475,2016-11-25,27.9675,27.7375,27.7825,tech,AAPL,45903688
    27.8925,2016-11-28,28.1163,27.8475,27.8575,tech,AAPL,108775932
    

    Tip

    Create the file using the following Example Command for your operating system.

    Example Command

    echo "close,date,high,low,open,category,symbol,volume
    27.72,2016-11-09,27.83,27.0125,27.47,tech,AAPL,236705444
    27.9475,2016-11-25,27.9675,27.7375,27.7825,tech,AAPL,45903688
    27.8925,2016-11-28,28.1163,27.8475,27.8575,tech,AAPL,108775932
    " \
    >> aapl
    
    (
    echo close,date,high,low,open,category,symbol,volume
    echo 27.72,2016-11-09,27.83,27.0125,27.47,tech,AAPL,236705444
    echo 27.9475,2016-11-25,27.9675,27.7375,27.7825,tech,AAPL,45903688
    echo 27.8925,2016-11-28,28.1163,27.8475,27.8575,tech,AAPL,108775932
    )>aapl
    
  2. Ingest the data to your dataset using a POST /data/:workspace/:id request as described in POST /data. For example, use the following command, replacing the WORKSPACE, YOUR_DATASET, and SECRET_TOKEN values with your own.

    Sample Request

    curl -H "Content-Type: application/json" \
        -X POST "https://api.iex.cloud/v1/data/WORKSPACE/YOUR_DATASET?token=SECRET_TOKEN" \
        -d @aapl
    

    Sample Response

    {
        "success": true,
        "message": "Data upload of 579B for YOUR_DATASET completed, jobId: 887b948762ff4b5c889112afb21ea463 has been created",
        "jobId": "887b948762ff4b5c889112afb21ea463",
        "jobUrl": "/v1/jobs/WORKSPACE/ingest/887b948762ff4b5c889112afb21ea463"
    }
    
  3. Validate your dataset’s record count using a GET /datasets/:workspace/:id request as described in Get a dataset. For example, use the following command with your values.

    Sample Request

    curl -X GET https:/api.iex.cloud/v1/datasets/WORKSPACE/YOUR_DATASET?token=TOKEN
    

    Sample Response

    {
      "columnMapping": {
        "date": "date",
        "key": "symbol",
        "subkey": "category"
      },
      "datasetId": "YOUR_DATASET",
      "date": 1689188903000,
      "description": "",
      "keys": 0,
      "parentDatasetId": null,
     "records": 3,
      "schema": {
        "properties": {
          "category": {
            "type": "string"
          },
          "close": {
              "type": "number"
          },
          "date": {
            "format": "date",
            "type": "string"
          },
          "high": {
              "type": "number"
          },
          "low": {
              "type": "number"
          },
          "open": {
              "type": "number"
                },
          "symbol": {
            "type": "string"
          },
          "volume": {
            "type": "integer"
          }
        },
        "required": [
          "symbol",
          "category",
          "date"
        ],
        "type": "object"
      },
      "smartLinkAttributes": [
        {
          "attributeName": "symbol",
          "type": "equity"
        }
      ],
      "updated": 1689188903000
    }
    

    The records value "records":3 matches the number of records in your data file. Your data is ready for apps to access!

Lastly check out your dataset’s auto-generated RESTful API and use it.

Using Your Dataset#

A RESTful API endpoint was automatically created for your dataset.

  1. Visit your API docs in your browser at the following URL, replacing the dataset name.

    URL: https://iexcloud.io/docs/datasets/YOUR_DATASET

  2. As a test, get your last record by pasting the following URL into your browser, replacing WORKSPACE, YOUR_DATASET, and TOKEN.

    URL: https://api.iex.cloud/v1/data/WORKSPACE/YOUR_DATASET?last=1&token=TOKEN

    Sample Response

    [
        {
            "close": 47.8925,
            "date": "2017-11-28",
            "high": 28.1163,
            "low": 27.8475,
            "open": 27.8575,
            "category": "tech",
            "symbol": "AAPL",
            "volume": 108775932
        }
    ]
    

Congratulations on making your data available using the datasets API!

What’s Next#

Learn more about the Apperate APIs at Use Apperate’s APIs.