Batch Write Data with POST /data

The POST /data endpoint is useful for ingesting large amounts of data into Apperate and for debugging failing record writes. Here we’ll demonstrate calling POST /data, viewing the ingestion job, and querying the ingested data.

Note

POST /data does not publish any events to the Rules Engine. To publish events with data writes, see Write Data with POST /record.

Prerequisite

IEX Cloud Apperate account - Create one here.

Call POST /data

1. Prepare a POST /data endpoint call. For example, use the following cURL command, replacing the WORKSPACE, DATASET_ID, and SECRET_TOKEN values with your workspace name, a target dataset, and your secret API token and replacing the data (the single-quoted array following -d) with your own JavaScript object array.

   curl -X POST https://WORKSPACE.iex.cloud/v1/data/WORKSPACE/DATASET_ID?token=SECRET_TOKEN \
   > -H 'Content-Type: application/json' \
   > -d '[{"headline": "Airbag recall", "content": "Here is what you need to know ...", "ticker":"GM", "source": "IEX Underground", "date": "2022-07-16"}]'
   

2. Execute the command.

   The method returns a response like the one below and starts a data ingestion job to write data to the target dataset.

   {"success":true,"message":"Data upload of 145 bytes for 30A8D8815A4120A8C57940FF0A24B2C0 completed, jobId: 5ae43de65a794b7f9eeb8f914abe528a has been created","jobId":"5ae43de65a794b7f9eeb8f914abe528a","jobUrl":"/v1/jobs/WORKSPACE/ingest/5ae43de65a794b7f9eeb8f914abe528a"}
   

Check the Data Ingestion

1. In the Console near the top, click Logs.

   

   The Log Stream messages appear.

2. Filter on messages that have the jobId returned from your POST /data response.

   

   The matching messages appear.

   

3. View the dataset’s ingestion jobs in the Ingestion Jobs tab.

   

   If the ingestion completed successfully, the status Current Status is INGESTED.

   If ingestion failed, check the Invalid Records column to debug the failure.

   See also

   See Troubleshoot Invalid Data Records for details.

Query the Data

1. Click the target dataset name to view the dataset. The dataset Database page appears and includes the record(s) you wrote.

   

2. Click the Overview tab. The Overview page appears and includes an Example Request.

   

3. Click the Example Request URL. A new tab opens with a response to the request.

   [
     {
       "content": "Here is what you need to know ...",
       "date": "2022-07-15",
       "headline": "Airbag recall",
       "source": "IEX Underground",
       "ticker": "GM"
     }
   ]
   

   The example request called the GET /data endpoint to get the last record.

   See also

   See Query Data for all the ways to query Apperate’s real-time data store.

Congratulations on your first step to using Apperate as your real-time data store!

What’s Next

Write Data with POST /record demonstrates writing data in real-time to Apperate. The data writes register as Rules Engine events you can filter on.

Update a Dataset Schema shows how to examine and modify your dataset schema.

Query Data demonstrates all the different ways to query your data using Apperate’s API gateway.

Connect with Data introduces all the ways you can access data to create rules on and enrich your application.