Ingest More Data into a Dataset

You can load more data into an existing dataset using any of these ways:

• Load data from a new data source

• Load data from an existing data source

• Schedule data ingestion

We’ll cover each of the above options. No matter the option you choose, however, you must decide how to handle when an incoming record matches (i.e., has the same Unique Index) an existing record.

You can handle matching records in either of these ways:

• Ignore matches: Skip ingesting the new record. (Default)

• Replace matches: Overwrite the existing record with the new record.

As we over each data ingestion method, we’ll call out the matched record handling options.

Load Data from a New Data Source

All your dataset pages have an Ingest button you can click to load data from a selected data source.

1. In any of your dataset’s pages, click Ingest. The Upload Data page appears.

2. Select the source type to ingest data from.

   

3. Specify the data source options and ingest the data.

   These articles show how to set up the data sources and ingest data from them.

   • AWS S3 bucket

   • URL

   • File

   Important

   Incoming matches are ignored. By default, Apperate skips ingesting new records that match (have same Unique Index as) existing records.

Data loads into your dataset.

Load Data from an Existing Data Source

If you have already have an existing data source (e.g., that you created separately or along with your dataset), you can load new data from it.

1. In the console, go to Sources → Data Sources.

   The Data Sources page appears.

   

2. Click a data source name to open it’s page. That data source’s page appears.

3. In the data source page, scroll down to the Ingest Data section. Here’s an example Ingest Data section:

   

4. Select a dataset from the Destination Dataset drop-down.

   

5. Select your option for handling matched records: Ignore matches or Replace matches.

6. When you’re done specifying options, click Ingest at the bottom of the page.

Apperate loads data from the data source.

Scheduling Data Ingestion

This last option allows you to ingest data from an existing data source into your dataset per a schedule.

Note

You can create a data source from the Data Sources page or as a part of creating a dataset (select the Save this data source? option).

See Schedule Data Ingestion for a demonstration on loading data automatically per a schedule you create.

Congratulations! You know three different ways to load more data into your datasets.

What’s Next?

• Create Views demonstrates joining datasets to create views.

• Query Data shows how to search datasets using the GET /data REST endpoint and using the iex.js JavaScript Library.

• Use Apperate’s APIs introduces Apperate REST endpoints to query for the exact data you want and CRUD endpoints to use Apperate programatically.