Paginate with Chunks of Data#

If you’re querying large amounts of data or if you’re paginating data in your application, it can be helpful to iterate over set chunks/pages of that data. Apperate’s API and iexjs library make this a snap.

For example, here’s output from working with chunks of five news articles at a time.

5 most recent articles starting from article 1
2023-04-27T23:24:07.000Z,  Headline: Microsoft’s US$69bil Activision deal blocked
2023-04-27T23:17:11.000Z,  Headline: Microsoft Is Ending Windows 10 Updates - CNET - [Sepe.gr]
2023-04-27T22:47:51.000Z,  Headline: Futures: Amazon Erases Gains On Cloud Fears; These Techs Dive
2023-04-27T22:21:20.000Z,  Headline: How to buy Microsoft stock (MSFT)
2023-04-27T22:01:55.000Z,  Headline: UK leaves Microsoft’s US$69bn takeover bid in peril
5 most recent articles starting from article 6
2023-04-27T21:55:08.000Z,  Headline: Microsoft’s Activision deal is on life support because cloud gaming still sucks
2023-04-27T21:26:06.000Z,  Headline: Microsoft Is Ending Windows 10 Updates
2023-04-27T21:20:00.000Z,  Headline: Microsoft Is Ending Windows 10 Updates - CNET
2023-04-27T21:16:51.000Z,  Headline: SCHG: Heavier Exposure To Apple, Microsoft, And Google
2023-04-27T21:16:17.000Z,  Headline: UK Watchdog Blocks Microsoft''s Acquisition Of Activision Blizzard
5 most recent articles starting from article 11
2023-04-27T21:15:31.000Z,  Headline: Microsoft blasts Britain over decision to block £60bn Call of Duty deal
// ... more articles

You can specify whatever number of records you need in your data chunks.

Here we’ll demonstrate working with pages/chunks of data using the apperate.queryData() iexjs method and using the GET /data REST endpoint directly. We’ll also look at sorting order and cover the importance of time-windowing chunk queries.

Example: Using apperate.queryData()#

Our open source iexjs library’s apperate.queryData() method is the easiest way to query Apperate data in JavaScript.

The following code iterates over Microsoft News records in groups of five, from March 27th until March 28th 2023.

Code:

'use strict';
const {Client} = require("@apperate/iexjs");

async function start() {
    const client = new Client({api_token: YOUR_TOKEN, version: "v1"});
    const increment=5;
    var oset=0;
    while (true) {
        const data = await client.apperate.queryData({
            workspace: "core", id: "news", key: "msft",
            from: "2023-04-27", to: "2023-04-28",
            offset: oset, last: increment});
        if (data.length > 0) {
            console.log(
                `${increment} most recent articles starting from article ${oset + 1}`);
            data.forEach((article) => {
                console.log(
                    `${new Date(article.datetime).toISOString()},  Headline: ${article.headline}`);
            });
            oset+=increment;
        }
        else {
            break;
        }
    }
    console.log("Done");
}

start();

Command:

If you want to use iexjs in a NodeJS environment, you can install iexjs with npm.

npm install @apperate/iexjs

Note

iexjs can also run in the browser via native fetch and eventsource, or from node via cross-fetch and eventsource.

Then you can save the example code to a script and run the script in NodeJS.

node your-script.js

Example Output:

5 most recent articles starting from article 1
2023-04-27T23:24:07.000Z,  Headline: Microsoft’s US$69bil Activision deal blocked
2023-04-27T23:17:11.000Z,  Headline: Microsoft Is Ending Windows 10 Updates - CNET - [Sepe.gr]
2023-04-27T22:47:51.000Z,  Headline: Futures: Amazon Erases Gains On Cloud Fears; These Techs Dive
2023-04-27T22:21:20.000Z,  Headline: How to buy Microsoft stock (MSFT)
2023-04-27T22:01:55.000Z,  Headline: UK leaves Microsoft’s US$69bn takeover bid in peril
5 most recent articles starting from article 6
2023-04-27T21:55:08.000Z,  Headline: Microsoft’s Activision deal is on life support because cloud gaming still sucks
2023-04-27T21:26:06.000Z,  Headline: Microsoft Is Ending Windows 10 Updates
2023-04-27T21:20:00.000Z,  Headline: Microsoft Is Ending Windows 10 Updates - CNET
2023-04-27T21:16:51.000Z,  Headline: SCHG: Heavier Exposure To Apple, Microsoft, And Google
2023-04-27T21:16:17.000Z,  Headline: UK Watchdog Blocks Microsoft''s Acquisition Of Activision Blizzard
5 most recent articles starting from article 11
2023-04-27T21:15:31.000Z,  Headline: Microsoft blasts Britain over decision to block £60bn Call of Duty deal
// ... more articles

Let’s examine the code.

The new Client({api_token: YOUR_TOKEN, version: "v1"}) statement creates an iexjs client. This client object uses API version v1 and an API token. Replace YOUR_TOKEN with a qualified API token.

Tip

If you set an IEX_TOKEN environment variable to your API token, you don’t have to specify an api_token value for your client; iexjs automatically applies the token from your environment.

The increment value (const increment=5) is passed to the query’s last parameter–this sets the number of query results to request. Set your increment to the number of results you want to work with in each chunk. The oset variable holds the offset value: the index of the first record to return in query result array.

Note

If you want to start with getting the oldest records first, use the first parameter instead of last. See Getting Oldest Records First below for details.

The while loop starts with querying data via an apperate.queryData() call. The above call queries the IEX Cloud News dataset for its last (most recent) five records that have the key index value msft AND that have date index values from 2023-04-27 up to (but not including) 2023-04-28.

If the query result array has elements (data.length > 0), the script iterates over them to print each article’s timestamp (article.datetime) and headline (article.headline).

Five new articles (increment=5) are processed on each loop iteration because the incremented offest (oset+=increment) skips past the previously retrieved records.

When the query result array is empty, the loop stops and Done is printed.

It’s that easy to chunk/page through data in Apperate!

If you want to use this approach (i.e., using apperate.queryData), you can skip ahead to Handle Data Your Way for some helpful details and best practices. Otherwise, read on to learn how to chunk data using the GET /data REST endpoint.

Example: Using GET /data#

You can also get records in chunks using the GET /data REST endpoint directly.

Code:

'use strict';
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));

async function yourFunction() {
    const token="YOUR_TOKEN";
    const increment=5;
    var oset=0;
    while (true) {
        const response = await fetch(`https://api.iex.cloud/v1/data/core/news/msft?from=2023-04-27&to=2023-04-28&offset=${oset}&last=${increment}&token=${token}`);
        const data = await response.json();
        if (data.length > 0) {
            console.log(
                `${increment} most recent articles starting from article ${oset + 1}`);
            data.forEach((article) => {
                console.log(
                    `${new Date(article.datetime).toISOString()},  Headline: ${article.headline}`);
            });
            oset+=increment;
        }
        else {
            break;
        }
    }
}

yourFunction();

Command:

You can save the example code to a script and run the script in NodeJS.

node your-script.js

The above code uses the node-fetch package to make HTTP requests. It also passes in an API token via a variable.

Note

Replace YOUR_TOKEN above with your own API token.

The main difference between the code above and the apperate.queryData() code example are these two lines:

const response = await fetch(`https://api.iex.cloud/v1/data/core/news/msft?from=2023-04-27&to=2023-04-28&offset=${oset}&last=${increment}&token=${token}`);
const data = await response.json();

The fetch call operates on a GET /data endpoint. The endpoint URL includes the following parameters.

Path Parameters:

core workspace
news dataset ID
msft key index

Query Parameters:

from date
to date
offset value
last number of records to get
token API token

const data = await response.json(); gets the query results from the response object.

All the other logic is the same as the first example–it prints five news articles in each loop iteration.

Getting Oldest Records First#

If you’d rather work with records in ascending time order, use the first parameter instead of last.

For example, the following query returns the first (oldest) five Microsoft news articles from 2023-04-27.

const data = await client.apperate.queryData({
    workspace: "core", id: "news", key: "msft",
    from: "2023-04-27", to: "2023-04-28",
    first: 5, offset: oset});

Note

The GET /data endpoint and apperate.queryData() method return oldest records last by default.

Handle Data Your Way#

You can use the simple code examples above however you want as a springboard for data chunking/paging. Here are key things to set your way:

Data endpoint - Query data from any IEX Cloud Data you’ve purchased access to (see Data Bundles) or from any accessible workspace endpoints.
Offset increment - Set the number of records you want from each query via a last (or first) parameter setting.

And of course, do whatever you want with your query results!

Time Windowing#

Time-windowing your chunk/page queries prevents new data records from affecting the records you expect to get based on your offset.

Whenever you offset a data query, you must account for new records. If you’re getting the last (most recent) records and using an offset, any new records since your last query sets your offset back by the number of new records.

You can avoid this behavior by confining your results to a time window, using the GET /data endpoint time related query parameters.

Note

The apperate.queryData() iexjs method uses the GET /data endpoint time related query parameters underneath.

What’s Next#

Batch Data Queries

Processing Events

Creating a Rule Using the API

Write Data in Real-time