Paginate with Chunks of Data#

If you’re querying large amounts of data or if you’re paginating data in your application, it can be helpful to iterate over set chunks/pages of that data. IEX Cloud’s Data API makes this a snap.

For example, here’s output from working with chunks of five news articles at a time.

5 most recent articles starting from article 1
2023-04-27T23:24:07.000Z,  Headline: Microsoft’s US$69bil Activision deal blocked
2023-04-27T23:17:11.000Z,  Headline: Microsoft Is Ending Windows 10 Updates - CNET - [Sepe.gr]
2023-04-27T22:47:51.000Z,  Headline: Futures: Amazon Erases Gains On Cloud Fears; These Techs Dive
2023-04-27T22:21:20.000Z,  Headline: How to buy Microsoft stock (MSFT)
2023-04-27T22:01:55.000Z,  Headline: UK leaves Microsoft’s US$69bn takeover bid in peril
5 most recent articles starting from article 6
2023-04-27T21:55:08.000Z,  Headline: Microsoft’s Activision deal is on life support because cloud gaming still sucks
2023-04-27T21:26:06.000Z,  Headline: Microsoft Is Ending Windows 10 Updates
2023-04-27T21:20:00.000Z,  Headline: Microsoft Is Ending Windows 10 Updates - CNET
2023-04-27T21:16:51.000Z,  Headline: SCHG: Heavier Exposure To Apple, Microsoft, And Google
2023-04-27T21:16:17.000Z,  Headline: UK Watchdog Blocks Microsoft''s Acquisition Of Activision Blizzard
5 most recent articles starting from article 11
2023-04-27T21:15:31.000Z,  Headline: Microsoft blasts Britain over decision to block £60bn Call of Duty deal
// ... more articles

You can specify whatever number of records you need in your data chunks.

Here we’ll demonstrate working with pages/chunks of data using the GET /data REST endpoint directly.

Example: Using GET /data#

You can also get records in chunks using the GET /data REST endpoint directly.

Code:

'use strict';
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));

async function yourFunction() {
    const token="YOUR_TOKEN";
    const increment=5;
    var oset=0;
    while (true) {
        const response = await fetch(`https://api.iex.cloud/v1/data/core/news/msft?from=2023-04-27&to=2023-04-28&offset=${oset}&last=${increment}&token=${token}`);
        const data = await response.json();
        if (data.length > 0) {
            console.log(
                `${increment} most recent articles starting from article ${oset + 1}`);
            data.forEach((article) => {
                console.log(
                    `${new Date(article.datetime).toISOString()},  Headline: ${article.headline}`);
            });
            oset+=increment;
        }
        else {
            break;
        }
    }
}

yourFunction();

Executing the Code:

You can save the example code to a script and run the script in NodeJS.

node your-script.js

The above code uses the node-fetch package to make HTTP requests. It also passes in an API token via a variable.

Note

Replace YOUR_TOKEN above with your own API token.

The main difference between the code above and the apperate.queryData() code example are these two lines:

const response = await fetch(`https://api.iex.cloud/v1/data/core/news/msft?from=2023-04-27&to=2023-04-28&offset=${oset}&last=${increment}&token=${token}`);
const data = await response.json();

The fetch call operates on a GET /data endpoint. The endpoint URL includes the following parameters.

Path Parameters:

  • core workspace

  • news dataset ID

  • msft key index

Query Parameters:

  • from date

  • to date

  • offset value

  • last number of records to get

  • token API token

const data = await response.json(); gets the query results from the response object.

All the other logic is the same as the first example–it prints five news articles in each loop iteration.

Getting Oldest Records First#

If you’d rather work with records in ascending time order, use the first parameter instead of last.

For example, the following query returns the first (oldest) five Microsoft news articles from 2023-04-27.

const data = await client.apperate.queryData({
    workspace: "core", id: "news", key: "msft",
    from: "2023-04-27", to: "2023-04-28",
    first: 5, offset: oset});

Note

The GET /data endpoint and apperate.queryData() method return oldest records last by default.

Handle Data Your Way#

You can use the simple code examples above however you want as a springboard for data chunking/paging. Here are key things to set your way:

  • Data endpoint - Query data from any IEX Cloud Data you’ve purchased access to (see Data Bundles).

  • Offset increment - Set the number of records you want from each query via a last (or first) parameter setting.

And of course, do whatever you want with your query results!

Time Windowing#

Time-windowing your chunk/page queries prevents new data records from affecting the records you expect to get based on your offset.

Whenever you offset a data query, you must account for new records. If you’re getting the last (most recent) records and using an offset, any new records since your last query sets your offset back by the number of new records.

You can avoid this behavior by confining your results to a time window, using the GET /data endpoint time related query parameters.

Note

The apperate.queryData() iexjs method uses the GET /data endpoint time related query parameters underneath.

What’s Next#

Batch Data Queries