Connect to Data via the Connectors API#

The Connectors API connects your Apperate workspace to various data sources including Coinbase BETA, Ethereum, and MySQL. A Data Source Connector (Connector) is a configured connection from Apperate to a specified data source. You’ll use connectors to enable data streaming from a source to an Apperate dataset.

Tip

You can create Rules that filter on events in the stream and that send alerts/output on those events.

Here’s what you’ll learn how to do with Connectors:

  • Get the connector schema: Get the latest schema for creating the connector type you want to configure.

  • Create a connector: Configure a connection for streaming data from a supported source to an Apperate dataset.

  • Discover streams and select one: Data sources have one or more streams–the kinds of data the source provides. You’ll learn how to list a source’s streams and set a stream for your connector.

  • Start the connector: Enable connector data flow into Apperate.

  • Check connector status: Monitor connector status: pending data stream names, received data stream names, ready to run, running, or stopped.

  • Stop the connector: Disable connector data flow into Apperate.

Get the Connector Schema#

Creating a connector requires posting an object that fits one of the following data source types:

  • COINBASE

  • ETHEREUM

  • MYSQL

You can get all the schemas by calling the GET /connectors endpoint or get an specific type’s schema by passing in the type as a parameter to the GET /connectors/:type? endpoint.

For example, the Coinbase schema looks like this:

[
  {
    "type": "object",
    "properties": {
      "id": {
        "type": "string",
        "minLength": 1,
        "pattern": "^[a-zA-Z0-9-_]+$",
        "title": "Coinbase Tap"
      },
      "datasetId": {
        "type": "string",
        "minLength": 0
      },
      "wait": {
        "type": "boolean",
        "default": "true"
      },
      "connectorType": {
        "type": "string",
        "enum": [
          "MYSQL",
          "COINBASE",
          "ETHEREUM"
        ]
      },
      "state": {
        "type": "object",
        "properties": {
          "status": {
            "type": "string",
            "enum": [
              "pending",
              "received",
              "ready",
              "running",
              "stopped",
              "errored"
            ]
          }
        },
        "required": []
      },
      "config": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string",
            "default": "wss://ws-feed.exchange.coinbase.com"
          },
          "productId": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        },
        "required": [
          "productId",
          "url"
        ]
      }
    },
    "required": [
      "config",
      "connectorType",
      "wait"
    ],
    "additionalProperties": true,
    "title": "Coinbase Tap Schema"
  }
]

The Coinbase schema declares that you must pass in these elements:

  • config: Consists of a url to Coinbase and an array of productId elements.

  • connectorType: Use the enumerated value COINBASE. ETHEREUM and MYSQL are also valid – but you’d have to abide by their schemas.

  • wait: If set to true, the connector writes data synchronousy and waits for confirmation; otherwise it writes data asynchronously which is faster.

Create a Connector#

You can create a connector by making an HTTP request to the POST /connectors/connector/:connectorName endpoint that includes a body (JSON) describing your connector, per its schema.

Coinbase Example (in Bash shell):

curl -H "Content-Type: application/json" \
-X POST "https://api.iex.cloud/v1/connectors/connector/YOUR_CONNECTOR?token=YOUR_SECRET_TOKEN" \
--data-binary @- << EOF
{
  "datasetId": "YOUR_DATASET",
  "config": {
    "url": "wss://ws-feed.exchange.coinbase.com",
    "productId": ["BTC-USD", "LINK-USD"]
  },
  "connectorType": "COINBASE",
  "wait": true
}
EOF

Response:

{
  "success": true,
  "message": "created connector 'YOUR_CONNECTOR'"
}

The above request creates a connector for streaming data from an Coinbase endpoint to a dataset called YOUR_DATASET.

The connector requests the names of the data source streams.

Note

Data does not yet stream through the connector until you select one of the source’s streams and set the connector status to running (described further below).

Let’s find out the source’s streams.

Discovering Streams and Selecting One#

The GET /connectors/connector/:connectorName endpoint returns information about the connector, including its status and the names of the streams it can access.

Example:

curl -X GET https://api.iex.cloud/v1/connectors/connector/YOUR_CONNECTOR?token=YOUR_SECRET_TOKEN&displayConfigs=true

Response:

[
  {
    "connectorType": "COINBASE",
    "datasetId": "YOUR_DATASET",
    "description": "",
    "id": "YOUR_CONNECTOR",
    "state": {
      "jobId": "your-connector",
      "status": "received"
    },
    "streams": [
      {
        "id": "ticker"
      }
    ],
    "updated": "2023-03-24T14:56:10.000Z",
    "wait": false
  }
]

The connector has one of these statuses:

  • pending: The connector is waiting to receive the names of the data source’s streams.

  • received: The connector has received the stream names; the streams are listed in the response’s streams element.

When the connector has received the stream names, you can pick one to connect to.

Let’s update the connector with the stream we want using the PUT /connectors/connector/:connectorName endpoint.

Example:

curl -H "Content-Type: application/json" \
-X PUT "https://api.iex.cloud/v1/connectors/connector/YOUR_CONNECTOR?token=YOUR_SECRET_TOKEN" \
-d '{"selectedStream": "ticker"}'

Response:

{
  "id": "YOUR_CONNECTOR",
  "state": {
    "jobId": "your-connector",
    "status": "ready"
  }
}

The above request selects the Coinbase endpoint’s blocks stream.

The connector’s status is now ready because it’s ready to connect to the stream. All that’s left is to start the connector.

Starting a Connector#

You can start the connector by making a PUT /connectors/connector/:connectorName request with status set to running in the body.

Example:

curl -H "Content-Type: application/json" \
-X PUT "https://api.iex.cloud/v1/connectors/connector/YOUR_CONNECTOR?token=YOUR_SECRET_TOKEN" \
-d '{"status": "running"}'

Response:

{
  "id": "YOUR_CONNECTOR",
  "state": {
    "jobId": "your-connector",
    "status": "running",
    "execId": "projects/iexcloud-1/locations/us-central1/jobs/your-connector/executions/your-connector-lzfz8"
  }
}

Data is flowing from the source into Apperate!

Note

Data can take a minute or so to start flowing into your dataset.

Here’s what Coinbase data looks like in the dataset Database page.

Stopping a Connector#

Stopping a connector is easy too; just set status to stopped.

Example:

curl -H "Content-Type: application/json" \
-X PUT "https://api.iex.cloud/v1/connectors/connector/YOUR_CONNECTOR?token=YOUR_SECRET_TOKEN" \
-d '{"status": "stopped"}'

You can stop and restart a connector all you want.

Deleting a Connector#

If you want to delete a connector, call the DELETE /connectors/connector/:connectorName endpoint.

Example:

curl -X DELETE https://api.iex.cloud/v1/connectors/connector/YOUR_CONNECTOR?token=YOUR_SECRET_TOKEN

What’s Next#

Now that you know how to connect with data using the Connectors API, you can work with the data in different ways. Here are some resources to get you started:

Create Views demonstrates enriching your data by joining datasets in views.

Processing Events shows you how to filter on specific events in your data and send notifications on them.