Fetching Large Result Sets¶

Async iterators¶

The driver supports asynchronous iteration of the ResultSet using the built-in Async Iterator, fetching subsequent pages automatically as the previous one has been consumed.

Large result sets can be iterated using the for await ... of statement:

const result = await client.execute(query, params, { prepare: true });

for await (const row of result) {
  console.log(row[columnName]);
}

Under the hood, the driver will get all the rows of the query result using multiple requests. Initially, when calling execute() it will retrieve the first page of results according to the fetch size (defaults to 5000). If there are additional rows, the driver fetches the next page automatically once the async iterator has yielded rows from the previous page.

If needed, you can use isPaged() method of ResultSet instance to determine whether there are more pages of results than initially fetched.

Each row callback¶

You can also iterate through pages by setting a per-page callback. eachRow() works in two modes, depending on the QueryOptions.autoPage configuration:

• Automatic paging (default): keeps fetching subsequent pages until all rows are processed.

• Manual paging: fetches one page at a time. This mode gives you access to result sets of intermediate pages. To fetch the next page, you need to call result.nextPage().

eachRow() calls:

• rowCallback(rowIndex, row) for each row as soon as it is received,

• callback(err, result):

  • when autoPage is set to true: after all pages are fetched and rowCallback was called for each row,

  • when autoPage is set to false: after the current page is fetched and rowCallback was called for each row,

  • when an error occurs.

client.eachRow(
  query,
  parameters,
  { prepare: true, autoPage },
  (rowIndex, row) => {
    // process row
  },
  (err, result) => {
    if (err) {
      // handle error
      return;
    }
    if (result.nextPage) {
      // Handle intermediate page result set. This branch will be taken only when autoPage is disabled.
      // When autoPage is enabled you will not have access to those result sets.
      result.nextPage();
    } else {
      // Handle last page result.
    }
  }
);

Row streams¶

If you want to handle a large result set as a Stream of rows, you can use stream() method of the Client instance. The stream() method automatically fetches the following pages, yielding the rows as they come through the network and retrieving the following page only after the previous rows were read (throttling).

client.stream(query, parameters, options)
  .on('readable', function () {
    // readable is emitted as soon as a row is received and parsed
    let row;
    while (row = this.read()) {
      // process row
    }
  })
  .on('end', function () {
    // emitted when all rows have been retrieved and read
  });