DynamoDB Pagination: Signed Cursors, Sorting, and the Count Problem

DynamoDB’s native Query API has a single pagination primitive: the response carries a LastEvaluatedKey, and you feed it back as ExclusiveStartKey to read the next page (PartiQL’s ExecuteStatement wraps the same mechanics behind a NextToken). There is no OFFSET, no ORDER BY over arbitrary attributes, and no cheap COUNT. For list endpoints on top of DynamoDB, the contract worth shipping is an opaque, HMAC-signed, scope-bound cursor with next and previous navigation and no total count in the response; sorting comes from the sort key, and every additional sort order is a deliberately provisioned GSI. The mechanics below use TypeScript and the AWS SDK for JavaScript v3. The patterns follow from documented API behavior; I have not yet run them under load, so the final section lists what to instrument once the contract ships.

What Limit and LastEvaluatedKey Actually Promise#

A few sentences from the official documentation determine most pagination behavior. The Query API reference (opens in new tab) defines Limit as “the maximum number of items to evaluate (not necessarily the number of matching items)”. A single call reads items until it reaches that limit or 1 MB of data, whichever comes first, and only then applies any FilterExpression. On the question of when you are done, the developer guide (opens in new tab) is explicit: “If LastEvaluatedKey is not empty, it does not necessarily mean that there is more data in the result set. The only way to know when you have reached the end of the result set is when LastEvaluatedKey is empty.”

Two consequences follow. First, Limit: 20 with a filter does not produce 20 items in the response. DynamoDB evaluates 20 items, the filter then removes the non-matching ones, and the page can carry 3 items, or 11, or zero, together with a non-null cursor. The termination condition for any pagination loop is therefore the absence of LastEvaluatedKey, never an empty Items array. A client loop written as while (items.length > 0) stops early and silently truncates results.

Second, filters shrink the payload while the bill stays the same. The filter expression documentation (opens in new tab) notes that a filter is applied after the read completes, so the capacity consumed covers everything scanned, including items the filter throws away. The Query response (opens in new tab) reports both numbers: ScannedCount is what was read, Count is what survived. The gap between them is read capacity you paid for and discarded.

A Cursor Page in TypeScript#

With @aws-sdk/lib-dynamodb, a cursor-paged list endpoint looks like this. The encodeCursor and decodeCursor functions come in the next section.

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, QueryCommand } from '@aws-sdk/lib-dynamodb';
import { encodeCursor, decodeCursor } from './cursor';

const doc = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const MAX_PAGE_SIZE = 100;

export async function listOrders(customerId: string, cursor?: string, pageSize = 20) {
  // Never trust a client-supplied page size: clamp it server-side.
  const limit = Math.min(Math.max(pageSize, 1), MAX_PAGE_SIZE);
  // The scope ties the cursor to this user and this exact query shape:
  // logical query name, index (base table here), direction, tenant.
  const scope = `orders:base:desc:${customerId}`;

  const res = await doc.send(
    new QueryCommand({
      TableName: 'app',
      KeyConditionExpression: 'pk = :pk AND begins_with(sk, :prefix)',
      ExpressionAttributeValues: {
        ':pk': `CUSTOMER#${customerId}`,
        ':prefix': 'ORDER#',
      },
      ScanIndexForward: false, // newest first; keep identical for every page
      Limit: limit,
      ExclusiveStartKey: cursor ? decodeCursor(cursor, scope) : undefined,
      ReturnConsumedCapacity: 'TOTAL',
    }),
  );

  return {
    items: res.Items ?? [],
    // null is the only honest "you are done" signal
    nextCursor: res.LastEvaluatedKey ? encodeCursor(res.LastEvaluatedKey, scope) : null,
  };
}

Two details in this sketch carry weight. ScanIndexForward must stay identical for every request in one pagination session, because the cursor encodes a position in a directed traversal; flipping the direction mid-session skips or duplicates items. And nextCursor: null is the only completion signal the endpoint can honestly give, for the reasons covered above. The handler also returns only nextCursor: previous-page navigation is its own reversed traversal, described in the breakage list near the end, and it mints its cursor from the first item of the current page rather than the last.

When the query targets a GSI, the returned LastEvaluatedKey contains more than the index keys. The GSI documentation (opens in new tab) notes that index key values do not need to be unique and that the base table’s primary key attributes are always projected into the index, so the key also carries the table’s partition and sort key attributes. Cursor validation, attribute allow-listing, and logging all need to account for that; a system that logs cursors on a GSI-backed endpoint is logging table keys.

For internal jobs that drain a result set instead of serving a page, skip the hand-rolled loop: paginateQuery from @aws-sdk/lib-dynamodb wraps the LastEvaluatedKey cycle in an async iterator, and AWS’s pagination example (opens in new tab) shows the same shape across SDKs. If your data layer runs on DynamoDB Toolbox, the DynamoDB Toolbox guide shows where a cursor encoder plugs into that stack; the snippet there ships bare base64url, so give it the signing treatment from the next section.

Sign the Cursor, Bind the Scope#

What should the cursor string itself be? The ecosystem’s public positions cluster around two poles. ElectroDB (opens in new tab), one of the most widely used typed DynamoDB clients, documents its cursor as a base64url-encoded copy of LastEvaluatedKey, and most tutorials ship the same bare encoding. At the other end, @emdgroup/dynamodb-paginator (opens in new tab) encrypts and signs its tokens, arguing that JSON-encoded tokens let clients read and modify key values. AWS itself sits in the second camp for its managed GraphQL layer: the AppSync resolver reference (opens in new tab) states that AppSync “encrypts and obfuscates the pagination token returned from DynamoDB” to keep table data from leaking to the caller, and adds that “these pagination tokens cannot be used across different resolvers”.

My recommendation is the middle position: sign the cursor with HMAC-SHA256 over the base64url payload plus a scope string, and reserve encryption for the case where the key values themselves are sensitive. Be precise about what signing buys, because this is where implementations overclaim: an HMAC provides integrity and authenticity, not confidentiality. Anyone holding a signed cursor can still base64url-decode the payload and read the key inside, so “opaque” here is a contract that clients must not interpret the token, enforced by rejecting the ones that tamper. It is not concealment. The options stack up like this:

Cursor format	Tamper-proof	Content hidden	Server-side state
Bare base64url	No	No	None
HMAC-signed base64url	Yes	No	None
Authenticated encryption (AEAD)	Yes	Yes	None
Random handle, position stored server-side	Yes	Yes	A cursor store to operate

Bare base64 fails the first column, and that failure is the serious one: the token is client-editable, so ExclusiveStartKey becomes attacker-controlled input to your data layer, and the readable key schema becomes a de facto public API that clients will parse and depend on. Signing closes the tampering hole; it does not stop a client from reading the schema, which only the last two rows do. Full encryption adds key management and extra crypto surface to hide something that in most schemas is a CUSTOMER#id prefix, which is hardly a secret, and a server-side handle brings back the very state that cursors exist to avoid. If your keys embed email addresses or other personal data, encryption stops being optional; at that point use a vetted library or your platform’s crypto layer rather than hand-rolling AES. The emdgroup paginator demonstrates the pattern, but its development has been dormant since 2023, so treat it as a reference implementation rather than a dependency.

The scope string is the part most implementations skip, and it is what AppSync’s cross-resolver restriction hints at. Include everything that defines what the position means: the tenant or user, the logical query name, the index, and the traversal direction. A cursor minted for one user’s order list must fail verification when replayed against another user’s list, and a cursor from query A must fail on query B.

import { createHmac, timingSafeEqual } from 'node:crypto';

export class CursorError extends Error {}

const SECRET = process.env.CURSOR_SECRET;
if (!SECRET) throw new Error('CURSOR_SECRET is not set');

const MAX_AGE_MS = 15 * 60 * 1000; // one browsing session, not forever

function sign(payload: string, scope: string): Buffer {
  return createHmac('sha256', SECRET).update(`${scope}.${payload}`).digest();
}

export function encodeCursor(key: Record<string, unknown>, scope: string): string {
  const payload = Buffer.from(JSON.stringify({ key, iat: Date.now() })).toString('base64url');
  return `${payload}.${sign(payload, scope).toString('base64url')}`;
}

export function decodeCursor(token: string, scope: string): Record<string, string | number> {
  const [payload, mac] = token.split('.');
  if (!payload || !mac) throw new CursorError('malformed cursor');

  const given = Buffer.from(mac, 'base64url');
  const expected = sign(payload, scope);
  // `given` comes from the client, so its length is attacker-controlled.
  // timingSafeEqual throws on length mismatch: compare lengths first.
  if (given.length !== expected.length || !timingSafeEqual(given, expected)) {
    throw new CursorError('invalid signature');
  }

  const parsed = JSON.parse(Buffer.from(payload, 'base64url').toString()) as {
    key: Record<string, string | number>;
    iat: number;
  };
  if (typeof parsed.iat !== 'number' || Date.now() - parsed.iat > MAX_AGE_MS) {
    throw new CursorError('cursor expired');
  }
  return parsed.key;
}

The iat timestamp matters because a cursor marks a position in an index; nothing about it snapshots the data, and the signature alone would keep it syntactically valid forever. Binding an issued-at time into the signed payload caps how long a stale or leaked token stays usable. The cap is a hygiene measure, not a consistency mechanism: within the window, concurrent writes still move items across page boundaries, so a paginating client can see an item twice or miss one. Cursor pagination over live data is best-effort by construction, with or without an expiry.

The server-side costs of this default are real but small: one HMAC verification for the inbound cursor plus one signing for the outbound one per request, a secret to store and rotate, and a new error path (an HTTP 400 with an invalid-cursor code) that clients must answer by restarting from the first page. The UX costs deserve the same honesty: no jumping to an arbitrary page, no visible result size, and once the cursor expires the browser back button restarts the list from page one. Rotation is the sharpest edge, because a new secret invalidates every in-flight cursor at once. In practice you verify against both the current and the previous key during a rotation window, and sign only with the current one; a version field in the payload buys the same grace for format changes.

Filling a Fixed Page Size Under a Filter#

Sometimes the API contract fixes the page size, the filter cannot move into the key condition today, and a page sequence of 3, then 0, then 11 items is unacceptable for the UI. The workable mitigation is an internal loop with a hard round cap:

import { QueryCommand, type QueryCommandInput } from '@aws-sdk/lib-dynamodb';

const MAX_ROUNDS = 5; // hard cap: without it one request can walk an entire partition

// `doc` is the shared DynamoDBDocumentClient from the previous section.
export async function fillPage(base: QueryCommandInput, pageSize: number) {
  const collected: Record<string, unknown>[] = [];
  let startKey = base.ExclusiveStartKey;
  let rounds = 0;

  while (collected.length < pageSize && rounds < MAX_ROUNDS) {
    const res = await doc.send(
      new QueryCommand({
        ...base,
        ExclusiveStartKey: startKey,
        Limit: (pageSize - collected.length) * 3, // over-fetch: the filter discards some
      }),
    );
    collected.push(...(res.Items ?? []));
    startKey = res.LastEvaluatedKey;
    rounds += 1;
    if (!startKey) break; // the index range is genuinely exhausted
  }

  const page = collected.slice(0, pageSize);
  const truncated = collected.length > page.length;
  const last = page.at(-1);

  return {
    items: page,
    // If surplus items were cut, LastEvaluatedKey points past rows the caller
    // never received. Resume from the last item actually returned instead.
    // Assumes string base-table keys named pk and sk: derive this from your
    // own schema, add the index key attributes on a GSI, and give binary
    // keys their own serialization.
    nextKey: truncated && last ? { pk: last.pk, sk: last.sk } : startKey,
  };
}

Three notes on this loop. The over-fetch multiplier is a tuning guess with no correct constant; it only reduces the expected number of rounds. The resume key is the subtle part: after slicing surplus items, LastEvaluatedKey points past rows the response never returned, so resuming from it would skip them, and deriving the key from the last returned item keeps the cursor honest. On a GSI that derived key must include both the index key attributes and the table key attributes. Finally, the cap is load-bearing: without MAX_ROUNDS, one request against an unfavorable filter walks an entire partition at your expense.

The loop is a mitigation, and it doubles as a diagnostic. If it routinely exhausts its rounds, the filter is doing work an index should do, and the predicate belongs in the key condition or in a GSI.

Sorting Comes From the Sort Key#

Query results arrive ordered by sort key value: numbers numerically, strings by UTF-8 byte order, and ScanIndexForward: false reverses the traversal. That is the entire sorting feature DynamoDB offers. Everything beyond it is data modeling:

• Byte order has sharp edges. The string "10" sorts before "9", so numeric values stored in string sort keys need zero-padding to a fixed width. ISO-8601 timestamps sort correctly precisely because their lexicographic order matches chronological order; keep them in UTC with a fixed format.
• Each additional sort order is an index. A GSI whose sort key is the attribute you need ordered gives you a second sort, at the price of extra storage and an index write for every mutation of the base item. Not every new order literally needs one: the reverse direction is free via ScanIndexForward, a composite sort key such as STATUS#DATE#ID serves several range shapes, and a small bounded collection can sort in application code. What costs an index is the recurring case, an independent attribute that must order a large collection. Because that cost recurs on every write, “add a sortBy parameter” is never a free feature on DynamoDB. Choosing the keys themselves is covered in the single-table design guide.
• Sparse indexes can filter and sort in one move. AWS’s sparse index guidance (opens in new tab) suggests an attribute that exists only while it is relevant and whose value orders usefully, such as an open-date attribute that is removed when an order is fulfilled. The sparse GSI then contains only open orders, already sorted by date.
• One overloaded GSI can serve several shapes. GSI overloading (opens in new tab) uses the table’s sort key as the index partition key and a generic attribute as the index sort key, so one index answers multiple query patterns.
• Equal index keys have no defined order. A GSI accepts multiple items with the same partition and sort key values, and their relative order in results is not guaranteed. A UI that needs a stable sequence appends a unique tie-breaker to the sort key, such as 2026-08-23T10:12:03Z#ORDER-7431.
• The hard boundary: sorting exists within one partition key. No single Query can order a result set that spans partitions. Bounded workarounds exist, a GSI with one fixed partition value (which caps write throughput at a single partition) or sharded index partitions merged in the application, so “sort all users by last name” is not impossible. As an open-ended requirement, though, it points at the search-index override discussed below.

The Total Count Problem#

The paginator widget wants to render page 7 of 42, and the 42 requires a total. DynamoDB offers five ways to produce one, and each pays differently:

Mechanism	Freshness	What it costs	Where it fits
Select: 'COUNT' per request	Fresh at read time, not a snapshot across pages	Read capacity for every item scanned, 1 MB per call	Small collections bounded inside one partition
DescribeTable ItemCount	Roughly six hours stale	Free	Dashboard tiles, capacity planning
Transactional counter item	Exact while every write path uses the transaction	A second write plus transaction overhead per mutation, hot-item risk	Counts that are product features
Streams-fed aggregate	Seconds behind, lag varies	A Lambda consumer you operate and monitor	High-write aggregation at scale
OpenSearch replica	Replication lag	A second datastore and an ingestion pipeline	Arbitrary sort, full-text search, exact hit counts

Select: 'COUNT' looks like the built-in answer and is the most commonly misunderstood one. It returns counts instead of items, but it still reads every item it counts, still stops at the 1 MB boundary per call, and still returns a LastEvaluatedKey you must follow; the capacity consumed matches fetching the data. Reads are also eventually consistent by default, and a GSI cannot be read strongly consistently at all, so the number is only as fresh as the reads behind it. For a bounded collection inside one partition that is fine. As a table-wide count it degenerates into a full Scan on every page load.

DescribeTable costs nothing but answers a different question. The API reference (opens in new tab) states that DynamoDB updates ItemCount “approximately every six hours” and that recent changes may not be reflected. It works for a dashboard tile showing an approximate item count and fails as a paginator input.

Maintained counters are exact but earn their keep only as product features. UpdateItem with ADD is atomic, yet the developer guide (opens in new tab) is direct about the catch: atomic counter updates are not idempotent, so a retried increment counts twice. The fix is to group the item write and the counter increment into one TransactWriteItems (opens in new tab) call with a client request token, which makes the transaction idempotent for the token’s ten-minute window. Exactness also has a perimeter: every write path must run through the same transaction, and anything that bypasses it, a TTL expiry, a bulk import, a manual fix, drifts the counter until a reconciliation job corrects it. Now every mutation costs two writes plus transaction overhead, and a counter shared by a high-throughput tenant becomes a hot item; the rate-limit strategies post shows what a hot counter item does under burst and how write-sharding spreads the load.

At larger write volumes, AWS’s materialized-aggregation pattern (opens in new tab) moves counting off the write path: DynamoDB Streams feed a Lambda that maintains aggregate items. The count now trails the table, typically by seconds but with no hard upper bound when the consumer errors or backlogs, and you operate a consumer with a dead-letter queue and iterator-age monitoring, in exchange for an unburdened write path.

When the requirement is arbitrary sorting plus exact hit counts plus full-text matching, the answer stops being DynamoDB alone. The zero-ETL integration with OpenSearch (opens in new tab) replicates the table through an initial point-in-time export and ongoing Streams changes, and the replica answers search-shaped questions natively. Its hit counts are exact for the index’s own state, which trails the source table by the replication lag.

For most product list UIs, though, the right move is to drop the total. What the interface needs is “there is more” and “there is a previous page”. A non-null cursor alone does not quite deliver the first: as covered at the top, it only promises that the traversal can continue, and the next call may come back empty. A true hasNextPage costs exactly one extra item: read until pageSize + 1 matches surface, return the first pageSize, and let the surplus answer the question; the fill loop above produces it with its target raised by one. The GraphQL Cursor Connections Specification (opens in new tab), the contract behind Relay-style pagination, works this way: pageInfo carries hasNextPage and hasPreviousPage, and no total field is required, though the spec also permits a total field for products that need one. The strongest counter-argument comes from Alex Reid’s write-up (opens in new tab) on numbered page URLs: when the product genuinely requires stable, shareable page links, you can precompute and store the page-break keys and serve them. The mechanics work. The cost in that write-up is a second datastore holding the page-break index, kept in sync by a Lambda consuming the table’s Stream, which is exactly the kind of machinery worth refusing until someone demonstrates the requirement.

When to Override the Default#

Each override is a real case, and each has a narrower footprint than it first appears. A collection bounded inside one partition, such as a user’s payment methods or a project’s members, makes per-request Select: 'COUNT' affordable, and an exact count is then harmless. A count that is itself a product feature (a quota, a billing meter, an unread badge) deserves a maintained counter with transactional writes, or Streams aggregation once write volume makes transactions expensive. A requirement to sort by attributes you cannot enumerate up front is a search workload; replicate to OpenSearch instead of adding a fifth GSI. Shareable numbered URLs, if the requirement survives scrutiny, take the precomputed page-break route. Back-office and audit screens are the version of that requirement that most often survives: support staff really do navigate by position and result size, and those internal collections are usually bounded enough to keep the cost contained. None of these overrides removes the cursor itself: even the OpenSearch path returns pagination state that deserves the same signing and scoping.

Where Pagination Code Breaks#

• Stopping on an empty page. An empty Items array with a non-null cursor is a normal page. Loop until the key is absent.
• Treating Limit as a page-size promise. It caps evaluation, and filters run after the read. Clamp and fill in the API layer instead.
• Raw LastEvaluatedKey in the query string. Partition keys end up in browser history, CDN and proxy logs, and Referer headers, and the token is editable by anyone holding it. Sign and scope the cursor and treat every inbound one as untrusted input; and since a signature does not hide the payload, reach for encryption when the key values themselves must not appear in those logs.
• One cursor, two queries. A cursor is meaningful only for the exact index, key condition, filter, and direction that minted it. Bind the query shape into the signed scope and reject mismatches; otherwise a client that changes a filter parameter mid-pagination gets silently wrong pages.
• Implementing previous-page by flipping ScanIndexForward on the same cursor. Reversing direction mid-session changes what the position means. Run the whole backward traversal as its own session from a cursor captured at the page boundary, then reverse the page array before returning it.
• Scan-based exports under live writes. A Scan cursor marks a position in key order and carries no snapshot guarantee, so concurrent writes can move items relative to it and an export can duplicate or miss rows. For consistent exports, a point-in-time export to S3 replaces the live Scan.
• Unclamped client page size. A missing ceiling turns one request into an arbitrarily expensive read. Clamp server-side, as in the handler above.

What to Measure Before Trusting the Contract#

None of the patterns above have been load-tested yet; that work starts after the contract ships. The instrumentation plan that makes the pagination layer observable is short. Track ConsumedCapacity per API page at p50 and p99: a page whose cost swings by an order of magnitude between requests means a filter is doing an index’s job. Track the Count to ScannedCount ratio per query; as a rough rule of thumb, when much less than half of what is scanned survives the filter, the endpoint is paying for reads it discards. Track internal rounds per API page, which should sit at 1, because a persistently higher number means the fill loop is masking a modeling problem. Add the empty-page rate (zero items with a non-null cursor) and the cursor rejection rate, where a sudden spike usually means a forgotten secret rotation or someone probing the endpoint.

The default holds for product list endpoints whose items live under a partition key’s natural order: signed, scope-bound cursors, next and previous navigation, no total. Reach for an override when a count is a product feature, when sorting must span partitions, or when stable numbered URLs are genuinely required by the product. The one step worth taking immediately is writing the cursor contract into the API spec (opaque token, expiry, the invalid-cursor error code) before a frontend builds a paginator around guarantees DynamoDB does not offer.

References#

• Query - Amazon DynamoDB API Reference (opens in new tab) - Definitive wording for Limit as items evaluated, the 1 MB read cap, ExclusiveStartKey, and ScanIndexForward
• Paginating table query results in DynamoDB (opens in new tab) - Why a non-empty LastEvaluatedKey does not prove more data exists, and the only correct stop condition
• Filter expressions for the Query operation (opens in new tab) - Filters run after the read, so they shrink the payload without reducing consumed capacity
• Using Global Secondary Indexes in DynamoDB (opens in new tab) - Index key values need not be unique, and the base table’s primary key attributes are always projected into the index
• TableDescription - Amazon DynamoDB API Reference (opens in new tab) - ItemCount is refreshed approximately every six hours, which rules it out as a paginator input
• Working with items and attributes in DynamoDB (opens in new tab) - Atomic counters and why their updates are not idempotent under retries
• TransactWriteItems - Amazon DynamoDB API Reference (opens in new tab) - Grouping an item write and a counter increment into one idempotent transaction with a client request token
• Take advantage of sparse indexes (opens in new tab) - Index key attributes whose values also produce a useful sort order
• Overloading Global Secondary Indexes (opens in new tab) - Serving several query and sort shapes from a single index
• Using GSIs for materialized aggregation queries (opens in new tab) - AWS’s Streams-based pattern for maintaining aggregates such as counts
• DynamoDB zero-ETL integration with Amazon OpenSearch Service (opens in new tab) - Replication via a point-in-time export plus Streams, for search-shaped read requirements
• AWS AppSync resolver reference for DynamoDB Query (opens in new tab) - AppSync encrypts and obfuscates its pagination token and scopes it to a single resolver
• Query a DynamoDB table with pagination using an AWS SDK (opens in new tab) - Official hand-rolled pagination-loop examples across SDK languages
• emdgroup/dynamodb-paginator (opens in new tab) - Encrypted and signed pagination tokens, and the argument against plain JSON tokens; dormant since 2023, cited for the argument rather than as a dependency
• ElectroDB: Pagination (opens in new tab) - A widely used typed client whose cursor is a base64url copy of LastEvaluatedKey
• DynamoDB pagination with page numbers in URLs (opens in new tab) - Precomputing page-break keys to serve stable numbered page URLs
• GraphQL Cursor Connections Specification (opens in new tab) - The pageInfo contract that maps onto DynamoDB cursors and requires no total count