DynamoDB for Identity Reads, OpenSearch for Search: The Zero-ETL Read Model

DynamoDB list endpoints accumulate a recognizable set of artifacts as a product grows: a GSI per sort option, a fill loop compensating for post-read filters, counter items standing in for facet counts. Each artifact is a search-shaped read forced into a store built for key-range access. The default worth adopting is a routing rule, not a migration: identity-shaped reads (fetch this order, list this customer’s orders by date) stay in DynamoDB, while arbitrary sort, ad hoc filter combinations, facets, exact totals, and full text move to a read model fed by the DynamoDB zero-ETL integration with Amazon OpenSearch Service (opens in new tab). One override is loud enough to state up front: a team still free to choose its system of record is usually better served by a single PostgreSQL than by two stores. I have not run this pipeline yet; everything below follows documented behavior, with a measurement plan for the day the pipeline exists.

Identity-Shaped Reads and Search-Shaped Reads#

An identity-shaped read carries its own address: an order ID, a customer’s partition key plus a date range on a sort key that was designed for exactly that question. DynamoDB answers these cheaply and predictably at any scale, and that is not an accident of implementation. AWS describes the service (opens in new tab) as a key-value and document database without JOIN support, and its NoSQL design guidance (opens in new tab) is direct about the consequence: data can be queried efficiently only in a limited number of ways, and schema design should not begin until you know the questions the table will need to answer. Everything DynamoDB is good at follows from that framing: point gets, item collections under one partition key, range scans along a sort key modeled in advance.

A search-shaped read describes a result set instead of naming one: orders matching three filters chosen at request time, sorted by whichever column the user clicked, with facet counts in the sidebar and a jump to page 7. No key design serves that, because the question did not exist when the keys were chosen. Forcing it produces the same symptoms in every codebase. A fourth and fifth GSI appear, and while the default quota (opens in new tab) is 20 GSIs per table, the binding constraint arrives much earlier: every index adds a write for every mutation that touches its indexed or projected attributes, permanently, while query flexibility grows far more slowly than that spend. Fill loops page through post-read filters at full read cost. Counter items appear for every facet value. The organizational tell arrives last: every new report request turns into a data-modeling meeting, which means read requirements have stopped being enumerable.

That enumerability is the usable test. Count how many of your read patterns you could list a year ago and can still list today. A stable set means DynamoDB alone is fine, and the modeling techniques in the single-table design guide cover it. A set that grows every sprint is a search workload, and no amount of key modeling will catch up with questions composed at request time.

How to paginate the identity side (signed cursors, sort keys as the only sort, dropping the total count) is covered in the pagination post. Its decision tree ends with one branch pointing out of DynamoDB entirely, labeled arbitrary sort across partitions. What follows is that branch argued out, with the override branches included:

The Routing Seam in the API Layer#

The part that survives whichever read store wins is the seam itself: one dispatcher, two backends, one explicit rule. Writing it down as code makes the decision testable and keeps it from dissolving into per-endpoint improvisation.

type OrderPage = {
  items: unknown[];
  nextToken: string | null;
  // The freshness contract, not the store's name: clients render staleness
  // from this, and the backend can change without breaking them.
  consistency: 'strong' | 'eventual';
  stalePossible: boolean;
};

// Every list read the product supports, declared by name. A profile that
// was modeled into keys or a GSI stays in DynamoDB even though it filters
// or sorts; a profile is search-shaped because the key model never met it,
// not because a filter parameter happens to be present.
const orderViews = {
  customerNewest: { backend: 'dynamodb' },      // sort key, ScanIndexForward
  customerOpenByDate: { backend: 'dynamodb' },  // sparse GSI on open orders
  customerByPrice: { backend: 'dynamodb' },     // GSI modeled for exactly this sort
  catalogSearch: { backend: 'opensearch' },     // free text, facets, ad hoc filters
} as const;

type ViewName = keyof typeof orderViews;

// The two implementations live elsewhere; the decision is the point here.
declare function queryOrders(view: ViewName, params: unknown): Promise<OrderPage>;
declare function searchOrders(params: unknown): Promise<OrderPage>;

export function listOrders(view: ViewName, params: unknown): Promise<OrderPage> {
  const { backend } = orderViews[view];
  return backend === 'dynamodb' ? queryOrders(view, params) : searchOrders(params);
}

The registry is the enumerability test written as code. Adding a view is a deliberate act, and the diff shows whether the new question earned a key model or landed in the search profile; a request shape nobody registered is not silently routed anywhere, it is a missing product decision. One trigger is deliberately absent: a numbered page parameter by itself only implies from = (page - 1) * size, not a total, so page numbers alone do not force the search path; rendering “page 7 of 42” does.

Two consequences deserve to be explicit in the contract. First, the response carries a freshness contract rather than a store name. A DynamoDB answer is not automatically fresh, because reads are eventually consistent by default and GSI reads always are, while an OpenSearch answer sits behind replication plus the index refresh interval. consistency and stalePossible tell the client what it may render, and they keep working if a view later changes backends. Second, the search path still needs an opaque pagination token: search_after sort values leak internal data exactly the way a raw LastEvaluatedKey does, so they get the same treatment the pagination post applies to DynamoDB cursors: signing and scoping for integrity, plus encryption or a server-side handle when the values themselves must stay hidden, since a signature alone leaves them readable.

What the Zero-ETL Integration Actually Builds#

The integration runs in two phases. It first takes a point-in-time export of the table to S3 and bulk-loads that snapshot into the index, then tails DynamoDB Streams through an OpenSearch Ingestion pipeline to replicate ongoing changes. The prerequisites follow from the phases: point-in-time recovery must be enabled for the export, and Streams must be enabled for the CDC tail. Neither phase competes with production traffic for table capacity, because the export reads from the PITR backup rather than the table and the change feed comes from Streams.

The pipeline configuration (opens in new tab) is YAML in Data Prepper’s format:

version: "2"
cdc-pipeline:
  source:
    dynamodb:
      acknowledgments: true
      tables:
        - table_arn: "arn:aws:dynamodb:eu-central-1:111122223333:table/app"
          export:
            s3_bucket: "my-export-bucket"
            s3_prefix: "export/"
          stream:
            start_position: "LATEST"
      aws:
        sts_role_arn: "arn:aws:iam::111122223333:role/pipeline-role"
        region: "eu-central-1"
  sink:
    - opensearch:
        hosts: ["https://search-mydomain.eu-central-1.es.amazonaws.com"]
        aws:
          sts_role_arn: "arn:aws:iam::111122223333:role/pipeline-role"
          region: "eu-central-1"
        index: "orders"
        document_id: "${getMetadata(\"primary_key\")}"
        action: "${getMetadata(\"opensearch_action\")}"
        document_version: "${getMetadata(\"document_version\")}"
        document_version_type: "external"
        include_keys: ["orderId", "customerId", "status", "productName", "discount", "total", "price", "rating", "revision"]
        template_type: "index_template"
        template_content: |
          { "template": { "mappings": { "properties": {
            "customerId": { "type": "keyword" }, "orderId": { "type": "keyword" },
            "status": { "type": "keyword" }, "price": { "type": "float" },
            "total": { "type": "float" }, "rating": { "type": "float" },
            "discount": { "type": "float" }, "revision": { "type": "long" } } } } }
        dlq:
          s3:
            bucket: "my-dlq-bucket"
            key_path_prefix: "dlq/orders"
            region: "eu-central-1"
            sts_role_arn: "arn:aws:iam::111122223333:role/pipeline-role"

Four settings carry the correctness of the whole arrangement, and the best-practices page (opens in new tab) backs them, the mapping template as a hard rule and the rest as strong defaults. document_id uses the table’s primary key, so an update to an item lands on the same document instead of creating a duplicate. action uses the opensearch_action metadata, so a delete in the table also deletes the document; without it, removed items linger in the index as stale search results. document_version with document_version_type: external closes the ordering hole: without it, a stream delivery that arrives out of order can overwrite newer data with older. And the sink declares a mapping template together with include_keys, so the index holds only the attributes you search, sort, and facet on; everything else is fetched from DynamoDB by ID, which is exactly what the primary-key document ID makes cheap. One spelling in that template deserves a validator check before deploying: the DynamoDB guide writes template_type: index_template while the Data Prepper sink reference lists index-template, so run the configuration through the pipeline validator rather than trusting either page.

The hydration step is its own small contract. BatchGetItem returns items in no particular order and can return unprocessed keys, a hit can reference an item DynamoDB has since deleted or changed past the filter, and a document ranked by yesterday’s price hydrates with today’s. The API layer re-orders by the search ranking, drops hits whose live item no longer matches, refills the page when it must, and counts those drops, because a rising drop rate is the replica falling behind. For the fields that drove the ranking it also picks a side: render the indexed value, or re-validate and re-sort by the hydrated one; ranking by one number while displaying another reads as a mis-sorted page.

One inconsistency in the documentation is worth knowing about in advance. At the time of writing, the DynamoDB developer guide asks for Streams with new and old images, while the OpenSearch ingestion page calls NEW_IMAGE sufficient and lists NEW_AND_OLD_IMAGES as also supported. Enabling NEW_AND_OLD_IMAGES satisfies both readings, at the cost of larger stream records.

The documented limitations shape the architecture more than the features do. The pipeline and table must live in the same account and Region, and each pipeline supports exactly one source table (the ingestion page’s limitations section is the operative text; the DynamoDB guide’s overview still promises “one or more DynamoDB tables”, the same kind of documentation conflict as the stream-view one below). DynamoDB Streams retains events for 24 hours, so an initial export of a very large table that takes longer than that loses changes at the seam, and a pipeline stalled for more than a day means a full re-export. The quotas page (opens in new tab) also says to design for at most two simultaneous reader processes per stream shard: if a Lambda consumer already tails the stream for aggregates or an outbox, the pipeline takes the second slot, and a third reader is a throttling risk rather than a hard stop; past two, fan the stream out through Kinesis Data Streams for DynamoDB or an EventBridge pipe instead of attaching another direct consumer. For global tables the same page recommends a single simultaneous reader, which the pipeline would then be. Single-table designs get one more instruction: one overloaded table still means one pipeline, and separating entity types into different indexes happens with routing inside that pipeline’s configuration instead of several pipelines on the same stream.

The Mapping Trap#

The failure most likely to arrive in week two is documented in the ingestion guide (opens in new tab). Without an explicit mapping, OpenSearch infers field types from the first document it sees: a whole number maps to long, a fractional number to float. A schemaless table meets a typed index, and the schemaless side loses. If the first replicated item carries "discount": 10 and a later one carries "discount": 9.5, the later document fails to ingest and lands in the dead-letter queue, assuming you configured one. The fix is declaring the template up front:

{
  "template": {
    "mappings": {
      "properties": {
        "discount": { "type": "float" },
        "status": { "type": "keyword" },
        "productName": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        }
      }
    }
  }
}

The keyword versus text decision is the other half of the template. text is analyzed for what users type into a search box; keyword is unanalyzed for what they sort and facet on. Getting this wrong does not fail in the obvious direction: sorting directly on a text field is rejected outright, because fielddata is disabled for text fields by default, and enabling fielddata to force it sorts by analyzed tokens rather than the visible name. The multi-field pattern above (analyzed for matching, raw for sorting) is the standard way to serve both. A mapping change to an existing field generally means reindexing (adding a new field can land in place), so the read model has a schema after all; the difference from a relational migration is that the recovery procedure is to drop the index and rebuild it from the table.

Pagination on the Search Side#

The pagination post’s core argument was that DynamoDB gives you a cursor and nothing else. OpenSearch gives you more, and each addition has its own price.

Numbered pages come back through from and size, the offset pagination that DynamoDB cannot do. The ceiling is the index.max_result_window index setting (opens in new tab), 10,000 results by default: the pagination documentation (opens in new tab) caps offset paging there because deep offsets make every shard materialize and discard the skipped results. Raising the window is possible and trades per-shard memory for depth, so page 7 of 42 works within the default window while page 700 does not.

Past the window, search_after continues from the sort values of the last hit with no depth limit. It is stateless by design, which means the document order can shift under a paginating client as documents are indexed or deleted, and it requires a fully deterministic sort, so a unique tiebreaker field belongs at the end of every sort list. A filtered, sorted page sent to /orders/_search looks like this:

{
  "size": 20,
  "track_total_hits": 10000,
  "query": {
    "bool": {
      "filter": [
        { "term": { "customerId": "CUST-2048" } },
        { "term": { "status": "shipped" } },
        { "range": { "total": { "gte": 100 } } }
      ]
    }
  },
  "sort": [
    { "price": "asc" },
    { "orderId": "asc" }
  ],
  "search_after": [129.99, "ORDER-7431"]
}

The customerId term is the tenant boundary, and it is not optional. The index is shared across customers, so the search layer injects that filter server-side on every query; a request that reaches OpenSearch without it returns other customers’ orders. The filter is an application control, not the whole boundary: anything that reaches the domain, the export bucket, or the DLQ directly bypasses it, so those surfaces need their own access story, and managed domains offer document-level security when the boundary must live in the store itself.

For deep pagination over a stable snapshot, the OpenSearch documentation recommends Point in Time with search_after: a PIT freezes the searched dataset for a keep_alive window you choose and should delete when done. On a managed domain the PIT documentation (opens in new tab) adds a caveat that matters operationally: PIT state has no resiliency, and node reboots, node replacements, blue/green deployments, and process restarts all discard it. A frozen snapshot that can evaporate mid-session is a different failure mode from an expired HMAC cursor, but the client-side answer is the same start-over path.

Exact totals, which DynamoDB cannot produce cheaply in a single call over a large, changing result set, come from track_total_hits. The default behavior in practice is to count accurately up to 10,000 matches and report anything larger as a lower bound with "relation": "gte". The Search API reference (opens in new tab) documents the parameter without stating that default; the clearest description of current behavior sits in a dormant proposal (opens in new tab) to change it that targeted the 3.x line, which has since shipped with the behavior intact, and the behavior itself is inherited from Elasticsearch 7. Verify it against your cluster version. Setting track_total_hits: true counts every match on every request and costs proportionally: exact counts exist when you pay for them, and a UI that can live with “10,000+” should say so. Either way the number is exact for the index’s view of the data, which trails the table, so it is a search figure, not a quota or billing figure.

Relevance is the one addition with no DynamoDB counterpart at any price: sorting by _score for a text query is native, and it is what turns a filter screen into search. The overall exchange is symmetrical enough to state plainly: DynamoDB’s pagination is cheap, predictable, and total-free; OpenSearch’s has totals, page numbers, and relevance, at a cost that grows with result-set depth. Facet counts carry a footnote of their own: a terms aggregation gathers candidate buckets per shard, so on high-cardinality fields the counts are approximate by default, with a documented error bound and a shard_size setting that trades memory for accuracy.

Read-Your-Own-Writes Across the Lag#

The seam between the stores is replication lag, and its most visible casualty is read-your-own-writes: a client creates an order, receives a 201, navigates to the list view, and the list is served by a replica that has not seen the order yet. The documentation treats lag as an operational metric and attaches no upper bound to it. The integration guide describes ongoing changes replicating in near real time; the best-practices page recommends alarming when replication delay exceeds one minute, on the reasoning that a one-minute threshold stays quiet in normal operation; and past 24 hours the stream ages out events and the replica needs a rebuild. That is the design budget: near real time in normal operation, a minute before a human is paged, a day before recovery stops being automatic.

Mitigations, ordered by cost:

1. Route the writer’s own next read back to DynamoDB. After a mutation, the detail view fetches the item by key, strongly consistent if needed. Cheap, and it covers the most common complaint.
2. Splice the just-written item into the first search page. The API layer holds the created item and prepends it to page one of that user’s list for a short window. No new infrastructure, some fiddly dedupe when the replica catches up. This stays honest only on a newest-first list whose active filter the item actually matches; under price, rating, or relevance ordering the item belongs at its correct sort position or not on the page at all. Prepending onto a full page also moves the boundary: return the first page-size entries and mint the next token from the last hit actually emitted, not from the original response, or the displaced final hit is skipped when the reader continues.
3. Wait for the write to surface. Stamp the item with an application-controlled revision attribute, carry it through include_keys into the index (the pipeline strips anything outside the allowlist), and poll for that exact revision before acknowledging; the stream-assigned document_version is no use here, because the writer never learns its value and a concurrent write can look like the one being awaited. Even a correct poll only delays the acknowledgment, because the DynamoDB write is already durable when the poll starts, so the timeout path needs an answer of its own. This buys a stronger guarantee by adding latency to the write path, so it belongs only where the product genuinely demands it.
4. Show the truth in the UI. A syncing indicator on the affected row costs the least and is often the correct answer, because the row genuinely is syncing.

What does not work is leaving the seam undesigned and letting support tickets discover it. The staleness window belongs in the API contract next to the pagination token format.

The Two-Store Tax#

The money floor is visible from list prices alone. Since I have not run this pipeline, what follows is arithmetic on the published pricing (opens in new tab), and it should be read as an estimate. OpenSearch Ingestion meters per OCU-hour, at a list price of USD 0.24 in us-east-1 at the time of writing, and a pipeline is provisioned capacity whose meter runs with or without traffic: one OCU around the clock is 730 hours times USD 0.24, roughly USD 175 a month, before the destination and before storage. AWS also recommends at least two Ingestion OCUs (opens in new tab) for 99.9 percent pipeline availability, which doubles that line item before anything else is counted. The destination forks. A managed domain runs continuously at whatever instance size you chose. A classic OpenSearch Serverless collection bills a minimum of 2 OCUs (opens in new tab) at the same rate, roughly USD 350 a month, with a dev-test option at half that; the floor applies per account and is shared across its collections, not paid again per collection. NextGen collection groups change the answer entirely: their minimum is 0 OCUs and they scale to zero (opens in new tab) after 10 minutes of inactivity, at the price of 10 to 30 seconds of latency on the first request after a wake. Which floor applies depends on a collection type many teams have not evaluated yet, so check the pricing page against your Region instead of averaging these numbers. Pipelines can also be stopped when idle, and a stopped pipeline accrues no OCU hours, which matters for staging. The saving has an edge, though: a pipeline stopped past the stream’s 24-hour retention is back to a full re-export, so check the stop-and-start behavior for the DynamoDB source before making the nightly stop a habit.

The security surface grows in a way the price sheet does not show. The pipeline role needs dynamodb:ExportTableToPointInTime plus stream-read permissions on the table, both write and read-back access on the export bucket (the pipeline writes the snapshot there and then loads it), write access on the DLQ bucket with the matching KMS permissions when the buckets are encrypted, and signed HTTP access to the destination domain. More consequentially, the index now holds a copy of production data governed by OpenSearch’s own access-control model instead of DynamoDB’s IAM-based one. Field-level authorization that IAM policies gave the table must be re-implemented in the search layer, or the sensitive fields must never enter the index; include_keys doubles as a security control here, though only for the index: the export bucket and the DLQ hold copies of the same data and need their own retention and access rules.

The operational load is the recurring part. A mapping template implies a reindex procedure for the day it changes. A DLQ implies a replay script, and the script is best written before the first backfill, while there is still time to test it. Lag needs alarms on the pipeline’s EndToEndLatency and PipelineLatency metrics, with the one-minute threshold above as the starting point. Capacity has documented formulas: for provisioned tables the best-practices page sizes the pipeline between (table WCU / 1000) - 1 and (table WCU / 1000) + 1 OCUs with a floor of one, on the rule of thumb that one OCU sustains about 1 MB per second, and each OCU processes up to 150 stream shards in parallel. None of these tasks is exotic; together they form a second on-call surface tied entirely to the second store.

When a Single PostgreSQL Wins#

The strongest argument against the architecture above is a single relational database, and it deserves a complete answer. One PostgreSQL or Aurora instance does point lookups, arbitrary ORDER BY over any indexed column, keyset or offset pagination, exact COUNT(*), ad hoc filter combinations, and respectable full-text search with tsvector and a GIN index, in one store, with one query language, one backup story, and one consistency model. Every feature that motivated the OpenSearch replica exists there in some form, without replication lag, without a mapping template, and without a second bill.

The relational ceilings are real but arrive later than commonly assumed. The documented text-search limits (opens in new tab) include a tsvector under 1 MB per document and lexemes under 2 KB, generous for product and back-office search. OFFSET pagination degrades linearly with depth, and the fix is keyset pagination long before it is a search engine. Exact COUNT(*) exists but is not free either: on a large, unselective result set it is a scan whose cost needs the same indexing and measurement discipline as everything else. The genuine gaps against OpenSearch are relevance tuning, typo tolerance, and faceting ergonomics at large scale, and a team should confirm it needs those before paying for them.

So the fork is about where you stand. Still choosing the system of record, or holding access patterns that were never enumerable: pick the relational database, and the whole two-store question dissolves. Already running DynamoDB as the system of record because the write path leans on single-digit-millisecond key access at any write rate, on-demand throughput that scales to zero, and freedom from connection pools: then the zero-ETL replica is an addition that preserves those strengths, while migrating to Postgres would mean re-platforming the write path to fix a read problem. The trade-off runs the other way too, though less starkly than it used to: a provisioned, always-on Aurora instance never bills at zero, Aurora Serverless v2 can pause to zero ACUs (opens in new tab) when idle at the cost of resume latency, and a quiet DynamoDB table on on-demand mode stops billing for throughput with nothing to configure, while storage accrues in both. Pick the two-store architecture when DynamoDB is earning its keep on the write path. Reaching for it to rescue a store choice that was wrong from the start adds a second system on top of the wrong first one.

What the Key-Value Framing Does Not Claim#

Calling DynamoDB a key-value store with pre-modeled range reads invites a fair objection: the platform is visibly more than that. Single-table design composes item collections, sparse indexes, and overloaded GSIs into rich access patterns, and it does so precisely because those patterns were named in advance; nothing in the routing rule argues against it. DynamoDB also now ships native vector indexes with a SearchVectors API for approximate nearest-neighbor queries, a new query shape inside the database itself. The vector search documentation (opens in new tab) and its quotas also mark the boundary: ANN search returns semantic similarity with a TopK ceiling of 100 results per request, responses are capped at 16 MB with no pagination, inline filters accept equality only, and it does not add arbitrary sort, boolean facets, exact totals, or lexical matching. A deep-paginated, filtered, sorted list screen is outside its shape.

The key-value framing is a claim about ad hoc reads, and only about those. Questions you can name at design time belong in the key model, however sophisticated that model gets. Questions composed at request time do not, and the artifacts from the opening are what accumulate when that line is ignored.

Where This Design Breaks First#

• A GSI per sort option until the write bill doubles. Count independent sort attributes; past two or three, price the replica before shipping the next index.
• Mirroring the whole item into the index. Use the mapping template plus include_keys, and fetch full items from DynamoDB by ID. Anything indexed is also exposed to the search layer’s access model.
• Letting dynamic mapping decide types. The first fractional value on an inferred long field sends documents to the DLQ. Declare numeric types, keyword for sort and facet fields, text for free text.
• No DLQ before the first backfill. Configure it with the pipeline, and write the replay script the same day.
• Skipping the stream-reader inventory. Two reader processes per shard is the design limit; an existing Lambda consumer plus the pipeline fills it.
• Treating the replica as a source of truth. Never write to the index directly. Every indexed attribute must be reconstructible from the table, because the recovery procedure is a rebuild.
• Leaking search_after values in pagination tokens. Sort values are internal data; sign and scope the token as the pagination post does for LastEvaluatedKey, and encrypt it when the values must stay unreadable.
• Maintained facet counters as a workaround. A counter item per facet value re-creates the hot-key problem under burst; the rate-limit strategies post shows how that fails before the replica does.

What to Instrument Once the Pipeline Runs#

None of this has run yet, so what follows is a measurement plan, not results. Replication lag comes first: the pipeline’s EndToEndLatency and PipelineLatency averages, alarmed at 60 seconds per the AWS guidance and watched during write bursts, where lag shows first. DLQ document counts should sit at zero, and a sustained non-zero rate usually points at the mapping template first, though permissions, oversized documents, and sink rejections land in the same queue. The sink’s version-conflict errors are a separate signal: they point at the write path, a second writer on the index or out-of-order document_version values, not at the mapping. Traffic share per store tells whether the routing rule holds: if the DynamoDB path answers almost nothing, either identity reads are routed wrong or the product really is search-first, and it is worth finding out which before concluding. The deep-page rate, requests with from beyond 1,000, predicts when the 10,000 window becomes a support ticket. Index size against table size shows whether include_keys is doing its job. And the number of GSIs on the base table should stop growing for search reasons, because the replica exists to absorb those; a new GSI for a genuinely identity-shaped pattern stays legitimate.

The split holds when DynamoDB is already the system of record, its write path leans on the store’s strengths, and search-shaped requirements keep arriving. Override toward a single PostgreSQL when the system of record is still open or the access patterns were never enumerable, and postpone the replica entirely while one back-office screen is the only search requirement in sight. The first concrete step is unglamorous: write the mapping template and the DLQ replay script before creating the pipeline, and let the staging table run the initial export before any user-facing traffic depends on the index.

References#

• DynamoDB zero-ETL integration with Amazon OpenSearch Service (opens in new tab) - The two-phase design: point-in-time export to S3, then DynamoDB Streams replication, with PITR and Streams as prerequisites
• Using an OpenSearch Ingestion pipeline with Amazon DynamoDB (opens in new tab) - Pipeline YAML, type-mapping inference behavior, the 150-shards-per-OCU rule, and the documented limitations
• Best practices for working with DynamoDB zero-ETL integration and OpenSearch Service (opens in new tab) - Primary key as document ID, mapping templates with include_keys, the one-minute lag alarm, and OCU sizing formulas
• Quotas in Amazon DynamoDB (opens in new tab) - The default 20-GSI quota, the two-readers-per-shard stream guidance, and vector index quotas
• NoSQL design for DynamoDB (opens in new tab) - AWS’s framing that data is queried efficiently only in a limited number of ways, starting from known questions
• What is Amazon DynamoDB? (opens in new tab) - The service description as a key-value and document database without JOIN support
• Paginate results - OpenSearch Documentation (opens in new tab) - The 10,000-result ceiling on from and size, search_after statelessness, and PIT as the preferred deep-pagination method
• Point in time search in Amazon OpenSearch Service (opens in new tab) - keep_alive semantics and the loss of PIT state across node restarts and blue/green deployments
• Search API - OpenSearch Documentation (opens in new tab) - The request body reference for track_total_hits, from, size, sort, and search_after
• Set track_total_hits=false by default for _search from 3.x (opens in new tab) - The dormant proposal, targeting a 3.x line that shipped without the change, that still best describes today’s default of counting up to 10,000 with relation gte beyond it
• Amazon OpenSearch Service pricing (opens in new tab) - OCU-hour list prices for OpenSearch Ingestion and Serverless, the classic 2-OCU minimum with its dev-test half floor, and the cost floor arithmetic
• Index settings - OpenSearch Documentation (opens in new tab) - The index.max_result_window setting whose 10,000 default caps offset paging
• Scaling pipelines in Amazon OpenSearch Ingestion (opens in new tab) - The recommendation to provision at least two Ingestion OCUs for 99.9 percent pipeline availability
• Automatic pause and resume for Aurora Serverless v2 (opens in new tab) - Aurora Serverless v2 scaling to zero ACUs when idle, with resume measured in seconds
• Scale to zero for Amazon OpenSearch Serverless (opens in new tab) - NextGen collection groups reaching 0 OCUs after 10 idle minutes, with 10-to-30-second wake latency
• Using vector indexes in DynamoDB (opens in new tab) - What approximate-nearest-neighbor indexes add and what they do not replace
• PostgreSQL: Text Search Limitations (opens in new tab) - The documented tsvector and lexeme ceilings for sizing the relational alternative