AWS OpenSearch provision checklist
What to decide before you stand up an Amazon OpenSearch Service domain. Numbers below are from the storage, sizing, and sharding docs. Test with a real workload after the estimate.
Disk and storage
Workload type. Long-lived index (search, catalog) vs rolling indexes (logs, time series). Retention is a number, not a feeling.
Storage formula. Source data × (1 + replicas) × 1.45. Full version: source × (1 + replicas) × 1.1 / 0.95 / 0.8. That is ~10% indexing overhead, 5% Linux reserved, 20% AOS overhead per node (capped at 20 GiB). Check real overhead later with _cat/indices pri.store.size. Undersized disk is how clusters go red.
Replicas in the math. Default is one. Keep at least one. Each replica is a full copy. Read-heavy: extra replicas buy query capacity and cost write amplification.
Tiers. Hot EBS/instance store for live data. UltraWarm / cold for old logs. UW does not pay the same replica tax as hot. ISM (OpenSearch; Elasticsearch called it ILM) moves indexes between tiers and deletes them. Decide hot/warm/cold ages now.
Rightsize the cluster
No formula is the cluster. Storage is the starting estimate, then instance type, then test.
Nodes. Three data nodes minimum. Three dedicated masters for anything that is not a toy. Multi-AZ with Standby if you care about AZ loss.
Per-node disk. Total storage ÷ data nodes, then leave headroom. Stay under ~80% disk. Start ~2 vCPU and 8 GiB RAM per 100 GiB of storage for light work; heavy aggs, updates, or query volume need more. Easier to start large and cut than to debug an undersized cluster.
Instance gen / JVM. Current gen. Graviton (r7g / m7g) unless you have a reason not to. Do not mix Graviton and Intel. Heap is half of RAM, capped at 32 GiB — going past 64 GiB RAM on a node does not grow the heap. OpenSearch uses G1GC. Old Elasticsearch 5/6 was CMS. T3 burst credit is not production. Enable Auto-Tune.
OR1. Worth it if ingest is the bottleneck.
Then: representative data, realistic clients, watch CPU / JVM / disk / queue. Iterate.
Write-heavy vs read-heavy
Indexing hits every primary and every replica. Search hits one copy of each primary (n shards). That is the whole tradeoff.
Write-heavy / logs. Shards 30–50 GiB. Refresh 30s or more. One replica unless you need more HA. Auto document IDs. Bigger bulks (start 3–5 MiB). Do not forcemerge an index that is still receiving writes.
Read-heavy / search. Shards 10–30 GiB. Extra replicas for query throughput. Refresh as low as freshness requires. Keyword (or multi-field) for filters and aggs. dfs_query_then_fetch if shard-local IDF is wrecking relevance.
Both. Primary count multiple of data-node count. Aim ~1.5 vCPU per shard as a first guess. ≤25 shards per GiB of heap. 50 GiB is the ceiling; plan growth.
Mappings
Decide this before the first document. Primaries do not change without a reindex.
Explicit mappings. Always, via index templates. Do not let the first document invent types.
dynamic. strict rejects unknown fields (AWS default recommendation for search). false stores them in _source and does not index them. true is how you get mapping explosions.
Text vs keyword. text is analyzed, for full-text. keyword is for filters, sorts, aggs. Most fields want keyword, or text + a .keyword subfield.
Fielddata. Off on text for a reason. It loads the inverted index into heap. If you agg on text, you wanted a keyword field.
ignore_malformed. Turn on if producers will send a string where you mapped a number, or a date that changes format. Otherwise one bad field rejects the whole document.
Indexing and query knobs
Indexing-heavy? Raise refresh interval, bulk, maybe translog flush size. Pause or buffer ingest when the cluster is sick.
Query-heavy? More replicas, smaller shards, watch search thread pool. Slow logs on in prod.
Refresh interval. Default 1s. AWS says 30s+ if you can live with the lag. This is also your read-after-write delay.
Replicas. HA + read capacity. Writes get more expensive with each one.
Translog. flush_threshold_size and sync interval. durability: request fsyncs every write (safer, slower). async is faster and you can lose the last few seconds.
Force merge. Only on read-only / rolled-over indexes, typically down to one segment before UltraWarm. It is I/O brutal. Never on a live write index.
Custom IDs. Auto IDs are cheaper. Custom IDs mean a lookup on every write. Use them for upserts / idempotency, not because the ID looks nice.
Ingest pipelines. Set @timestamp here. Grok, drop, rename, enrich, before the document hits the index. Set default_pipeline on the template.
Query profiler. Profile API for debugging one query. Not a default on production traffic.
Scroll. Holds a context on the cluster. Prefer search_after or PIT. Cap scroll contexts. Do not use scroll for user pagination.
Consistency
OpenSearch is not a database. Spell out what “consistent” means for this workload.
Read after write. Refresh interval is the lag. Need it now? refresh=wait_for on that write, and pay for it.
Acks. wait_for_active_shards defaults to the primary only. Set all (or a number) if you need the write acked on replicas too.
Scoring. BM25 IDF is per-shard. More shards → noisier relevance. Replicas can serve a slightly different view between refreshes. dfs_query_then_fetch if that matters.
Flattened arrays. Arrays of objects get flattened. {color:red} AND {size:L} can match a doc that has red on one object and L on another. Use nested if inner objects must stay together.
Custom shard routing. _routing colocate related docs (tenant, user). Uneven keys make hot shards. You must send routing on every read, or you miss.
Sharding
Shard size strategy. Search: 10–30 GiB. Logs: 30–50 GiB. Formula: (source + growth) × 1.1 / desired shard size ≈ primary count. Do not pre-split into tiny shards “for later.” Reindex when they grow.
Shard count alert. shards.active ≥ 25 × heap GiB × data nodes. Rotating daily indexes with too many primaries is how you get there. ISM deletes old ones.
Skew. Uneven shard or storage distribution = hot nodes. _cat/allocation. Custom routing and bad primary counts are the usual causes.
Design and scaling
Index templates for mappings and settings. ISM for rollover / retain / delete. Dedicated masters. Multi-AZ. Buffer ingest (SQS, Firehose, OSIS) so a brief domain blip does not drop data. Coordinator nodes if query fan-out is the problem. A second domain + CCR or dual-write if one region is not enough.
Blue/green. Most config changes (instance type, count, volume, version) start a blue/green. You briefly pay for a second fleet. Full Swap needs the capacity up front. Capacity Optimized (better at 30+ data nodes) can batch if the AZ is tight. Dedicated masters keep the cluster available during it. Do not schedule this as “a quick tweak.”
Backup and DR
Write the RPO and RTO down. Snapshots are not a strategy by themselves.
Automated snapshots. Hourly, 14 days, AWS-managed bucket. Restore to the same domain for red-cluster / data-loss recovery. They do not restore onto a different domain.
Manual snapshots. Your S3 bucket, your IAM repo role. Cross-domain, cross-account, migrations. This is the actual DR copy. Register the repo before you need it.
Reingest. If the source of truth is still upstream (logs in S3, a DB, a stream), rebuilding indexes can beat snapshot restore. Often the better RTO for logs. Not for unique search corpora you only have in the cluster.
Red cluster for two weeks = automated snapshots stop working and you can lose the lot.
Security
Turn these on at create time. Several of them are painful to retrofit.
Encryption at rest. KMS. Indexes, UW, automated snapshots. Watch KMSKeyError / KMSKeyInaccessible.
In transit. Node-to-node encryption. Enforce HTTPS.
Access policies. Resource-based policy on the domain. Do not leave Principal: * on a public endpoint.
FGAC. Cluster / index / document / field. Needs EAR + node-to-node. Internal users, IAM, or SAML.
VPC. Put the domain in the VPC. Security groups, not a public endpoint. PrivateLink if other VPCs need in.
Dashboards auth. Cognito, or SAML against your IdP. Pick one. Do not expose Dashboards to the internet with basic auth and hope.
Index rotation, ISM, UltraWarm
ISM policy on the index pattern: rollover (age or size), snapshot, forcemerge, hot → UW → cold, delete. Match the retention you used in the disk math. Rollover on size also keeps shard size in the 10–50 GiB band. Delete unused indexes; idle shards still cost heap.
CloudWatch alarms
Start from the AWS recommended set. Minimum:
ClusterStatus.red/yellowFreeStorageSpace(25% of node disk, not a blind 20 GiB)ClusterIndexWritesBlockedNodesAutomatedSnapshotFailureCPUUtilization/JVMMemoryPressure/OldGenJVMMemoryPressure- Master CPU and master JVM (keep masters bored)
shards.activevs 25 × heap × nodesThreadpoolWriteQueue/ThreadpoolSearchQueue- 5xx share of requests
KMSKeyError/KMSKeyInaccessible
If you use UW: migration queue, migration failures, warm JVM, warm free space.
Slow logs (search + index) on, with thresholds set, or CloudWatch captures nothing. Error logs too.