ElasticsearchでValidation Failed: 1: this action would add [10] shards, but this cluster currently has [996]/[1000] maximum normal shards open
状況
- JaegerからElasticsearchへトレースを保存するよう設定してある．
- あるときからJaegerからElasticsearchへトレースが保存されなくなった．
- 上記の原因を知りたい．
調査
Jaegerコンテナのログを確認する．
$ kubectl logs jaeger-7ff9d7bb46-t744w -n istio-system -f
ログの中身は以下であった．
{"level":"error","ts":1729646980.1071966,"caller":"config/config.go:129","msg":"Elasticsearch part of bulk request failed","map-key":"index","response":{"_index":"jaeger-span-2024-10-23","_type":"_doc","status":400,"error":{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] shards, but this cluster currently has [996]/[1000] maximum normal shards open;"}},"stacktrace":"github.com/jaegertracing/jaeger/pkg/es/config.NewClient.func2\n\tgithub.com/jaegertracing/jaeger/pkg/es/config/config.go:129\ngithub.com/olivere/elastic.(*bulkWorker).commit\n\tgithub.com/olivere/elastic@v6.2.37+incompatible/bulk_processor.go:588\ngithub.com/olivere/elastic.(*bulkWorker).work\n\tgithub.com/olivere/elastic@v6.2.37+incompatible/bulk_processor.go:501"}
Reasonに着目すると以下のメッセージが見つかった．メッセージからも読み取れるがシャードの上限数に到達しているためエラーがでている．
Validation Failed: 1: this action would add [10] shards, but this cluster currently has [996]/[1000] maximum normal shards open;
調べたところ以下の記事が見つかった．やはりシャード数の上限に到達していた．