登录
当前位置： 首页 > 工具软件 > Node-limits > 使用案例 >

Elasticsearch recovery.RecoveryFailedException: [xxx][0]: Recovery failed on {node-1} --- 采坑

程峻
2023-12-01

问题描述;

    服务器断电，es 直接挂掉，再次启动时，因以前数据量过大，大约500g，单机版es 启动恢复数据很是吃力，jvm 回收频繁。

[2019-06-17T16:24:14,303][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][20] overhead, spent [319ms] collecting in the last [1s]
[2019-06-17T16:26:05,781][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][131] overhead, spent [263ms] collecting in the last [1s]
[2019-06-17T16:28:39,668][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][284] overhead, spent [337ms] collecting in the last [1.1s]
[2019-06-17T16:29:38,902][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][343] overhead, spent [272ms] collecting in the last [1s]
[2019-06-17T16:30:31,004][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][395] overhead, spent [305ms] collecting in the last [1s]
[2019-06-17T16:32:14,218][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][498] overhead, spent [359ms] collecting in the last [1s]
[2019-06-17T16:33:16,289][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][560] overhead, spent [421ms] collecting in the last [1s]
[2019-06-17T16:35:25,630][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][689] overhead, spent [286ms] collecting in the last [1s]
[2019-06-17T16:36:08,858][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][727][3] duration [5.7s], collections [1]/[6.1s], total [5.7s]/[6.3s], memory [3.2gb]->[2.2gb]/[7.8gb], all_pools {[young] [839.5mb]->[8.7mb]/[865.3mb]}{[survivor] [108.1mb]->[0b]/[108.1mb]}{[old] [2.2gb]->[2.2gb]/[6.9gb]}
[2019-06-17T16:36:08,859][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][727] overhead, spent [5.7s] collecting in the last [6.1s]
[2019-06-17T16:36:09,146][WARN ][o.e.i.c.IndicesClusterStateService] [node-1] [[audit-net-udpflow-2019-05-24][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2019) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:402) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer
	at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:564) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	... 4 more
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[?:?]
	at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:550) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	... 4 more
[2019-06-17T16:36:09,163][WARN ][o.e.c.a.s.ShardStateAction] [node-1] [audit-net-udpflow-2019-05-24][0] received shard failed for shard id [[audit-net-udpflow-2019-05-24][0]], allocation id [MP_rxUh6Slu1ZGwpoGFtog], primary term [0], message [failed recovery], failure [RecoveryFailedException[[audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2019) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:402) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer
	at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:564) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	... 4 more
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[?:?]
	at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:550) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
	... 4 more

解决如下：

    1. ulimit -v 和 ulimit -m  返回必须为  unlimited。

    2. vm.max_map_count    的 值，要比上面报错信息中大一点。

         查看（root用户）：sysctl -a|grep vm.max_map_count

         临时设置：sysctl -w vm.max_map_count=xxxxx

         永久设置：

                 vi   /etc/sysctl.conf

           文件最后添加一行

                vm.max_map_count=262144

    3.jvm 调优。

       参考：

               elasticsearch性能调优

               Elasticsearch 6.0 性能调优策略

    4.如果服务器内存不够的话，只能把es数据备份出来，重新搭建，在使用脚本手动恢复，这样特别麻烦。

如果启动后还有问题参考：Elasticsearch 清除 unssigned,解决 red 问题 --- 采坑

 类似资料：

相关阅读

真/假vs MySQL中的0/1系统差异 Java中的exit（0）、System.exit（-1）、System.exit（1）JSONDecodeError：预期值：第1行第1列（字符0）SELECT…ORDER BY XXX LIMIT 1 FOR UPDATE将锁定多少行？在java中为什么 1/3 == 0?

相关文章

政采云java一面京东采销三面政采云一面凉经10.11 政采云 Java 一面政采云前端二面

相关问答

将数组[“null”、“0”、“1”、“2”、“2”、“1”、“0”]转换为[“1”、“1.1”、“1.1.1”、“a”、“b”、“1.1.2”、“1.2”]val=val==0是什么？0:1; 意思是[副本]0/1背包见证生成System.out.println（1/10）打印0？[副本]python中A[0]和A[0:1]numpy数组的区别

相关文档

Node Debug 三法三例数据采集从入门到放弃蓝天数据采集发布系统Java 高并发核心编程 NIO、Netty、Redis、ZooKeeper 卷1JavaScript 踩坑指南
快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.