如何修复Ignite性能问题？

Question

我们在服务器和客户机模式下使用Ignite 2.7.6：两个服务器和六个客户机。

[00:59:55,628][SEVERE][sys-#44759][GridDhtPartitionsExchangeFuture] Failed to notify listener: o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2@3287dcbd
java.lang.OutOfMemoryError: Java heap space

正如我们所看到的，现在所有服务器节点的CPU负载都很高，约为250%（更新前为20%)，而长G1 Young Gen的停顿时间高达5毫秒（更新前为300微秒）。

服务器配置为：

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
  <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="workDirectory" value="/opt/qwerty/ignite/data"/>
    <property name="gridLogger">
      <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
        <constructor-arg type="java.lang.String" value="config/ignite-log4j2.xml"/>
      </bean>
    </property>
    <property name="dataStorageConfiguration">
      <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="defaultDataRegionConfiguration">
          <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
            <property name="maxSize" value="#{24L * 1024 * 1024 * 1024}"/>
            <property name="pageEvictionMode" value="RANDOM_LRU"/>
          </bean>
        </property>
      </bean>
    </property>
    <property name="discoverySpi">
      <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
        <property name="localAddress" value="host-1.qwerty.srv"/>
        <property name="ipFinder">
          <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="addresses">
              <list>
                <value>host-1.qwerty.srv:47500</value>
                <value>host-2.qwerty.srv:47500</value>
              </list>
            </property>
          </bean>
        </property>
      </bean>
    </property>
    <property name="communicationSpi">
      <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
        <property name="localAddress" value="host-1.qwerty.srv"/>
      </bean>
    </property>
  </bean>
</beans>

在Ignite服务器节点的内存转储中，我们看到大量org.apache.Ignite.internal.marshaller.optimized.optimizedObjectStreamRegistry$StreamHolder，大小为21MB

Problem Suspect 1

One instance of "org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager" loaded by "jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x400000100" occupies 529 414 776 (10,39 %) bytes. The memory is accumulated in one instance of "java.util.LinkedList" loaded by "<system class loader>".

Keywords
jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x400000100
java.util.LinkedList
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager

Problem Suspect 2

384 instances of "org.apache.ignite.thread.IgniteThread", loaded by "jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x400000100" occupy 3 023 380 000 (59,34 %) bytes. 

Keywords
org.apache.ignite.thread.IgniteThread
jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x400000100

Problem Suspect 3

1 023 instances of "org.apache.ignite.internal.processors.cache.CacheGroupContext", loaded by "jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x400000100" occupy 905 077 824 (17,76 %) bytes. 

Keywords
jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x400000100
org.apache.ignite.internal.processors.cache.CacheGroupContext

Answer 1

2000个缓存是很多的。一个缓存可能占用多达40M的数据结构。

我建议对于用途和组成相似的所有缓存至少使用相同的cachegroup，以共享其中的一些数据结构。