点燃结点carshed[ttl-cleanup-worker]

Question

我有Ignite2.7和5节点集群。超过40Mil的数据正在生成并存储在ignite缓存中。我已经设定了3天的有效期。今天其中一个点火节点停止了，并显示如下错误。请帮助我确定并解决这个问题。

[2019-09-11 07:45:59,570][错误][TTL-Cleanup-Worker-#170][根]检测到严重系统错误。将根据配置的处理程序[hnd=stopnodeorhaltfailurehandler[trystop=false,timeout=0,super=abstractfailuretypes=[SYSTEM_WORKER_BLOCKED]]]，Failurectx=failurecontext[type=system_worker_terminate，err=java.lang.IllegalStateException:未知页面类型：1 pageID:000102210006d4ac]]java.lang.IllegalStateException:未知页面类型：1 he.ignite.internal.processors.cache.persistence.bplustree.access$200(bplustree.java:90)在org.apache.ignite.internal.processors.cache.persistence.bplustree$abstractForwardCursor.nextPage(bplustree.java:5330)在org.apache.ignite.internal.processors.persistence.bplustree.nforwardCursor.next(bplustree.java:5566)在经理。java:2232）在org.apache.ignite.internal.processors.cache.persistence.gridcacheoffheapmanager$gridcachedatastore.purgeExpertif(Gridcacheoffheapmanager.java:2157)在org.apache.ignite.internal.processors.cache.persistence.gridcacheoffheapmanager.expire（gridcacheoffheapmanager.java:845)在DCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:139)在org.apache.ignite.internal.util.worker.gridworker.run(Gridworker.java:120)在java.lang.thread.run(thread.java:748)[2019-09-11 07:45:59,575][WARN][TTL-Cleanup-Worker-#170][FailureProcessor]没有检测到死锁线程。Iseconds.[2019-09-11 07:46:40,831][错误][SYS-STRIPE-0-#1][G]已检测到阻塞的系统关键线程。这可能导致群集范围内未定义的行为[ThreadName=GRI d-nio-worker-tcp-comm-23,BlockedFor=41S][2019-09-11 07:46:40,832][WARN][SYS-Stripe-0-#1][G]线程[name=“GRID-NIO-Worker-TCP-Comm-23-#143”,ID=173,State=Runnable,BlockCNT=0,WAITCNT=0]

配置如果ignite是，

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

        <!-- Enabling native persistance-->
        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="metricsEnabled" value="true"/>
                <property name="defaultDataRegionConfiguration">
                    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                    </bean>
                </property>
                <property name="storagePath" value="/ignite_data/ignite/persistance"/>
                <property name="walPath" value="/ignite_data/ignite/wal"/>
                <property name="walArchivePath" value="/data/disk01/ignite/archive"/>
            </bean>
        </property>

        <!-- Enable authentication for ignite-->
		<property name="authenticationEnabled" value="true"/>


        <!-- Enabling expiry policy -->
        <property name="cacheConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="CACHE_L4_TRIGGER_NOTIFICATION"/>
                    <property name="expiryPolicyFactory">
                        <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
                            <constructor-arg>
                                <bean class="javax.cache.expiry.Duration">
                                    <constructor-arg value="DAYS"/>
                                    <constructor-arg value="3"/>
                                </bean>
                            </constructor-arg>
                        </bean>
                    </property>
                </bean>
            </list>
        </property>


        <!-- Enable Ignite matric logged into logs in every 10 min-->
        <property name="gridLogger">
            <bean class="org.apache.ignite.logger.log4j.Log4JLogger">
                <constructor-arg type="java.lang.String" value="/home/trigger_be/apache-ignite-2.7.0/config/log4j.xml"/>
            </bean>
        </property>
        <property name="metricsLogFrequency" value="#{60 * 10 * 1000}"/>

        <!-- Set Cluster by giving IPs-->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                        <property name="addresses">
                            <list>
                                <value>172.16.5.36:49500..49509</value>
                                <value>172.16.5.37:49500..49509</value>
                                <value>172.16.5.38:49500..49509</value>
                                <value>172.16.5.39:49500..49509</value>
				                <value>172.16.5.40:49500..49509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

Answer 1

这看起来像是数据损坏问题。建议从该节点完全删除持久性数据，并将其重新添加到集群的基线拓扑中。然后数据将被重新平衡，前提是您有足够的备份。

这看起来有点像Ignite-10767问题。是否启用了MVCC（transactional SQL，TRANSACTIONAL_SNAPSHOT缓存）？