Strategy Platform Analytics Consumer service crashes randomly and must manually startup the service. Platform Analytics consumer log sees intermittent connectivity issue with Redis and sometimes without entry of "Shutdown process finished". Based on Redis logs it appears to be up and running.
Example 1:
2021-07-14 06:01:58,763 ERROR ConsumerProcess$ConsumerRecordsProcessor:329 - Fatal exception occurred when processing records: java.lang.reflect.InvocationTargetException: nullat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]at com.Strategy.pa.kafka.consumer.main.ConsumerProcess$ConsumerRecordsProcessor.call(ConsumerProcess.java:324) [platform-analytics-consumer.jar:?]at com.Strategy.pa.kafka.consumer.main.ConsumerProcess$ConsumerRecordsProcessor.call(ConsumerProcess.java:234) [platform-analytics-consumer.jar:?]at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]at java.lang.Thread.run(Thread.java:834) [?:?]Caused by: com.Strategy.pa.common.exception.PARedisException: Unable to connect to Redis. Please restart Rediscom.Strategy.pa.cache.RedisCache.verifyRedisConnectionAndCacheStatus(RedisCache.java:515) ~[platform-analytics-core.jar:?]at com.Strategy.pa.cache.RedisCache.isMemberOfSet(RedisCache.java:492) ~[platform-analytics-core.jar:?]at com.Strategy.pa.persistence.dao.impl.EtlRelSourceSubTypeObjectTypeDaoImpl.isSelectedObjectType(EtlRelSourceSubTypeObjectTypeDaoImpl.java:79) ~[platform-analytics-consumer.jar:?]at com.Strategy.pa.etl.ETLForLookupTableFromMd.ETLprocessing(ETLForLookupTableFromMd.java:44) ~[platform-analytics-consumer.jar:?]at com.Strategy.pa.etl.PAETL.processRecords(PAETL.java:216) ~[platform-analytics-consumer.jar:?]... 10 moreCaused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the poolredis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.0.jar:?]at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:?]at com.Strategy.pa.cache.RedisCache.isMemberOfSet(RedisCache.java:488) ~[platform-analytics-core.jar:?]at com.Strategy.pa.persistence.dao.impl.EtlRelSourceSubTypeObjectTypeDaoImpl.isSelectedObjectType(EtlRelSourceSubTypeObjectTypeDaoImpl.java:79) ~[platform-analytics-consumer.jar:?]at com.Strategy.pa.etl.ETLForLookupTableFromMd.ETLprocessing(ETLForLookupTableFromMd.java:44) ~[platform-analytics-consumer.jar:?]at com.Strategy.pa.etl.PAETL.processRecords(PAETL.java:216) ~[platform-analytics-consumer.jar:?]... 10 moreCaused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused: connectredis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:?]at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:?]at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:?]at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:?]at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868) ~[commons-pool2-2.4.1.jar:2.4.1]at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.4.1.jar:2.4.1]at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.1.jar:2.4.1]at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:?]at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:?]at com.Strategy.pa.cache.RedisCache.isMemberOfSet(RedisCache.java:488) ~[platform-analytics-core.jar:?]at
2021-10-22` 21:07:02,930 ERROR ScheduledETLProcess:151-
Cache is in an inconsistent state. Shutting down consumer. Please restart Platform Analytics Consumer to rebuild the cache and resolve the issue
Example 1:
[3884] 01 Jul 09:12:49.721 * The server is now ready to accept connections on port 6379[3668] 14 Jul 10:02:54.721 * Redis 3.2.100 (00000000/0) 64 bit, standalone mode, port 6379, pid 3668 ready to start.
[18412] 22 Oct 21:07:11.638 - 50 clients connected (0 slaves), 27183464 bytes in use[18412] 22 Oct 21:07:11.971 - Accepted 127.0.0.1:53666[18412] 22 Oct 21:07:11.973 - Client closed connection[18412] 22 Oct 21:07:16.410 - Reading from client: An existing connection was forcibly closed by the remote host.
This is a known issue.
There is bug in the Jedis version 2.9.0 with current Strategy release.
Per Jedis release notes fixed in newer version.
2.10.2 Bug fix version
#1920 JedisPool exhausted in Jedis 2.10.0
This issue has been fixed in Strategy 2021 Update 4. Upgrade to this version to take advantage of the fix.
Third Party Software Installation WARNING
The third-party product(s) discussed in this technical note is manufactured by vendors independent of Strategy. Strategy makes no warranty, express, implied or otherwise, regarding this product, including its performance or reliability
