KB485588: Kafka service stops after upgrading to MicroStrategy 2021 Update 5, 5.1 and 5.2

Symptom 

Kafka service stops. When trying to restart it, it fails to come back and in the server.log will throw an error such as:

[2022-04-12 16:18:56,401] ERROR Failed to clean up log for __consumer_offsets-13 in dir C:\Program Files (x86)\Strategy\Messaging Services\tmp\kafka-logs due to IOException (kafka.server.LogDirFailureChannel) java.nio.file.FileSystemException: C:\Program Files (x86)\Strategy\Messaging Services\tmp\kafka-logs\__consumer_offsets-13\00000000000000000000.timeindex.cleaned -> C:\Program Files (x86)\Strategy\Messaging Services\tmp\kafka-logs\__consumer_offsets-13\00000000000000000000.timeindex.swap: The process cannot access the file because it is being used by another process. 

​
 

Cause

This is a known issue with Strategy 2021 Update 5, 5.1 and 5.2.
This can be a third-party Kafka defect on Windows KAFKA-13334 impacting the versions 11.3.4 and 11.3.5 as Kafka build 2.8.1. This defect is impeding Kafka to do a cleanup of logs and to load new ones.
 

Action 

This issue has been resolved with a fix for version 2021 U6 (11.3.6). Strategy will continue to investigate and post updates as soon as they are available. There is also an existing patch for only MSTR 2019 and MSTR 2020, however these patches are not useful for MSTR 2021.
 

Workaround 1

Note: The following workaround will eliminate the saved data in the temporary folders, and it will not be accessible to be consumed for Platform Analytics project. If this is a clustered environment, please make sure to erase the temporary files in all nodes before starting the services again.
 

1. Stop both zookeeper and Kafka services.
2. Delete Kafka log and zookeeper folder: /tmp/kafka-logs and /tmp/zookeeper/version-2
3. Start Zookeeper then Kafka services and by last Platform Analytics Consumer services
4. Start Platform Analytics normally

Please review the article KB484071 after the deletion of the offsets in order to preventing the inability of Platform Analytics Consumer to insert data into the Data Warehouse due to a unique constraint violation.
 

Workaround 2

Important note: The following workaround will disable the Kafka cleanup. This is exclusively recommended to do to an active system if a substantial free disk space is available. This is a temporary solution that will consume disk space to work until the next upgrade (11.3.6) has been performed. After performing the upgrade of the software, it is needed to confirm that these settings are no longer active, and the cleanup is set to work as normally.
 

1. Stop both zookeeper and Kafka services.
    

Disable Kafka cleanup in server.properties file in Strategy\Messaging Services\Kafka\kafka_X.XX-X.X.X\config:

1. Modify the line for retention hours to:
   

   log.retention.hours=-1

2. Add the following line to the end of server.properties file:
   

   log.cleaner.enable=false

3. Start all the services in the following order: Apache Zookeeper, Apache Kafka, Redis and by last Platform Analytics Consumer