KB484109: Apache Kafka service fails to start in MicroStrategy 2020 on Windows

Description

In a Windows environment running Strategy 2020, the Apache Kafka service fails to start or restarts automatically. Errors similar to the following can be seen in the Kafka

server.log

file:

[2020-02-24 09:03:48,133] ERROR [ReplicaManager broker=0] Error while making broker the follower for partition Topic: Mstr.AnalyticsServer.Diag.servername123.CastorServer.RegtrySettings; Partition: 0; Leader: None; AllReplicas: ; InSyncReplicas: with leader -1 in dir None (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Error while creating log for Mstr.AnalyticsServer.Diag.servername123.CastorServer.RegtrySettings-0 in dir C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\kafka-logs
Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
at java.io.RandomAccessFile.setLength(Native Method)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175)
at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:238)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:238)
at kafka.log.LogSegment.recover(LogSegment.scala:377)
at kafka.log.Log.recoverSegment(Log.scala:500)
at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:482)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791)
at kafka.log.Log.loadSegmentFiles(Log.scala:454)
at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:565)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2034)
at kafka.log.Log.loadSegments(Log.scala:559)
at kafka.log.Log.(Log.scala:292)
at kafka.log.Log$.apply(Log.scala:2168)
at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:716)
at scala.Option.getOrElse(Option.scala:138)
at kafka.log.LogManager.getOrCreateLog(LogManager.scala:674)
at kafka.cluster.Partition.$anonfun$getOrCreateReplica$1(Partition.scala:202)
at kafka.utils.Pool$$anon$1.apply(Pool.scala:61)
..............

[2020-02-24 09:03:48,258] ERROR Shutdown broker because all log dirs in C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\kafka-logs have failed (kafka.log.LogManager)

Why is this happening?

In Windows, the file system is case-insensitive by default. When Strategy Intelligence Servers are configured to send diagnostic log information to Strategy Messaging Services, Kafka topics that include the name of the Intelligence Server machine are created. If the case of one or more characters in the machine name are later changed from upper-case to lower-case or vice versa, Messaging Services will create a new Kafka topic that has the same name but in a different case. Due to the case-insensitivity of the file system, Strategy Telemetry Server will attempt to access the same physical file for the two Kafka topics and these access attempts will block each other, causing the Kafka server to crash.

Solution

To confirm if multiple topics exist with the same name but different case, the following command can be run from the

C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.12-2.2.0\windows bin 

directory:

kafka-topics.bat --list --zookeeper localhost:2181 

In the above example, this command would show the following results:

...
Mstr.AnalyticsServer.Diag.SERVERNAME123.CastorServer.RegistrySettings
Mstr.AnalyticsServer.Diag.servername123.CastorServer.RegistrySettings
...

To resolve this problem, delete the Kafka topic that does not match with the current machine name using the following steps:
 

1. Delete the problematic topic directory on each Telemetry Server machine. In the Kafka log folder located  in

   C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\kafka-logs

   , delete the whole folder that matches the topic name.
2. Open Command Prompt and navigate to

   C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-1.1.0\bin\windows.

3. Run the following command to connect to the Zookeeper instance:

   zookeeper-shell.bat localhost:2181

4. Within the Zookeeper shell:
   
1. List the topics using:

    ls /brokers/topics

2. Remove the topic folder from Zookeeper using:

   rmr /brokers/topics/

3. Exit the Zookeeper shell (Ctrl+C)
5. Restart Kafka Server.
6. Confirm the topic was removed with the command:

    kafka-topics.bat --list --zookeeper localhost:2181