Unable to run seasonal or temporal event training
When you run data ingestion and train commands to get data into Cassandra, training does not complete and policies are not displayed in the manage policies screen. Scaling of stateful sets in the product is not supported. If you have scaled down the Cassandra stateful set, the following errors might be observed.
Problem
The following errors can be seen after you run seasonal or temporal event
training:
21/06/02 17:57:43 WARN TaskSetManager: Lost task 1.0 in stage 1.0 (TID 2) (10.254.26.209 executor 0): com.datastax.oss.driver.api.core.servererrors.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (1 required but only 0 alive)
...
21/06/02 17:57:50 ERROR TaskSetManager: Task 1 in stage 1.0 failed 4 times; aborting job
...
21/06/02 17:57:50 INFO DAGScheduler: ShuffleMapStage 1 (head at SparkUtils.java:65) failed in 10.663 s due to Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 9) (10.254.26.209 executor 0): com.datastax.oss.driver.api.core.servererrors.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (1 required but only 0 alive)
Driver stacktrace:
21/06/02 17:57:50 INFO DAGScheduler: Job 1 failed: head at SparkUtils.java:65, took 10.685935 s
21/06/02 17:57:50 ERROR SparkRunner: Unexpected stage execution failure.
java.lang.IllegalArgumentException: Event dataset cannot be empty. Check the configuration file for correct information. Exiting
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at com.ibm.itsm.analytics.se.app.SeasonalEventsProcess.train(SeasonalEventsProcess.java:114)
at com.ibm.itsm.analytics.se.app.SeasonalEventsProcess.process(SeasonalEventsProcess.java:89)
at com.ibm.itsm.analytics.se.app.SeasonalEventsProcess.process(SeasonalEventsProcess.java:36)When you log in to Cassandra NoHostAvailable is displayed.
This is a known issue, scaling down any stateful set in the product is not supported.