SQL1229 during activation of a member after hard crash of a DB2LUW DPF host

Body

There was a situation faced lately by a customer where a hard crash happened in one physical hosts of a DPF setup.   There were some hardware issues.

They brought back the DPF instance in all the partitions. But, trying to activate the database was continuously getting SQL1229 from one of the members.

SQL1229N  The current transaction has been rolled back because of a

      system error.

The database was restarted with,

db2_all "db2 restart database DW_AUDIT"

Sill problem was present from one of the partitions,

Unauthorized use of or access to the system is prohibited.

SQL1229N  The current transaction has been rolled back because of a System error.  SQLSTATE=40504

myhost: db2 restart database ... completed rc=4

The db2diag.log showed,

2017-08-02-10.43.28.293536-240 I5770E530           LEVEL: Warning

PID     : 23166                TID  : 46920269556032PROC : db2sysc 14

INSTANCE: db2inst1             NODE : 009          DB   : MYDB

APPHDL  : 1-47418              APPID: <>

 EDUID   : 121116               EDUNAME: db2agent (idle) 14

FUNCTION: DB2 UDB, base sys utilities, sqeApplication::AppSecondaryStartUsing, probe:611

DATA #1 : <preformatted>

Agent executing rqst type = 0x8000001e; acbInfo = 0x40; app_status = 0xa; sqlcode = -1229

It showed all the members were successfully doing deactivation except the problem partition.  Following messages from all the other partitions,

SQL1496W  Deactivate database is successful, but the database was not activated.

Deactivation was stuck in the problem member which was causing issues when trying to activate it back.

Once the entire instance restarted the problem got resolved.