IBM i Health Checklist in Event of Unattended/Unexpected IPL – Kyndryl Cloud Uplift Support

Note: Before you take any corrective actions, please confirm regional component health at Kyndryl Cloud Uplift Status Page.

Actions

If the IBM i VM is accessible via 5250 or SRA:
1. Perform a graceful shutdown
2. Power on the VM from the Kyndryl Cloud Uplift portal
If the IBM i VM is not accessible:
1. Power off from the Kyndryl Cloud Uplift UI
2. Power on the VM from the Kyndryl Cloud Uplift portal

Initial Checks:

Once logged in, please perform the following:
- Check for any new WRKPRB entries
- Run WRKDSKSTS to verify if all disks match the count before and after
- Run WRKPTFGRP and compare it with the old data to ensure it's matching
- Run GO LICPGM > Option 10 to verify all Licensed Programs (LLPs) match the pre-IPL state

System Object and Views Check:

It is possible data in RAM was not written to disk during an unexpected event. In such cases, it is recommended to perform the following checks:

Check Command

The following commands should return 0 for all output, indicating a successful check. If not, proceed to the online recovery steps
```
CALL QCMD   
CALL QSYS/QSQSQLCAT PARM(*CHECK)
```

Online Recovery

If the check command identifies errors, attempt online recovery

Enable job logging and program logging:

CHGJOB LOG(4 00 *SECLVL) LOGCLPGM(*YES)

Check database referential integrity:
```
RCLDBXREF OPTION(*CHECK)
```
Sample output:
Verify system integrity:
```
CALL QSQIBMCHK
```
Sample output:

The red boxes in the following screenshot represent if there are any missing objects.
```
CALL QSYS/QSQSYSIBM CHECKONLY
```
Sample output:
Post verification
CALL QSQIBMCHK
This time the output should show 0 for the missing objects, if you still see any missing objects then please proceed with the Offline recovery process.

Offline Recovery (If Online Recovery Fails)

The VM must be in a restricted state before performing these steps.

Reorg database cross-reference

With the system in a restricted state, perform the following command
```
RCLSTG SELECT(*DBXREF)  // Run only in a Restricted state
```
Note: Execution time can take minutes to several hours depending on the workload
Perform an IPL and run the following checks/commands
```
RCLDBXREF OPTION(*CHECK)
CALL QSYS/QSQIBMCHK
```
If missing objects persist, proceed to additional recovery procedure

Additional recovery procedure

Before performing these steps, it is recommended to take a template of the VM.

Add library QSYS2924 (if it exists on the system):

CHGSYSLIBL LIB(QSYS2924) OPTION(*ADD)  // Only if you have library QSYS2924

Enable job logging and program logging with CCSID(37):
```
CHGJOB LOG(4 00 *SECLVL) LOGCLPGM(*YES) CCSID(37)
```
Set application security group to none:
```
SETASPGRP *NONE
```
Check database referential integrity:
```
RCLDBXREF OPTION(*CHECK)
```

Rebuild QSYS2 library (if necessary):

CALL QSYS2/QSQXRLF ('DLT' 'QSYS2')
CALL QSYS2/QSQXRLF ('CRT' 'QSYS2')

All the missing objects should be recovered at this point. If you still have any issues please contact Kyndryl Cloud Uplift Support.

Damaged Object check

1. Create the library to contain the output file:

CRTLIB IBMKENT

2. Display object description:

DSPOBJD OBJ(*ALL/*ALL) OBJTYPE(*ALL) OUTPUT(*OUTFILE) OUTFILE(IBMKENT/FIND)

3. Start interactive SQL:

STRSQL

Note: Other SQL interfaces also work.

4. Search for damaged objects with the following SQL query:

SELECT ODLBNM, ODOBNM, ODOBTX FROM ibmkent/find WHERE odobdm <> '0'

This will give you a list of damaged objects if any.

If you find any damaged objects that are related to user, please perform a restore from the recent save.

Related articles