Ex-TAC Secrets: Troubleshooting the Top 5 Cisco ISE 3.5 Upgrade Failures

Upgrading to Cisco ISE 3.5 introduces substantial schema restructuring, persona state-machine modifications, and stricter hardware IOPS thresholds that frequently cause upgrades to stall at the database index migration phase. Resolving these failures requires CLI-level diagnostic interrogation of ADE-OS services, Oracle table space health, and automated rollback workflows.

Relying solely on web-based progress bars obscures underlying database lockouts and daemon crashes during maintenance windows.

1. Database Indexing and Schema Migration Stalls

The most common failure occurs when the upgrade progress bar stalls between 75% and 80% during the Secondary Policy Administration Node (PAN) database migration.

Root Cause Mechanics

ISE 3.5 implements extensive database schema rebucketing to support high-scale contextual telemetry and AI-driven profiling models. During the schema upgrade, the ADE-OS migration script drops and rebuilds database indices across the operational tablespace.

If existing historical reporting tables contain fragmented indices, or if inter-node latency between PANs experiences packet jitter exceeding 10 ms, the Oracle database listener encounters TNS-12535: TNS:operation timed out. The management web interface fails to reflect the backend timeout and remains indefinitely in an "In Progress" state.

CLI Diagnostics and Validation

Verify raw database and daemon status directly from the console:

# Interrogate active ADE-OS application services
show application status ise | include Database|Process

# Inspect real-time database migration logs
show logging application upgrade-mon.log tail

Confirm that both Database Listener and Database Server report as running. If the upgrade hangs, inspect /opt/CSCOcpm/logs/upgrade-db.log:

# Check for active Oracle index lockouts
tech-support application | include ORA-|TNS-

Remediation Protocol

1. Before initiating major upgrades, execute index defragmentation and table space pruning while on the source version:

   application reset-config ise
   

2. Verify virtual machine storage performance. The underlying storage cluster hosting the PAN must deliver at least 300 sustained write IOPS with storage latency below 5 ms. Throttled storage volumes invariably cause index rebuilds to exceed the script watchdog threshold.

2. ADE-OS Zombie Persona Services

Following the initial reboot into ISE 3.5, nodes can enter an inconsistent persona state where the node claims PAN status but core application services refuse to bind to local sockets.

Root Cause Mechanics

When transitioning from ISE 2.x or 3.x to 3.5, the local configuration database must deregister legacy runtime components and register the updated cpm-runtime container. If the node encounters an unclean shutdown or memory pressure during the first post-upgrade boot, the legacy service lockfile (/opt/CSCOcpm/run/ise-service.pid) persists. ADE-OS assumes the service is already active, creating a zombie process state.

Resolution Steps

Access the root shell or execute application process resets:

# Stop application services cleanly
application stop ise

# Inspect port binding status on RADIUS and HTTPS listeners
show ports | include 443|1812|8443

# Restart application stack
application start ise

Allow 15 to 20 minutes for JVM initialization. Monitor initialization via show application status ise. All node personas must display running before re-enabling node replication.

3. PKI Certificate Chain Validation Failures

ISE 3.5 enforces strict RFC 5280 certificate path validation and deprecates SHA-1 intermediate certificates across all node communication channels.

Root Cause Mechanics

Prior versions tolerated missing intermediate certificates in the local trust store if the remote peer supplied them during the TLS handshake. ISE 3.5 mandates that the complete certificate chain (Root CA and all Issuing Sub-CAs) be explicitly present in the Trusted Certificates store with the Trust for authentication within Cisco ISE and Trust for client authentication and Syslog flags asserted.

If an intermediate certificate is absent, inter-node pxGrid and database replication fails immediately with: ERROR: Peer certificate validation failed: certificate signed by unknown authority.

Pre-Upgrade Certificate Verification

Inspect installed certificates prior to upgrading:

# Export and verify the system certificate expiration and subject alternative names
show crypto certificate system-certs

Ensure the Subject Alternative Name (SAN) includes both the fully qualified domain name (FQDN) and the short hostname for every node in the deployment.

4. Hardware Resource Constraints and IOPS Throttling

Upgrading virtual ISE nodes allocated with legacy minimum specifications (e.g., 16 GB RAM and 4 vCPUs) causes critical failures during 3.5 post-install database indexing.

Production Resource Requirements

For reliable operation under ISE 3.5:

• Memory: Minimum 32 GB RAM for dedicated PSNs; 64 GB RAM for combined PAN/MNT nodes.
• CPU: 8 to 16 vCPUs with 2.4 GHz reservations.
• Storage: 300 GB to 600 GB high-IOPS solid-state storage.

During the upgrade sequence, memory utilization spikes to 95%. When swap memory is exhausted, the Linux Out-Of-Memory (OOM) killer terminates the migration sub-process, corrupting the database.

5. Post-Upgrade Authorization Profile and SGT Drift

When the database migration converts legacy Authorization Profiles to the updated ISE 3.5 policy model, security group tag (SGT) assignments and dynamic downloadable ACLs (dACLs) can fail to map if custom dictionary attributes have changed.

Prevention and Post-Upgrade Diffing

1. Export all Authorization Profiles and Policy Sets to XML/CSV from the primary PAN prior to initiating maintenance.
2. Immediately post-upgrade, run an automated configuration diff against the exported baseline.
3. Validate that critical Authorization Profiles retain their explicit SGT bindings rather than falling back to default untagged traffic policies.

Structured Upgrade Verification Checklist

1. Execute full operational and configuration backups off-box to an external repository via SFTP.
2. Confirm DNS forward and reverse resolution across all PAN, MNT, and PSN node IP addresses.
3. Validate database consistency using CLI commands prior to kicking off upgrade scripts.
4. Verify VM disk latency and IOPS allocations to prevent database indexing timeouts.
5. Upgrade the Secondary PAN first, verify persona stability, promote to Primary, and sequentially upgrade remaining PSNs.