Zyxel, a prominent provider of networking solutions, has issued a warning about a flawed security signature update that has triggered severe issues in its USG FLEX and ATP Series firewalls. These problems include devices being stuck in a boot loop, system crashes, and other significant functionality issues.
A Troubling Update Causes Major Disruptions
Zyxel’s advisory highlights the impact of an Application Signature Update distributed between January 24 and January 25. The problematic update has led to various complications for affected devices, including:
- Devices entering reboot loops.
- System login failures, with errors like “504 Gateway timeout.”
- High CPU usage.
- “ZySH daemon is busy” messages in system logs.
- Console issues, including inability to execute commands.
- Frequent coredump messages during device operation.
Affected devices also display flashing system LEDs, signaling critical system issues. Zyxel clarified that this disruption is not related to a Common Vulnerabilities and Exposures (CVE) flaw or a broader security vulnerability. However, the practical challenges for administrators are substantial.
Impacted Devices and Scope of the Problem
The issue appears limited to USG FLEX and ATP Series firewalls operating on ZLD Firmware Versions, provided they have active security licenses. Notably, Zyxel devices on the Nebula platform or USG FLEX H (uOS) series remain unaffected, offering some relief to administrators managing mixed environments.
As first reported by Born City, the situation underscores the significant inconvenience caused by such errors, particularly when remote management is disabled.
Recovery: A Manual Process Requiring On-Site Access
Unfortunately, resolving the problem demands physical access to the affected firewalls. Zyxel specifies that recovery requires an RS232 serial cable to connect directly to the device console.
Key steps for recovery include:
- Backing up the device configuration.
- Downloading and applying a specially provided firmware update.
- Restoring the backed-up configuration through the web GUI.
These steps ensure that the firewall resumes normal functionality, but they involve manual intervention, which may disrupt operations for organizations with numerous distributed devices.
Important Considerations
- Physical access is mandatory for recovery. Remote troubleshooting is not viable for this specific issue.
- Zyxel advises admins to review detailed instructions in its advisory before proceeding with recovery efforts.
For organizations reliant on Zyxel firewalls, the need for manual intervention raises logistical challenges, particularly in distributed setups or environments with limited technical staffing.
Zyxel’s Response: Support and Transparency
Acknowledging the gravity of the issue, Zyxel is taking proactive measures to support affected customers. The company has scheduled Microsoft Teams Open Question Sessions on Saturday, January 25, to provide real-time assistance. These sessions are set to run from 9 a.m. to 12 p.m. and 1 p.m. to 5 p.m. (GMT +1). This initiative underscores Zyxel’s commitment to resolving the issue while ensuring that administrators have access to expert guidance.
Despite these efforts, Zyxel has not disclosed the underlying cause of the faulty update or the exact number of affected devices. Additionally, BleepingComputer has reached out to the company for further information, but no immediate response has been received.
Practical Implications for Admins
The incident serves as a reminder of the challenges posed by automatic updates in critical infrastructure. While such updates are intended to enhance security, errors like these highlight the need for contingency planning. Administrators managing Zyxel firewalls should:
- Ensure backup configurations are always updated and stored securely.
- Maintain a supply of necessary recovery tools, including RS232 serial cables.
- Regularly monitor vendor advisories to preemptively address potential issues.
Minimizing Future Risks
For IT administrators, this incident underscores the importance of balancing automation with oversight. While automation simplifies updates, it also increases the potential for widespread disruptions when errors occur. Mitigation strategies should include staggered rollouts, robust testing protocols, and fallback measures.
Zyxel, for its part, will likely face scrutiny over the robustness of its update verification processes. For now, the company’s immediate response and willingness to engage with affected users are positive steps. Still, the long-term impact on user trust remains to be seen.