How to Rapidly Remedy the OpenSSH Vulnerability

Last week, Qualys published a security advisory exposing vulnerabilities (CVE-2016–0777 and 0778) in OpenSSH client versions 5.4 to 7.1. When exploited, these vulnerabilities can result in the theft of private key material. To eliminate the vulnerability, administrators can either patch OpenSSH with version 7.1p2 or configure existing SSH clients to disable the exploitable feature.

Discovering vulnerable versions of software like OpenSSH in an enterprise of any scale is a major problem, as most enterprises lack the basic ability to see all the endpoints within its network. Anything you’d want to do to a single system, such as killing processes, quarantining systems or deploying patches, across all the endpoints can be difficult, which is one of the reasons Tanium is a great resource.

Initial research

After reading the release by Qualys, we needed to try and find more information about workarounds, ideally from the folks providing the patch. Theo de Raadt, an authority on the topic, posted an undocumented configuration directive UseRoaming no that prevents the bug from surfacing. At first glance, it appeared the workaround would be trivial: simply whip up a shell script to add the entry to the configuration file. A related discussion astutely noted that appending the configuration to the config file might fail if the configuration file defines host patterns. Thus, the script must append the configuration setting for all hosts:

Quickly deploying a script to vulnerable machines

Modifying active configuration files en-masse should make any rational administrator nervous. This is not something to be done without a measure of caution and a back-out plan. If we modify the file, how can we undo the changes quickly on one or all systems? We began by backing up our existing configuration to a new file and then applying the UseRoaming no fix to ssh_config:

In early testing, we discovered that the -e flag to echo is not available on all systems, so we opted to use the $’’ construct for portability. So, we enhanced the command with the following:

We tested again on a single system. Here’s an image depicting the last lines of the old file, manually executing the fix via the command line and the last lines of the modified config file:

We tested the edge case of appending the fix multiple times to the config file and determined that the ssh client continues to function with the redundancy. Operationally, this a good safeguard, but we noted this as something to clean up later in development.

Fail safety

The simplest thing to do is copy the backup file over the config file. Ideally, we would never have to do this, but it’s an ounce of prevention that could pay serious dividends and provide additional confidence as we quickly remediate across the enterprise.

The script is basic:

Determine scope of OpenSSH vulnerability

Now that we have scripts to apply and remove the fix, it’s time to write another script to determine applicability. We will wrap this script in a Tanium sensor so that we can run the query across the entire network of endpoints. Because multiple tests showed the workaround to have a positive impact, the simplest method available to us is to check that the ssh_config contains the output that was intended from the fix.

For portability, the implementation method we chose was an awk one-liner, which prints the lines of a file that exist between other regular expression matches:

This matches only cases where the full three-line fix sequence has been written to the file. We opted to return a string of “Potentially Vulnerable” instead of “Vulnerable” because our solutions works independently of the OpenSSH version. By casting a wider net, independent of version, we won’t need to update our code should researchers continue to expose additional vulnerabilities to the same Roaming feature of OpenSSH. (The Roaming feature is never useful in practice, so a configuration-based disablement strengthens security without negatively impacting operations.) Some might recall Shellshock, which took a couple of tries and software updates before the community got everything right. Our approach provides resilience against this outcome repeating with the SSH client.

Here’s the output from our test against a default ssh_config file:

And now against one that we had just fixed:

This looks good! We then wrapped these scripts into a Tanium Sensor we will call “SSH Key Leak Workaround Status.”

User-specific SSH configs and polishing the content

While researching, we didn’t find many in the community discussing machines on which users may have defined their own ~/.ssh_config files. Because these settings would supersede global configs, fixes must be applied here to close the vulnerability. This led us to further enhancements of our content that also loops through home directories. Because of platform differences, we ended up with two different versions of the scripts to support OS X and *nix flavors equally.

Additionally, we improved the logic from our earlier scripts that applied the fix so that we don’t apply it multiple times and bloat config files. They’re long enough to no longer tidily fit into a blog post, but still just over 40 lines of code and comments.

Our end result is a solution that identifies systems as “Potentially Vulnerable” across *nix and OS X systems in seconds. If any resident SSH config files do not contain the mitigation to secure the enterprise, users can apply or revert the fix to any or all systems across the enterprise in just a couple of minutes.

Tanium the platform

Tanium makes it easy to address the OpenSSH vulnerability because it provides the industry’s only real-time visibility into the status of all endpoints, regardless of the number of machines. The platform is the fastest and easiest way to implement the approach described in this post: start with data from a reputable source, develop scripts, scale testing and refine and then fully deploy what essentially becomes a new feature of the platform.

Let us know what you think, or please reach out to your Technical Account Manager if you’re interested in running the content in your own deployment. If you’re unfamiliar with Tanium and would like to learn more, please contact us for a demo.

Rory Prendergast, Sr. Director
Peter Oelschlaeger, Sr. Director

Interested in seeing Tanium in action? Schedule a one-to-one demo or attend our weekly webinar. Talk to our Tanium experts at our upcoming events.