I just want to provide a clear answer to this thread based on my observation.
In a nutshell:
On Windows Server 2012/2012 R2, the WSUS SSL configuration command WSUSUtil.exe configuressl must be set to the WSUS host server's local host name (FQDN), and the WSUS admin console when executed locally, must also connect to the local host name, in order
to work properly. This may require the IIS to be configured with both public domain name and local host name SSL bindings.
Detailed Explanation:
A company contoso.com wishes to secure their WSUS server with SSL running on Windows Server 2012/2012 R2 and also make sure that people from outside of the company network (with laptops etc) can still connect to the SSL WSUS server for updates over the
internet.
To achieve this goal, the company setup the public domain name wsus.contoso.com that points to the public IP of their WSUS server, and issued or purchased a trusted certificate for wsus.contoso.com. All firewalls/NAT/port forwarding if applicable are setup.
The WSUS IIS site has been configured with the correct binding and certificate for wsus.contoso.com, and selected applications have been configured with "Require SSL" per Microsoft's documentation. Each client computer has been pointed to https://wsus.contoso.com:8531
via group policy for Windows update, and configured to trust wsus.contaso.com's certificate (either an automatically trusted public CA issued certificate or a manually self-signed certificate).
However, the company's WSUS server itself has its own host name, oreo.ad.contoso.com, which is apparently different from wsus.contoso.com. Nonetheless, the company used the command
WSUSUtil.exe configuressl wsus.contoso.com
to setup the WSUS SSL because naturally wsus.contoso.com is what they intended to use. However, errors occur when the WSUS Admin Console is launched from the local WSUS server to connect to wsus.contoso.com at port 8531 for management. Windows Event Log also
shows several "... services not running" errors when WSUS service starts.
Solution:
Use the command
WSUSUtil.exe configuressl oreo.ad.contoso.com
instead to configure the server. In IIS WSUS site, add a binding for oreo.ad.contoso.com at port 8531, listen on ALL IP's or at least the local IPv6 IP (this is because by default the local FQDN resolves to IPv6 in Windows) with an appropriate certificate trusted by the server. Please note that the certificate does *not* necessarily have to list oreo.ad.contoso.com as the main "issued to" name. It can be listed as a Subject Alternative Name (SAN) (verified using Godaddy's SAN SSL). I have not personally tried a wildcard SSL so not sure if that also works. This means if the same certificate used for wsus.contoso.com also covers oreo.ad.contoso.com via SAN, then the same certificate can be used. Otherwise, a separate certificate for oreo.ad.contoso.com needs to be issued or purchased, and must be correctly installed/trusted on the server for use with IIS.
If configured correctly, the IIS site should have three bindings: the default non-SSL http binding (do not remove this as it is required for WSUS to function correctly), a SSL binding for wsus.contoso.com on port 8531, and a SSL binding for oreo.ad.contoso.com on port 8531 (all IP). The local WSUS Admin Console should launch correctly (if no server is listed, add server oreo.ad.contoso.com, check use SSL and port 8531).
So overall, the http://wsus.contoso.com:8531 address will be used for Windows clients seeking to obtain updates from the server, and the oreo.ad.contoso.com entry is used exclusively for the WSUS server itself and its local admin console.
Hope this will help someone who got stuck in a similar situation.