I have changed it to use 80 and 443 instead of 8530 and 8531 using wsusutil.
Did you confirm that it was working on port 80 without SSL after running 'usecustomwebsite'?
Perhaps this is not an SSL issue but a simple site migration / console connection issue?
And then I can't connect to the server in the console.
Assuming the site was working on port 80 prior to enabling SSL, the logical conclusion here is that something you did (or did not do) while implementing the SSL configuration is the cause. Either way, I'd say the best approach here is to completely rollback the SSL implementation, confirm that the environment works without SSL, and then re-implement the SSL configuration.
And, once more, to my original points.. nothing in the procedure cited discusses configuring the WSUS Server as anSSL CLIENT so you also need to perform those steps.
From the WSUS Deployment Guide - Configure WSUS:
Configure SSL on client computers
When you configure SSL on client computers, you should consider the following issues:
- You must include a URL for a secure port on the WSUS server. Because you cannot require SSL on the server, the only way to make sure that client computers can use a security channel is by using a URL that specifies HTTPS. If you use any port other than 443 for SSL, you must include that port in the URL also.
- The certificate on a client computer must be imported into the Local Computer Trusted Root CA store or Automatic Update Service Trusted Root CA store. If the certificate is imported to the Local User's Trusted Root CA store only, Automatic Updates will fail server authentication.
- The client computers must trust the certificate that you bind to the WSUS server. Depending on the type of certificate that is used, you might have to set up a service to enable the client computers to trust the certificate that is bound to the WSUS server.
Lawrence Garvin, M.S., MCSA, MCITP:EA, MCDBA
SolarWinds Head Geek
Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2014)
My MVP Profile: http://mvp.microsoft.com/en-us/mvp/Lawrence%20R%20Garvin-32101
http://www.solarwinds.com/gotmicrosoft
The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.