What's the error (support code) on the WM5 device?  How to troubleshoot Server ActiveSync  How to troubleshoot server ActiveSync HTTP error codes

Have you tried the following article? Error message when you use ActiveSync to synchronize a Windows Mobile 5.0-based device: "The security certificate on the server is invalid"

Have you tried manually installing the certificate on the WM5 device?  How to install root certificates on a Windows Mobile-based device

Have you verified the correct configuration (and requirement of SSL) of your IIS virtual directories?  http://www.pocketpcfaq.com/faqs/digital_certificates.htm

If you want a workaround to so your WinMo devices don't need to use SSL but keep the ActiveSync vDir secure, read this article: http://support.microsoft.com/kb/817379/en-us. 

Side Note: If you have configured IIS with a wildcard (*.domain.com) certificate, unfortunately WM5 devices are not supported.  However, WM6 devices do support wildcard certs for EAS SSL.