What about binaries that might have OpenSSL statically linked? Even if you update the system libraries you could still be vulnerable.
Or the appliance (or out-of-band management card, print server, etc, etc) that you can't login to in order to be able to tell what version of the libraries it's using.
Or the system you did update the libraries on, but forgot to restart the webserver to pickup the change?
The best answer is normally going to be to do both - check the system itself to make sure it doesn't have an impacted version installed, but also check the individual services to make sure they are not impacted and/or have been fixed.