I’m assuming we’re talking about OpenSSH. There are other things out there like Putty and (incredibly) commercial versions of SSH, but thankfully I don’t have to touch those.
The easiest mechanism for matching up the keys are the fingerprints. The fingerprint is always of the public key. There is no fingerprint for the private key. There doesn’t need to be, as the private key contains enough information to generate the public key as well (for example, ssh-keygen -y -f private.key). So, when you ssh-add a private key and then run ssh-add -l, you are seeing the fingerprint of the public key. You can then use ssh-keygen -l -f publickeyfile on the remote host to generate a fingerprint to compare against the fingerprint returned by ssh-add -l.
Annoyingly the -f option to ssh-keygen does not support STDIN, so you have to write the test key to a file before generating a fingerprint. This complicates, say, generating fingerprints for all of the public keys in an authorized_keys file on a host or fingerprinting a private key with ssh-keygen -y private.key | ssh-keygen -l -f –