解決：https://askubuntu.com/questions/762541/ubuntu-16-04-ssh-sign-and-send-pubkey-signing-failed-agent-refused-operation/844089
Looks like an ssh-agent is running already but it can not find any keys attached. To solve this add the private key identities to the authentication agent like so:
ssh-add
Then you can ssh into your server.
in addition, you can see the list of fingerprints of all identities currently added by:
ssh-add -l