usb4java-javax-examples | Example programs demonstrating how to use | Learning library

openssl pkeyutl -sign with an RSA private key and -pkeyopt digest:$alg does steps 2-5 of EMSA-PKCS1-v1_5 plus the private modexp (i.e. 8.2.1 step 2 using RSASP1 from 5.2.1); without that -pkeyopt it does not do step 2, which encodes the hash value in a simple ASN.1 structure.

The Java Signature algorithms that include a hash like SHA1withRSA do all steps of EMSA-PKCS1-v1_5 plus modexp, while the scheme NONEwithRSA does only steps 3-5 plus modexp, and neither of these matches what you want. If you can't supply the data to let Signature do the hashing, you'll need to do step 2 yourself and then NONEwithRSA. Although ASN.1 encoding in general can be complicated (and decoding even more so), this case can be done simply by concatenating a prefix determined entirely by the hash algorithm to the hash value; see note 1 on page 47.

Meta: there have been lots of Qs about the differences in RSA v1_5 signatures between openssl rsautl, pkeyutl with and without -pkeyopt digest, or dgst -sign and other systems like Java, all to do with the issue of including or not including the ASN.1 encoding step (and many of them older than rfc8017!) but I can't find any that is a good dupe for this Q.