To encrypt a file with RSA public key, run command:
openssl rsautl -in plain_file -out encrypted_file -inkey public_key.pem -pubin -encrypt
where plain_file is the file you want to encrypt, encrypted_file is the output. public_key.pem is the RSA public key. -pubin tells that the key file is a public key.
To decrypt the output encrypted_file from the last command, we can only use the paired private key.
openssl rsautl -in encrypted_file -out decrypted_file -inkey private_key.pem -decrypt
If private_key.pem and public_key.pem are not paired, trying to decrypt will get errors. To generate a pair of RSA private and public keys, see this post.
On Linux, you can run man rsautl to see the manual of the commands.
Now, run command diff plain_file decrypted_file to verify that they are the same.
We also have a snippet of C++ code in this post to show how to decrypt encrypted_file with the OpenSSL library.