How to find out which key to use to decrypt a S/MIME message
Daniel Nashed – 23 October 2021 09:53:29
While troubleshooting a S/MIME issue for a customer I had to figure out why mails have not been decrypted.
Similar to Notes encryption, S/MIME uses a symmetric "session" key to encrypt the message.
This symmetric key is encrypted with the public key of each recipient.
But how does S/MIME know which key to use to decrypt the symmetric if the user has more than one key?
The solution is to find the serial number of the certificate used to encrypt the message.
An easy way on Windows would be to use the following command to get the serial number.
certutil.exe smime.p7m|findstr Serial
Serial Number: 1e4a7e00d54d8237dd839a14afae3162
(the find string might differ in other languages)
This serial number is what you should look for in your e-mail certificate (which hopefully has a matching key).
You find the matching certificate for my example at the end of the post.
Now that we know what to search for, let me show the OpenSSL way.
It's a bit more complicated but using OpenSSL I can demonstrate encryption and analysis with an example.
And it also helps to understand the steps Microsoft certutil.exe performs automatically for you.
-- Daniel
Encrypt some text via OpenSSL
First encrypted some text with an e-mail certificate I created with my CA.
The result of this encryption is in message format.
echo The yellow fox jumps .. > plain.txt
openssl smime -encrypt -aes-256-cbc -in plain.txt -out mail.msg daniel.pem
Get PKCS7 from message format
Getting the PKCS7 information from a message format does not need any extra format specification.
openssl smime -in mail.msg -pk7out -out msg.pk7
smime.p7m - Binary format needs the encoding specified
When exporting a smime.p7m attachment from Notes, you have to specify the binary format (DER).
openssl smime -in smime.p7m -inform DER -pk7out -out msg.pk7
To get the binary format of this test message, you can take the base64 part of the msg and convert it to binary format (I copied the base64 data into msg.b64).
openssl enc -d -base64 -in msg.b64 -out smime.p7m
Parse the PKCS7 information
Once you have the PKCS7 data, you can dump the results:
openssl asn1parse -in msg.pk7
Most of the details in the file are quite cryptic -- It's a dump of the ASN1 structure.
But after the CA certifier, you find the serial number, you can match with your certificate.
--------------------------------------------------------
Example output (email-addressed changed to acme.com):
0:d=0 hl=4 l= 809 cons: SEQUENCE
4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-envelopedData
15:d=1 hl=4 l= 794 cons: cont [ 0 ]
19:d=2 hl=4 l= 790 cons: SEQUENCE
23:d=3 hl=2 l= 1 prim: INTEGER :00
26:d=3 hl=4 l= 705 cons: SET
30:d=4 hl=4 l= 701 cons: SEQUENCE
34:d=5 hl=2 l= 1 prim: INTEGER :00
37:d=5 hl=3 l= 164 cons: SEQUENCE
40:d=6 hl=3 l= 143 cons: SEQUENCE
43:d=7 hl=2 l= 11 cons: SET
45:d=8 hl=2 l= 9 cons: SEQUENCE
47:d=9 hl=2 l= 3 prim: OBJECT :countryName
52:d=9 hl=2 l= 2 prim: PRINTABLESTRING :DE
56:d=7 hl=2 l= 12 cons: SET
58:d=8 hl=2 l= 10 cons: SEQUENCE
60:d=9 hl=2 l= 3 prim: OBJECT :stateOrProvinceName
65:d=9 hl=2 l= 3 prim: UTF8STRING :NRW
70:d=7 hl=2 l= 15 cons: SET
72:d=8 hl=2 l= 13 cons: SEQUENCE
74:d=9 hl=2 l= 3 prim: OBJECT :localityName
79:d=9 hl=2 l= 6 prim: UTF8STRING :Hilden
87:d=7 hl=2 l= 16 cons: SET
89:d=8 hl=2 l= 14 cons: SEQUENCE
91:d=9 hl=2 l= 3 prim: OBJECT :organizationName
96:d=9 hl=2 l= 7 prim: UTF8STRING :NashCom
105:d=7 hl=2 l= 11 cons: SET
107:d=8 hl=2 l= 9 cons: SEQUENCE
109:d=9 hl=2 l= 3 prim: OBJECT :organizationalUnitName
114:d=9 hl=2 l= 2 prim: UTF8STRING :IT
118:d=7 hl=2 l= 26 cons: SET
120:d=8 hl=2 l= 24 cons: SEQUENCE
122:d=9 hl=2 l= 3 prim: OBJECT :commonName
127:d=9 hl=2 l= 17 prim: UTF8STRING :R2 NashCom MiniCA
146:d=7 hl=2 l= 38 cons: SET
148:d=8 hl=2 l= 36 cons: SEQUENCE
150:d=9 hl=2 l= 9 prim: OBJECT :emailAddress
161:d=9 hl=2 l= 23 prim: IA5STRING :certificates@acme.com
186:d=6 hl=2 l= 16 prim: INTEGER :1E4A7E00D54D8237DD839A14AFAE3162
204:d=5 hl=2 l= 13 cons: SEQUENCE
206:d=6 hl=2 l= 9 prim: OBJECT :rsaEncryption
217:d=6 hl=2 l= 0 prim: NULL
219:d=5 hl=4 l= 512 prim: OCTET STRING [HEX DUMP]:2724735F0707CC2B01C05CF7BBFD8C130DB96F5E90AA591AF3A63A0ED697E83541474911834154DBBA044E497D5470D31D98688F1F05E851F9E192E3A717CB10F1422C74F809E28C290B198162D093315F8F20AD30F7718D27E2AE9D0C07B8E2EBE9EEE0E884A934282F8A28EF1019C08B63A7C3913B7DE4215EA442E396C548F96E3BA3E4CF8AAE25A9DD16FF4F7A4CA289156595F14781B8856F7841A0DDD27FE006E4242A8B4EB2C10F3A61C17A1FAD5CAE78BAE745D6819B8DFC291CA0DF4E2BC2E44841A4A3B236B71A0B7D865E44B81B0EBF130A61333847D3C05A5CA52719AA3A12B4D3EA8D528D2B4EC7E7455499A9134E903705F4FE163832A21ED08176EA1C3DCA70000F4A8A3F71A797F45EF801995A8E7A3AED90C46A097C71DBB9B0AA6B65FF72D15996B4D6F30FD3EC1DF993118029202E785840267A59F118F4D36E078827CF72C5F415E12A9888E305179DFDADBD097E301178DB45181EDAE7E24FAEE3F2BCBB9572819B41E7A2F8C63B6D98AB2F1B8E0E763E33DD367AB674F5E882DBE0E05EC452AF94E6EACF35DF8A8E87FF1350B9A0AA6665042D5BE6ADFE0EFCB579128986490323019D6458160BB907C23DBD57E6EA238353931FDA62AD64AFC1C8979EEE471E52DDC16B8CDEDA7A632DA6C49D354C888B7B26431BA1FD476443D39FB56E95CCEDF1707B5B91FDFCA493B85E3A1053DDEB2011040B
735:d=3 hl=2 l= 76 cons: SEQUENCE
737:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-data
748:d=4 hl=2 l= 29 cons: SEQUENCE
750:d=5 hl=2 l= 9 prim: OBJECT :aes-256-cbc
761:d=5 hl=2 l= 16 prim: OCTET STRING [HEX DUMP]:5E3D08C36CDC0848F6602059DBA444A3
779:d=4 hl=2 l= 32 prim: cont [ 0 ]
--------------------------------------------------------
Certificate
#0
Subject : DE/NRW/Hilden/NashCom/IT/Daniel Nashed/nsh@acme.com
SAN e-mail : nsh@acme.com
Iss e-mail : certificates@acme.com
Issuer : DE/NRW/Hilden/NashCom/IT/R2 NashCom MiniCA/certificates@acme.com
Not Before : 2021.10.22 09:00:46
Not After : 2022.10.23 09:00:46 (expires in 364.0 days)
Serial : 1E4A7E00D54D8237DD839A14AFAE3162
Sign Alg : sha256WithRSAEncryption
KeyUsage : DigitalSignature
Extensions : KeyUsage, ExtKeyUsage
ExtKeyUsage: E-mail Protection
Key : RSA 4096 bit
AuthKeyId : 00:A2:6E:B2:B3:8D:0E:C7:0E:E3:7D:65:C8:0A:9E:B7:8E:04:E5:B1
SubjKeyId : 9D:F1:7D:07:6C:91:26:1D:91:83:BC:00:7B:CF:B2:AE:7A:EC:1B:C2
MD5 : 96:19:0C:88:CE:A2:B9:00:22:E7:DC:49:B7:10:8E:03
SHA1 : 9F:C5:11:50:23:59:79:FB:A4:8A:C5:8C:DF:50:AC:2B:C5:FF:0E:04
SHA256 : 58:74:4B:A0:22:14:1B:87:76:26:DC:14:B8:E8:8A:0F:7F:E6:0A:2E:87:52:9F:3E:24:46:2C:05:6C:F1:87:7E
- Comments [0]
![[HCL Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}