[Crypto] SSL/TLS, part 1: Toy TLS 1.2 client in ~1200 SLOC of Python.
This a ToyTLS client supports TLS 1.2 (but not 1.3).
It can connect to many high-profile webservers.
It supports:
TLS_RSA_WITH_NULL_SHA,
TLS_RSA_WITH_AES_128_CBC_SHA,
TLS_RSA_WITH_AES_256_CBC_SHA256.
It has been crumpled by me into ~1200 SLOC of Python.
The only one important external dependency is required: a well-known cryptography.io Python module.
Many books use 'SSL/TLS' name for the protocol.
But for brevity, I'll call it only 'TLS'.
SSL is just the old version of the TLS, of vice versa, the TLS is a new version of the SSL.
A 'cert' word is used here for brevity, short for 'certificate'.
TLS is similar to SSH in essense.
I would go that far and say that TLS and SSH are similar because they have similar very goals.
But SSH is somewhat simpler.
Probably, SSH can be better studied before TLS -- as I did.
SHA1 is still used by many high-profile servers, despite the fact it was cracked.
I use it as well, just to make hex dumps smaller/shorter.
Of course, we should switch to at least SHA2-256, which is also supported by ToyTLS.
NULL cipher
TLS protocol allows using TLS_RSA_WITH_NULL_SHA ciphersuite, which doesn't encrypt packets.
But it must be allowed explicitely.
Of course, it's easier to start learning on simpler versions of protocol.
So I did.
OpenSSL can work as webserver.
Here I generate cert pair for it, turn on debug tracing and explicitely allow the NULL cipher:
This unencrypted traffic can be inspected by Wireshark.
This helped me a lot.
Also, you can use a great service: badssl.com.
It has a server that supports NULL cipher: https://null.badssl.com
(Firefox wouldn't connect to it, because it disallows NULL cipher.)
OK, let's start our journey.
All TLS packets has 6-byte header: packet/message type (byte), protocol (16 bits) and length (16 bits).
ClientHello
Client sends some important data:
32 bytes of random data.
Ciphersuites supported.
Extensions. Like, options supported.
ToyTLS handles: encrypt_then_mac and extended_master_secret.
Another important extension is server name.
Of course, webserv + TLS server may host multiple virtual hosts and TLS server must know,
which one client wants to connect.
After receiving server name, TLS server will send cert chain for it.
Sending ClientHello (dump without header):
00000000: 01 00 00 59 03 03 6E CE 1E A0 41 D4 7E AB 1D 14 ...Y..n...A.~...
00000010: DF BE F3 9C 7A 58 34 F5 35 EA CC 26 DF BB 0C C6 ....zX4.5..&....
00000020: 1A 89 53 F2 F5 A9 00 00 02 00 02 01 00 00 2E 00 ..S.............
00000030: 00 00 14 00 12 00 00 0F 6E 75 6C 6C 2E 62 61 64 ........null.bad
00000040: 73 73 6C 2E 63 6F 6D 00 23 00 00 00 16 00 00 00 ssl.com.#.......
00000050: 17 00 00 00 0D 00 06 00 04 04 01 02 01 .............
ServerHello
Server responds with:
32 bytes of random data.
Ciphersuite it picked. Last decision is up to server.
If no shared suites are found, connection will be closed.
Extensions it want to use. Last decision again is up to it.
If it sends encrypt_then_mac, it will be used.
Otherwise -- no.
Then server sends its full chain of certificates.
Its format is no different from what is in PEM files on webserver.
At this point, TLS client should (or MUST) check them.
Handshake_type 11
unpack_SSL3_MT_CERTIFICATE
00000000: 0B 00 0F 80 00 0F 7D 00 04 F6 30 82 04 F2 30 82 ......}...0...0.
00000010: 03 DA A0 03 02 01 02 02 12 03 76 EC 3C A6 35 7A ..........v...5z
00000020: 87 BF 46 20 4E DA DF 1F 52 F8 75 30 0D 06 09 2A ..F N...R.u0...*
00000030: 86 48 86 F7 0D 01 01 0B 05 00 30 32 31 0B 30 09 .H........021.0.
...
00000F60: 0D FB E9 EC E3 86 00 DE 9D 10 E3 38 FA A4 7D B1 ...........8..}.
00000F70: D8 E8 49 82 84 06 9B 2B E8 6B 4F 01 0C 38 77 2E ..I....+.kO..8w.
00000F80: F9 DD E7 39 ...9
length 3968
certs_length 3965
cert_length 1270
Got cert , ...)>
Issuer
Warning: mask in wildcard certs is not handled yet: *.badssl.com, but cert is checked anyway
RSA pub key in the (first) cert:
e 65537
n 19212207875868015586500485265412594314671282995330955285385772967743584571414239180595665799566284471068116331464171112513717683861150315002928051619837848326253438754011334279833529199777693013586804259523407030092831110565630039285928700797484546041433042746262835281560141542950412356158605018125080759872915356450280995315411805546117132749581064621275461831043776193468902706318904248372076656649592149715572489093183533612567035844205967056177857678138617694721240465214169316753289374075360578460564810362327067385440248228289475854349178044218846474879628233034019842338955527654364825022370803623697018141699
log_2(n) 2047.2497297743314
cert_length 1306
Got cert , ...)>
Issuer
cert_length 1380
Got cert , ...)>
Issuer
Certs imported from /etc/ssl/certs: 414
Cert checked. Found the issuer in local store.
Cert from serv: , ...)>
Issuer from local store: , ...)>
Server done message
Server indicates it finished.
00000000: 0E 00 00 00 ....
Create (pre)master secret and setup keys
At this moment, client has enough information to setup all encryption keys.
Info to be used: server random (32 bytes), client random, RSA public key (first cert from server).
Client generates random 32 bytes (premaster secret) and encrypts it to server using its RSA public key.
This will be called 'encrypted premaster secret'.
Client has it.
Server can obtain it by RSA decryption, because it has RSA private key (and no one else is supposed to have it).
The PRF (pseudorandom function) is to be used.
(RFC 5246.)
What is this?
To my taste, this is just seeded CPRNG, nothing else.
But unlike cryptographical primitives, it can generate data of specific (maybe even unlimited) length,
by calling hash function again and again.
Why using PRF?
At some point, set of keys is generated for encryption/MACs (so called 'key block').
The 'key block' size may be variable, because different ciphersuites use keys of different sizes.
SSH do this almost like PRF, but they don't call it 'PRF', they just call hash function as long as needed.
At this stage, master secret is generated by PRF with 'seed' including:
premaster secret, the 'master secret' string, 32 random bytes from client, 32 random bytes from server.
If extended_master_secret extension is used, the string is 'extended master secret', and packets seen
between client and server are hashed as a whole, instead of only random values.
This is similar to SSH again.
Master secret is then used to generate the 'key block' of needed size.
Key block is then sliced by 6 parts:
MAC key from client to server
MAC key from server to client
encryption key from client to server
encryption key from server to client
IV from client to server
VI from server to client
These are keys to be used.
But if NULL cipher is used, these keys are not used.
'Client finished' message
This is a message containing a hash generated by PRF 'seeded' with the 'client finished' string and
all packets (seen before) between client and server, hashed.
Plus master key.
This message serves as proof that the client has correctly set up all keys on the server.
This message is the first one (to server) that is MAC'ed and encrypted.
But if NULL cipher is used, it's only MAC'ed.
(MAC key is already generated by client.)
Server sends 'newsession ticket' message
Ignore it so far.
Server sends 'change cipher spec' message
Ignore it so far.
Server sends 'server finished' message
This is the first encrypted and MAC'ed message from server.
Again, it's a proof that server has set up all correctly.
If the NULL cipher is used, this message is only MAC'ed.
It's generated like the 'client finished' message, but slightly differently.
That text string is 'server finished' and hashed packed (seen before) are hashed slightly differently.
A client SHOULD (or even MUST) decrypt this message and check its correctness.
But needless to say that this can be omitted.
Early versions of ToyTLS didn't this, and it worked.
Exchange begins
In TLS lingo, this is called 'application data'.
This may be HTTP traffic.
Or IMAP, SMTP.
Or even old-school IRC (Libera only accepts connections via TLS).
Jabber/XMMP can be used with TLS.
And many more protocols.
MySQL client/server
can use it
as well.
But we simply send 'GET /' HTTP request.
Here is it, in plain text, but MAC'ed:
got buf:
00000000: 48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D HTTP/1.1 200 OK.
00000010: 0A 53 65 72 76 65 72 3A 20 6E 67 69 6E 78 2F 31 .Server: nginx/1
00000020: 2E 31 30 2E 33 20 28 55 62 75 6E 74 75 29 0D 0A .10.3 (Ubuntu)..
...
000002B0: 6F 6D 0A 20 20 3C 2F 68 31 3E 0A 3C 2F 64 69 76 om. </h1>.</div
000002C0: 3E 0A 0A 3C 2F 62 6F 64 79 3E 0A 3C 2F 68 74 6D >..</body>.</htm
000002D0: 6C 3E 0A BF F0 B4 15 06 03 76 DA 15 B5 51 A1 6E l>.......v...Q.n
000002E0: C5 7E 6A 58 F8 0C 31 .~jX..1
chk_MAC_and_decrypt(): start. 1
remove_MAC_and_chk(): MAC:
00000000: BF F0 B4 15 06 03 76 DA 15 B5 51 A1 6E C5 7E 6A ......v...Q.n.~j
00000010: 58 F8 0C 31 X..1
remove_MAC_and_chk(): 723
remove_MAC_and_chk(): calculated (on our side) MAC:
00000000: BF F0 B4 15 06 03 76 DA 15 B5 51 A1 6E C5 7E 6A ......v...Q.n.~j
00000010: 58 F8 0C 31 X..1
remove_MAC_and_chk(): MAC is correct
chk_MAC_and_decrypt(): buf with MAC removed:
00000000: 48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D HTTP/1.1 200 OK.
00000010: 0A 53 65 72 76 65 72 3A 20 6E 67 69 6E 78 2F 31 .Server: nginx/1
00000020: 2E 31 30 2E 33 20 28 55 62 75 6E 74 75 29 0D 0A .10.3 (Ubuntu)..
...
000002B0: 6F 6D 0A 20 20 3C 2F 68 31 3E 0A 3C 2F 64 69 76 om. </h1>.</div
000002C0: 3E 0A 0A 3C 2F 62 6F 64 79 3E 0A 3C 2F 68 74 6D >..</body>.</htm
000002D0: 6C 3E 0A l>.
get_data_from_serv() start
Timeout during receive. Server supposed client should close the connection?
Written to output_null.badssl.com_TLS_RSA_WITH_NULL_SHA.bin 723 bytes
This message is MAC'ed.
And again, client can check MAC or not.
It SHOULD, or even MUST, in order to prevent MITM attack.
But my early ToyTLS didn't it, and it worked.
Also, you see here 64-bit sequence number, in all application packets.
it's incremented after each packet.
Even if no encryption is used, sequence number is used in MAC calculation.
So if you send 'GET /' request twice, MAC will be different for these two packets.
In other words, it's calculated as: MAC(seq_n || plaintext)
A sequence number is also used in SSH.
It's important, even if traffic is encrypted.
For example, you send 'GET /status.php' to your server from time to time.
To get some info, maybe.
Once a minute.
Or if you execute some Unix command via SSH from time to time, each time the same one.
Even if an attacker can't decrypt these messages, that fact can give him some hints,
that you send the same message in specific timestamps, between specific intervals.
This is unacceptable in cryptography (even of WW2 times), hence packets must be randomized.
'Close notify' alert
Polite client should send (but not required to) the 'close notify' message,
which is also MAC'ed:
encrypt_MAC_send(). seq_n 2 _type 21
buf to send:
00000000: 01 00 ..
encrypt_MAC_send() verify_data:
00000000: 0A C5 DC D1 24 22 7C 1B 49 46 7B C9 A9 54 0E 7E ....$"|.IF{..T.~
00000010: B6 46 3A 6E .F:n
Server also sends it to client.
A word about Wireshark
Wireshark groups packets into bundles:
But in fact, there are 3 TCP/IP or TLS packets in each bundle.
This can be confusing, because 'Finished' message (from both server and client) is MAC'ed
and encrypted, and that MAC 'signature' is calculated only against the 'Finished' packet,
not against 'bundle'.
It's NULL cipher, but...
This is fun thing to know -- an attacker can read all the traffic, since NULL ciphersuite is used, yes.
But he can't modify any packet, because all they are 'signed' or MAC'ed.
In order to generate MAC key, he must generate master secret, and this problem leads to a problem of cracking RSA.
Also, MITM attack is not possible, if ToyTLS checks server's cert.
In case of null.badssl.com, it do so, and cert is issued by Let's Encrypt, as you see.
So even with NULL cipher, there is some level of protection in TLS.
Not bad, huh?
Turning on AES encryption
As of TLS 1.2, mandatory ciphersuite is TLS_RSA_WITH_AES_128_CBC_SHA
(RFC 5246).
But as we will see later, not all high-profile websites support it.
MTE: MAC-then-Encrypt
This is: encrypt(plaintext || MAC(plaintext)).
In our case: MAC(seq_n || plaintext) is added to plaintext.
Then it's padded.
And the padding is very much alike PKCS7, but slightly different.
The resulting buffer is encrypted by AES.
Header for MAC is sequence number + 6 bytes which will be a header for TLS packet.
Plus plaintext.
MAC is then added to plaintext.
The resulting buffer is then padded at the end.
You see series of 0x06 bytes.
The last 0x06 means that there are more 6 bytes to be removed.
But the total length of padding is 7.
This is slightly different from PKCS7.
Also, according to
RFC 5246
I add 16-byte zero header, which is supposed to be IV.
Ignore it yet.
You see, plaintext is padded (so that it can be encrypted with AES), zero IV is prepended.
Then it is encrypted and MAC(encrypted) is sent along with encrypted buffer.
Certificates
Certificate is an (RSA/EC) public key + information about owner (CN, at least domain) + signature of issuer.
Certificate can be issued (or signed) by CA like Let's Encrypt.
privkey.pem is the private RSA key, not supposed to leave your webserver.
The 'fullchain.pem' file contains server's cert + LE's certs.
Each consecutive cert is an issuer of previous cert.
Or, in other words, each cert is signed/issued by consecutive cert.
This strict order must be checked by client, including ToyTLS.
The last (root) cert in chain is often present in your local store.
It is often self-signed, as those (root certs) in local store.
OpenSSL can show this chain.
There are three in my case:
% openssl s_client -connect cryptography.re:443
CONNECTED(00000003)
---
Certificate chain
0 s:CN = cryptography.re
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
's' is information about cert.
'i' is information about issuer. Issuer, who signed that cert.
ToyTLS enumerates all certs in chain and tries to verify each one using each cert
from /etc/ssl/certs (as in Ubuntu):
for cert in chain received from webserver:
for CA_cert in local_store:
if cert is signed by CA_cert:
return cert verified, webserver is secure
return cert cannot be verified
Server's certificate issued by CA (like Let's Encrypt) is simply signed with RSA key.
This is how it checked in ToyTLS:
By default, ToyTLS tries all certs from /etc/ssl/certs (as in Ubuntu).
But you can point it to the 'local_cert_store_from_Ubuntu_22' directory.
These files I just copied from Ubuntu.
So it's possible to run ToyTLS on any exotic OS (with Python interpreter) without local store.
My ToyTLS illustrates an important thing -- certificate checking may be removed and it will work.
It will make HTTP(s) requests, download HTML pages.
In fact, early versions of ToyTLS didn't check it.
Just because I decided to add this later.
And this is important.
TLS client may not check certificate.
Or it may check it incorrectly, partially/incompletely.
Certificate checker functions may have bugs.
And this may go unnoticed for a long time.
This allows MITM attack.
There is a huge gap between a code that 'just works' and a code that 'very hard/impossible to crack remotely'.
Early version of my ToyTLS 'just worked'.
Tools
Several important tools I found useful.
badssl.com website(s) is/are very useful for TLS client testing.
As well-known service: https://www.ssllabs.com/ssltest
can test a TLS server and show ciphersuites supported +
extensions + many more information.
But there are also very useful command-line tools for that.
sslscan,
testssl.sh.
By the way.
ssllabs.com service and testssl.sh utility can simulate OS/browser of old version.
For example, testssl.sh:
Running client simulations (HTTP) via sockets
Android 6.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 7.0 (native) TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305, 256 bit ECDH (P-256)
Android 8.1 (native) TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305, 253 bit ECDH (X25519)
Android 9.0 (native) TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
Android 10.0 (native) TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
...
IE 6 XP No connection
IE 8 Win 7 No connection
IE 8 XP No connection
IE 11 Win 7 TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
IE 11 Win 8.1 TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
IE 11 Win Phone 8.1 TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
IE 11 Win 10 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
How they do this?
Do they run a huge VM farm with all that ancient OS-es and browsers?
No.
They just send prerecorded ClientHello packet to your TLS server, the same, as it was once generated by that ancient browser.
Then they get your server's reply and see, what server agreed on or not.
Which ciphersuite your server picked or not.
But many high-profile website don't support neither TLS_RSA_WITH_AES_128_CBC_SHA or TLS_RSA_WITH_AES_256_CBC_SHA256.
For example, microsoft.com (as sslscan says, as of October 2023):
Supported Server Cipher(s):
Preferred TLSv1.3 128 bits TLS_AES_128_GCM_SHA256 Curve 25519 DHE 253
Accepted TLSv1.3 256 bits TLS_AES_256_GCM_SHA384 Curve 25519 DHE 253
Preferred TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve 25519 DHE 253
Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve 25519 DHE 253
Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-SHA384 Curve 25519 DHE 253
Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-SHA256 Curve 25519 DHE 253
They removed ciphersuites with older RSA key exchange method in favour of newer ECDHE.
Other high-profile websites that switched to ECDHE ciphersuites for TLS 1.2 (as of October 2023):
nsa.gov,
twitter.com,
www.instagram.com,
en.wikipedia.org.
Stay tuned.
I should add support of it to my next version(s) of ToyTLS.
And also TLS 1.3, which is slightly different protocol.
Yes, I know about these lousy Disqus ads.
Please use adblocker.
I would consider to subscribe to 'pro' version of Disqus if the signal/noise ratio in comments would be good enough.
