Learn reverse engineering: but where to start?

I've heard many times that x86 assembly is hard to learn.

One reason is that x86 arch (or visible API) is baroque. Linus Torvalds used this term as well: baroque instruction encoding

Also: It is well-known that the x86 instruction set is baroque, overcomplicated, and redundantly redundant.

Corresponding entries from The Jargon File: baroque, rococo.

16-bit 8086 CPU was toyish, not taken very seriously at the time of introduction (1976). No one could imagine this arch will be number one one day. It was extension already, of 8-bit 8080 CPU -- this is why 8086 has 8-bit parts of registers, like AL and AH, etc. Since then, it was extended at least twice -- to 32-bit arch, then to 64-bit arch.

Now you have RAX register that has 32-bit part (EAX). In turn, EAX register has 16-bit part (AX). Which, in turn, has 8-bit parts (AH, AL).

Many other x86 parts also reminds matryoshka/russian/nested doll.

And all this may be very confusing.

(I myself started learning 16-bit 8086 in ~1993-1994, ignoring 32-bit 80386 arch. It was easier at the time.)

When CPU is developed from scratch, no one would reintroduce such rudiments. RISC CPUs never had anything like that.

So a better idea is to start learning RISC arch, ignoring x86 for a moment.

Bottom line (TL;DR): I would recommend starting at ARM64. 32-bit ARM is complicated with thumb modes. MIPS is complicated with delay slots. So you have ARM64 left. ARM64 devices are popular these days, pick any.

(the post first published at 20221109.)


List of my other blog posts.

Yes, I know about these lousy Disqus ads. Please use adblocker. I would consider to subscribe to 'pro' version of Disqus if the signal/noise ratio in comments would be good enough.