RC4 (Rivest Cipher 4) - How it operates & how to recognise it

RC4 (Rivest Cipher 4)

This is a very commonly used Stream Cipher (this transforms a plaintext string byte-by-byte, they are linear meaning they are a symmetric type of encryption).

Typically has 3 stages:

1. Key Scheduling Algorithm (KSA) -> Creates a list of scrambled values

   • Creates a list of values form 0 to 256
   • Each value is swapped with another based on a calculation (essentially scrambles the list of values)

2. Pseudo Random Generation Algorithm (PRGA) -> Creates the Keystream

   • Generates and outputs the Keystream using the list
   • It will generate however many bytes it needs (up to 256)
   • The value ‘256’ or ‘0x100’ is often an easy way to work out that the encryption may be RC4 in assembly.

3. XOR Operation -> Encrypts plaintext string using the Keystream

   • It XOR’s each byte of the Plaintext with a byte from the Keystream

Recognising RC4

When looking through assembly you should be able to spot the following which may give you an indication that it is using RC4:

• The use of 256 and 0x100 constants in assembly
• Recognising the 3 stages of RC4