Serpent Cipher Explained: How It Works, With Code in C, C++, Java, Python and C#

Serpent encrypts every block with 32 rounds. The best published key-recovery attack reaches 12 of them, and needs more known plaintext than anyone could collect. That margin was the point of the design: Serpent was built to be conservative, and it finished second in the vote that chose the Advanced Encryption Standard, 59 votes to Rijndael’s 86.

This guide explains how Serpent works, from its eight 4-bit S-boxes and bitslice round structure to its key schedule, and implements it in C, C++, Java, Python and C#. It also covers the byte-order trap that makes two correct Serpent implementations disagree, four mistakes that a quick test will not catch, and how fast a readable implementation really is. Every program was compiled and run for this article on Ubuntu 24.04 (x86-64): C with GCC 13.3 and Clang 18.1.3 under -std=c11 and -std=c17, C++ with g++ 13.3 and clang++ 18.1.3 under -std=c++17 and -std=c++20, all with -Wall -Wextra -pedantic -Werror; Java on OpenJDK 21 with --release 17 -Xlint:all -Werror; Python 3.13 with ruff and mypy --strict clean; and C# on the .NET 10 SDK with analyzer warnings as errors. All five pass the same 1,043 published test vectors and print byte-identical output, the C and C++ tests run clean under AddressSanitizer and UndefinedBehaviorSanitizer, and every output block is captured verbatim. The code is in five GitHub repositories whose workflows repeat these checks on each commit, including MSVC, Windows and macOS jobs that were not run on the test machine.

What Is the Serpent Cipher?

Serpent is a symmetric block cipher that encrypts 128-bit blocks with a key of up to 256 bits. It is a substitution–permutation network of 32 rounds; each round mixes in a 128-bit subkey, passes the block through one of eight 4-bit S-boxes, and stirs the result with a linear transformation. Serpent was a finalist for the Advanced Encryption Standard and is in the public domain.

Ross Anderson, Eli Biham and Lars Knudsen designed Serpent and submitted it to the AES competition in 1998. It was one of the five finalists, with MARS, RC6, Rijndael and Twofish. At the last AES candidate conference Rijndael received 86 votes and Serpent 59, as the Serpent home page records, and NIST announced Rijndael as the winner on October 2, 2000. The designers placed Serpent in the public domain, so anyone may use it without a licence. It is still offered today, for example as one of the volume ciphers in VeraCrypt. For a classical cipher to compare it with, see the Vigenère cipher in C++, which a computer can break in seconds.

How Serpent Works

Serpent treats the 16-byte block as four 32-bit words, x0 to x3. A round does three things:

1. XOR the round’s subkey into the four words.
2. Substitute. Take bit j of each of the four words; together they form a 4-bit number. Replace it with its entry in the round’s S-box, and write the four output bits back to position j. Do this for all 32 bit positions. Round r uses S-box r mod 8.
3. Mix the four words with the linear transformation: fixed rotations, shifts and XORs that spread every output bit of one round across many S-boxes of the next.

Rounds 0 to 30 follow that pattern. Round 31 replaces the linear transformation with an XOR of the last subkey, K32, so the cipher both starts and ends with key material.

Arranging the block this way is called bitslicing. The 32 S-box lookups of a round all read bit j from the same four words, so an implementation can compute all 32 of them at once with a short sequence of AND, OR, XOR and NOT instructions on whole words. Serpent was designed for exactly that. The specification also describes the cipher with an initial and a final bit permutation around a conventional layout; that description gives the same ciphertext, and the bitslice form is the one implementations use.

The code below favours clarity over speed. Each S-box is a 16-entry table, and the substitution loops over the 32 bit positions, so you can see the bitslice structure directly. “How Fast Is It?” further down measures what that costs.

The Eight S-Boxes

/* The eight 4-bit S-boxes from the Serpent specification. */
static const uint8_t SBOX[8][16] = {
    { 3, 8, 15, 1, 10, 6, 5, 11, 14, 13, 4, 2, 7, 0, 9, 12 },
    { 15, 12, 2, 7, 9, 0, 5, 10, 1, 11, 14, 8, 6, 13, 3, 4 },
    { 8, 6, 7, 9, 3, 12, 10, 15, 13, 1, 14, 4, 0, 11, 5, 2 },
    { 0, 15, 11, 8, 12, 9, 6, 3, 13, 1, 2, 4, 10, 7, 5, 14 },
    { 1, 15, 8, 3, 12, 0, 11, 6, 2, 5, 4, 10, 9, 14, 7, 13 },
    { 15, 5, 2, 11, 4, 10, 9, 12, 0, 3, 14, 8, 13, 6, 7, 1 },
    { 7, 2, 12, 5, 8, 4, 6, 11, 14, 9, 1, 15, 13, 3, 10, 0 },
    { 1, 13, 15, 0, 14, 8, 2, 11, 7, 4, 12, 10, 9, 3, 5, 6 },
};

Each row is a permutation of 0 to 15, so every S-box can be inverted, and decryption uses the eight inverse tables. The C file lists them too; the C++ version computes them from these tables at compile time, and the Java, Python and C# versions at start-up, which leaves one less table to mistype.

The Bitslice Substitution

/* Bitslice S-box: bit j of x[0..3] forms a 4-bit input, x[0] the low bit.
 * All 32 bit positions go through the same 4-bit table. */
static void apply_sbox(const uint8_t box[16], uint32_t x[4])
{
    uint32_t y[4] = { 0, 0, 0, 0 };
    for (unsigned j = 0; j < 32; ++j) {
        unsigned in = ((x[0] >> j) & 1u) | ((x[1] >> j) & 1u) << 1 |
                      ((x[2] >> j) & 1u) << 2 | ((x[3] >> j) & 1u) << 3;
        unsigned out = box[in];
        for (unsigned b = 0; b < 4; ++b)
            y[b] |= (uint32_t)((out >> b) & 1u) << j;
    }
    memcpy(x, y, sizeof y);
}

The input nibble takes its low bit from x[0] and its high bit from x[3]. Getting that order backwards still produces a working cipher that decrypts its own output, which is why the test vectors matter (see “Four Mistakes That Break a Serpent Implementation”).

The Linear Transformation

/* The linear transformation that mixes the four words after each S-box. */
static void linear_transform(uint32_t x[4])
{
    x[0] = rotl(x[0], 13);
    x[2] = rotl(x[2], 3);
    x[1] ^= x[0] ^ x[2];
    x[3] ^= x[2] ^ (x[0] << 3);
    x[1] = rotl(x[1], 1);
    x[3] = rotl(x[3], 7);
    x[0] ^= x[1] ^ x[3];
    x[2] ^= x[3] ^ (x[1] << 7);
    x[0] = rotl(x[0], 5);
    x[2] = rotl(x[2], 22);
}

The rotations by 13, 3, 1, 7, 5 and 22 bits move each word’s bits to new positions, and the two plain shifts (<< 3 and << 7) are deliberately not rotations: they drop the bits that fall off the top. Decryption runs the same ten steps backwards, with right rotations in place of left ones. The two lines with shifts need no inverse operation of their own, because each XORs in a value computed from words that step leaves unchanged.

The Key Schedule

int serpent_set_key(serpent_key *ks, const uint8_t *key, size_t key_len)
{
    if (key_len == 0 || key_len > SERPENT_MAX_KEY_SIZE)
        return -1;

    /* Pad the key to 256 bits: one 1 bit after the key, then zeros. */
    uint8_t padded[SERPENT_MAX_KEY_SIZE] = { 0 };
    memcpy(padded, key, key_len);
    if (key_len < SERPENT_MAX_KEY_SIZE)
        padded[key_len] = 0x01;

    /* w[0..7] hold the padded key; w[8..139] are the 132 prekey words. */
    uint32_t w[140];
    for (unsigned i = 0; i < 8; ++i)
        w[i] = load_le32(padded + 4 * i);
    for (unsigned i = 8; i < 140; ++i)
        w[i] = rotl(w[i - 8] ^ w[i - 5] ^ w[i - 3] ^ w[i - 1] ^ PHI ^ (uint32_t)(i - 8), 11);

    /* Subkey i passes prekeys 4i..4i+3 through S-box (3 - i) mod 8. */
    for (unsigned i = 0; i < 33; ++i) {
        uint32_t *k = ks->subkey[i];
        memcpy(k, &w[8 + 4 * i], 4 * sizeof(uint32_t));
        apply_sbox(SBOX[(3 - i) & 7], k);
    }

    memset(padded, 0, sizeof padded);
    memset(w, 0, sizeof w);
    return 0;
}

Every key is first padded to 256 bits: a single 1 bit straight after the key, then zeros. The eight padded words then grow into 132 prekeys with the recurrence wᵢ = (wᵢ₋₈ ⊕ wᵢ₋₅ ⊕ wᵢ₋₃ ⊕ wᵢ₋₁ ⊕ φ ⊕ i) ⋘ 11, where φ = 0x9e3779b9 is the fractional part of the golden ratio and i counts the prekeys from 0. Groups of four prekeys pass through the S-boxes, in the order S3, S2, S1, S0, S7, S6 and so on, to give the 33 subkeys. The expanded key, 528 bytes, is computed once and reused for every block.

Encrypting and Decrypting a Block

void serpent_encrypt(const serpent_key *ks, const uint8_t in[SERPENT_BLOCK_SIZE],
                     uint8_t out[SERPENT_BLOCK_SIZE])
{
    uint32_t x[4];
    for (unsigned i = 0; i < 4; ++i)
        x[i] = load_le32(in + 4 * i);

    for (unsigned r = 0; r < 32; ++r) {
        xor_key(x, ks->subkey[r]);
        apply_sbox(SBOX[r & 7], x);
        if (r < 31)
            linear_transform(x);
        else
            xor_key(x, ks->subkey[32]);   /* the last round replaces LT with K32 */
    }

    for (unsigned i = 0; i < 4; ++i)
        store_le32(out + 4 * i, x[i]);
}

void serpent_decrypt(const serpent_key *ks, const uint8_t in[SERPENT_BLOCK_SIZE],
                     uint8_t out[SERPENT_BLOCK_SIZE])
{
    uint32_t x[4];
    for (unsigned i = 0; i < 4; ++i)
        x[i] = load_le32(in + 4 * i);

    for (int r = 31; r >= 0; --r) {
        if (r == 31)
            xor_key(x, ks->subkey[32]);
        else
            inverse_linear_transform(x);
        apply_sbox(SBOX_INV[r & 7], x);
        xor_key(x, ks->subkey[r]);
    }

    for (unsigned i = 0; i < 4; ++i)
        store_le32(out + 4 * i, x[i]);
}

Words are read little-endian: byte 0 of the block is the lowest byte of x0. This is the byte order of the NESSIE test vectors, which libgcrypt and Botan follow. A second, equally valid convention exists; the byte-order section below explains it.

A Complete Example in C

/* serpent_demo.c - encrypts and decrypts one block with a 256-bit key, then
 * checks two published test vectors: one in NESSIE byte order, and one from
 * the Serpent AES submission, which writes every value byte-reversed.
 * Build: gcc -std=c11 -Wall -Wextra -pedantic serpent_demo.c serpent.c -o serpent_demo
 */
#include <stdio.h>
#include <string.h>

#include "serpent.h"

static void print_hex(const char *label, const uint8_t *p, size_t n)
{
    printf("%-11s", label);
    for (size_t i = 0; i < n; ++i)
        printf("%02x", p[i]);
    printf("\n");
}

static void reverse(uint8_t *p, size_t n)
{
    for (size_t i = 0; i < n / 2; ++i) {
        uint8_t t = p[i];
        p[i] = p[n - 1 - i];
        p[n - 1 - i] = t;
    }
}

int main(void)
{
    uint8_t key[32];
    for (size_t i = 0; i < sizeof key; ++i)
        key[i] = (uint8_t)i;                         /* 00 01 02 ... 1f */
    static const char text[] = "Serpent: 128-bit";   /* 16 characters + NUL */
    uint8_t plain[16];
    memcpy(plain, text, sizeof plain);                /* the block is the 16 characters */

    serpent_key ks;
    if (serpent_set_key(&ks, key, sizeof key) != 0)
        return 1;
    uint8_t cipher[16], back[16];
    serpent_encrypt(&ks, plain, cipher);
    serpent_decrypt(&ks, cipher, back);
    serpent_wipe_key(&ks);

    print_hex("key", key, sizeof key);
    print_hex("plaintext", plain, sizeof plain);
    print_hex("ciphertext", cipher, sizeof cipher);
    printf("decrypts back: %s\n", memcmp(back, plain, 16) == 0 ? "yes" : "no");

    /* NESSIE: 128-bit key 80 00 .. 00, zero block -> 264e5481... */
    uint8_t k128[16] = { 0x80 }, zero[16] = { 0 }, out[16];
    const uint8_t nessie[16] = { 0x26, 0x4e, 0x54, 0x81, 0xef, 0xf4, 0x2a, 0x46,
                                 0x06, 0xab, 0xda, 0x06, 0xc0, 0xbf, 0xda, 0x3d };
    serpent_set_key(&ks, k128, sizeof k128);
    serpent_encrypt(&ks, zero, out);
    printf("NESSIE vector:          %s\n", memcmp(out, nessie, 16) == 0 ? "match" : "MISMATCH");

    /* AES submission: 256-bit zero key, zero block -> 89104945... once the
     * output bytes are reversed (the key and block are all zero either way). */
    const uint8_t submission[16] = { 0x89, 0x10, 0x49, 0x45, 0x04, 0x18, 0x19, 0x50,
                                     0xf9, 0x8d, 0xd9, 0x98, 0xa8, 0x2b, 0x67, 0x49 };
    uint8_t k256[32] = { 0 };
    serpent_set_key(&ks, k256, sizeof k256);
    serpent_encrypt(&ks, zero, out);
    reverse(out, sizeof out);
    printf("AES submission vector:  %s\n", memcmp(out, submission, 16) == 0 ? "match" : "MISMATCH");
    serpent_wipe_key(&ks);
    return 0;
}

Output:

key        000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f
plaintext  53657270656e743a203132382d626974
ciphertext 8f9d6a59c08901a421126fcffc13c5f6
decrypts back: yes
NESSIE vector:          match
AES submission vector:  match

The ciphertext line was also produced, independently, by libgcrypt’s Serpent with the same key and block. The two check lines are known-answer tests from two published sources, and the next section explains why the second needs reverse().

Byte Order: Why Two Correct Serpent Implementations Disagree

Serpent’s AES submission wrote keys, blocks and ciphertexts as hexadecimal numbers with the most significant digit first. The NESSIE project, which published the test vectors most libraries now use, wrote them as byte strings in memory order. The cipher is the same; the bytes are listed in opposite orders. For a 256-bit all-zero key and an all-zero block:

Each string is the other one backwards, byte by byte. To convert, reverse all 16 bytes of the plaintext, reverse the key bytes, encrypt, and reverse the 16 output bytes. Bouncy Castle ships the submission-order variant as a separate cipher named TnepresEngine, “Serpent” spelled backwards, and its 13 test vectors all matched the code on this page after that reversal.

If your Serpent output looks like a scrambled version of a published vector, check the byte order before you check anything else. And when two systems must exchange Serpent ciphertext, agree on the convention explicitly; the cipher’s name alone does not settle it.

Serpent in C++, Java, Python and C#

Each of the four implementations below follows the C code step for step, prints exactly the output above, and passes the same 1,043 test vectors. These excerpts show what each language changes; the complete files are in the repositories linked in “Source Code and Tests”.

Serpent in C++

class Serpent {
public:
    static constexpr std::size_t block_size = 16;
    static constexpr std::size_t max_key_size = 32;
    using Block = std::array<std::uint8_t, block_size>;

    // Expands a key of 1 to 32 bytes; throws std::invalid_argument otherwise.
    Serpent(const std::uint8_t* key, std::size_t key_len);

    // Any contiguous byte container: std::array, std::vector, std::string_view...
    template <class Bytes>
    explicit Serpent(const Bytes& key)
        : Serpent(reinterpret_cast<const std::uint8_t*>(std::data(key)), std::size(key)) {}

    ~Serpent();                                  // wipes the subkeys
    Serpent(const Serpent&) = default;
    Serpent& operator=(const Serpent&) = default;

    [[nodiscard]] Block encrypt(const Block& in) const;
    [[nodiscard]] Block decrypt(const Block& in) const;

private:
    using Words = std::array<std::uint32_t, 4>;
    std::array<Words, 33> subkeys_{};            // K0..K32
};

The C++ version is a header-only class in namespace mycplus. Block is a std::array<std::uint8_t, 16>, so a block cannot be the wrong length, and the templated constructor accepts any contiguous byte container. encrypt and decrypt are [[nodiscard]], because discarding a ciphertext is always a bug, and the destructor overwrites the subkeys. A key of the wrong length throws std::invalid_argument. The inverse S-boxes are built by the compiler:

struct InverseSboxes {
    std::uint8_t box[8][16]{};
    constexpr InverseSboxes() {
        for (int s = 0; s < 8; ++s)
            for (int i = 0; i < 16; ++i)
                box[s][sbox[s][i]] = static_cast<std::uint8_t>(i);
    }
};
inline constexpr InverseSboxes sbox_inv{};

Usage reads naturally:

    const Serpent cipher(key);
    const Serpent::Block encrypted = cipher.encrypt(plain);
    const Serpent::Block back = cipher.decrypt(encrypted);

Serpent in Java

    /** Expands a key of 1 to 32 bytes; shorter keys are padded as the specification requires. */
    public Serpent(byte[] key) {
        if (key == null || key.length == 0 || key.length > MAX_KEY_SIZE) {
            throw new IllegalArgumentException("Serpent keys are 1 to 32 bytes");
        }
        // Pad to 256 bits: one 1 bit after the key, then zeros.
        byte[] padded = Arrays.copyOf(key, MAX_KEY_SIZE);
        if (key.length < MAX_KEY_SIZE) {
            padded[key.length] = 1;
        }
        // w[0..7] hold the padded key; w[8..139] are the 132 prekey words.
        int[] w = new int[140];
        for (int i = 0; i < 8; i++) {
            w[i] = loadLe32(padded, 4 * i);
        }
        for (int i = 8; i < 140; i++) {
            w[i] = Integer.rotateLeft(w[i - 8] ^ w[i - 5] ^ w[i - 3] ^ w[i - 1] ^ PHI ^ (i - 8), 11);
        }
        // Subkey i passes prekeys 4i..4i+3 through S-box (3 - i) mod 8.
        for (int i = 0; i < 33; i++) {
            System.arraycopy(w, 8 + 4 * i, subkeys[i], 0, 4);
            applySbox(SBOX[Math.floorMod(3 - i, 8)], subkeys[i]);
        }
        Arrays.fill(padded, (byte) 0);
        Arrays.fill(w, 0);
    }

Java has no unsigned 32-bit type, but Serpent needs only XOR, shifts and rotations, which give the same bits on a signed int. Two details elsewhere in the file matter: the S-box loop uses >>>, the unsigned shift, and the byte loader masks each byte with & 0xff, because Java’s byte is signed. Integer.rotateLeft and Integer.rotateRight supply the rotations, and Math.floorMod(3 - i, 8) keeps the S-box index non-negative. The class’s wipe() method clears the subkeys, although a garbage collector that moves objects may leave older copies in memory, which Java gives no way to clear.

Serpent in Python

class Serpent:
    """Serpent with a key of 1 to 32 bytes; encrypts and decrypts 16-byte blocks."""

    def __init__(self, key: bytes) -> None:
        if not 1 <= len(key) <= MAX_KEY_SIZE:
            raise ValueError("Serpent keys are 1 to 32 bytes")
        # Pad to 256 bits: one 1 bit after the key, then zeros.
        padded = bytes(key) + (b"\x01" if len(key) < MAX_KEY_SIZE else b"")
        padded = padded.ljust(MAX_KEY_SIZE, b"\x00")
        # w[0..7] hold the padded key; w[8..139] are the 132 prekey words.
        w = [int.from_bytes(padded[i : i + 4], "little") for i in range(0, 32, 4)]
        for i in range(8, 140):
            w.append(_rotl(w[i - 8] ^ w[i - 5] ^ w[i - 3] ^ w[i - 1] ^ PHI ^ (i - 8), 11))
        # Subkey i passes prekeys 4i..4i+3 through S-box (3 - i) mod 8.
        self._subkeys: list[Words] = [
            _apply_sbox(SBOX[(3 - i) % 8], w[8 + 4 * i : 12 + 4 * i]) for i in range(33)
        ]

    def encrypt(self, block: bytes) -> bytes:
        x = _load(block)
        for r in range(32):
            x = [a ^ k for a, k in zip(x, self._subkeys[r])]
            x = _apply_sbox(SBOX[r % 8], x)
            if r < 31:
                x = _linear_transform(x)
            else:  # the last round replaces LT with K32
                x = [a ^ k for a, k in zip(x, self._subkeys[32])]
        return _store(x)

    def decrypt(self, block: bytes) -> bytes:
        x = _load(block)
        for r in range(31, -1, -1):
            if r == 31:
                x = [a ^ k for a, k in zip(x, self._subkeys[32])]
            else:
                x = _inverse_linear_transform(x)
            x = _apply_sbox(SBOX_INV[r % 8], x)
            x = [a ^ k for a, k in zip(x, self._subkeys[r])]
        return _store(x)

Python integers never overflow, so the rotation and linear-transformation helpers in the same file mask every result with & 0xFFFFFFFF to stay within 32 bits; forgetting one mask lets a value grow past 32 bits and silently changes the output. int.from_bytes(..., "little") and to_bytes(4, "little") handle the byte order, and (3 - i) % 8 is already non-negative in Python. This version needs no third-party packages, and it is more than 200 times slower than the compiled ones.

Serpent in C#

    /// <summary>Expands a key of 1 to 32 bytes; shorter keys are padded as the specification requires.</summary>
    public Serpent(ReadOnlySpan<byte> key)
    {
        if (key.Length is 0 or > MaxKeySize)
            throw new ArgumentException("Serpent keys are 1 to 32 bytes.", nameof(key));

        // Pad to 256 bits: one 1 bit after the key, then zeros.
        Span<byte> padded = stackalloc byte[MaxKeySize];
        padded.Clear();
        key.CopyTo(padded);
        if (key.Length < MaxKeySize)
            padded[key.Length] = 1;

        // w[0..7] hold the padded key; w[8..139] are the 132 prekey words.
        Span<uint> w = stackalloc uint[140];
        for (int i = 0; i < 8; i++)
            w[i] = BinaryPrimitives.ReadUInt32LittleEndian(padded[(4 * i)..]);
        for (int i = 8; i < 140; i++)
            w[i] = BitOperations.RotateLeft(w[i - 8] ^ w[i - 5] ^ w[i - 3] ^ w[i - 1] ^ Phi ^ (uint)(i - 8), 11);

        // Subkey i passes prekeys 4i..4i+3 through S-box (3 - i) mod 8.
        for (int i = 0; i < 33; i++)
        {
            Span<uint> k = _subkeys.AsSpan(4 * i, 4);
            w.Slice(8 + 4 * i, 4).CopyTo(k);
            ApplySbox(Sbox[(35 - i) % 8], k);        // 35 = 3 mod 8, never negative
        }
        CryptographicOperations.ZeroMemory(padded);
        CryptographicOperations.ZeroMemory(System.Runtime.InteropServices.MemoryMarshal.AsBytes(w));
    }

    /// <summary>Encrypts one 16-byte block from <paramref name="input"/> into <paramref name="output"/>.</summary>
    public void Encrypt(ReadOnlySpan<byte> input, Span<byte> output)
    {
        Span<uint> x = stackalloc uint[4];
        Load(input, x);
        for (int r = 0; r < 32; r++)
        {
            XorKey(x, r);
            ApplySbox(Sbox[r % 8], x);
            if (r < 31)
                LinearTransform(x);
            else
                XorKey(x, 32);                   // the last round replaces LT with K32
        }
        Store(x, output);
    }

The C# class works on spans, so a caller can encrypt in place inside a larger buffer without allocating. BinaryPrimitives.ReadUInt32LittleEndian fixes the byte order whatever the machine, BitOperations.RotateLeft becomes a hardware rotate instruction where the processor has one, the padded key and prekeys live on the stack with stackalloc, and CryptographicOperations.ZeroMemory clears them in a way the JIT will not optimize away. Serpent implements IDisposable, and Dispose clears the subkeys.

Four Mistakes That Break a Serpent Implementation

A broken cipher still produces output that looks random, and that is what makes cipher bugs dangerous. Each mistake below was checked against the 1,043 test vectors, using small programs in the C repository’s pitfalls directory or a deliberately altered copy of the code.

1. Testing Only That Decryption Undoes Encryption

The most common test of a cipher is the round trip: encrypt, decrypt, compare. It cannot detect a bug that encryption and decryption share. round_trip_only.c flips one bit of subkey K5, as a key-schedule bug would, and runs all the vectors:

round trip succeeds:    1043 of 1043
matches the test vector: 0 of 1043

Every block decrypts perfectly, and not one ciphertext is Serpent. The same happened on a copy of the C code with a wrong golden-ratio constant, a wrong prekey index and subkeys taken through the wrong S-boxes: all 1,043 round trips succeeded and no vector matched. A wrong S-box entry is subtler. In C, which lists the inverse tables separately, it broke every round trip. In Python, which computes the inverses from the forward tables, the same typo decrypted all 1,043 blocks correctly and matched no vector. Deriving the inverse removes one table to mistype, and with it one way to notice the mistake. Known-answer vectors from an independent source are the only test that proves you implemented the cipher rather than a cipher.

2. Reading the Words in the Wrong Byte Order

Loading each 32-bit word big-endian, the way a cast through ntohl() or a port to a big-endian machine would, gives a different and still reversible cipher. word_byte_order.c simulates it by swapping the bytes of each word around the correct code:

big-endian words match the test vectors: 0 of 1043

Write explicit little-endian loads and stores, as all five implementations do, rather than copying bytes into a uint32_t with memcpy and hoping the machine is little-endian.

3. Testing Only 256-Bit Keys

Padding is applied only to keys shorter than 256 bits. When the padding bit was removed from a copy of the C code, the test suite reported:

1043 vectors: 0 with 128-bit keys, 0 with 192-bit, 458 with 256-bit; 585 failed

All 458 vectors with 256-bit keys still passed. A suite that tests only the longest key would have passed this bug, and with it every 128-bit key the program ever expanded would have been wrong. Setting the padding bit to 0x80 instead of 0x01, the other plausible slip, gave the same counts. Test every key size you accept.

4. Encrypting a Message One Block at a Time

A block cipher encrypts exactly 16 bytes. Applying it to each 16-byte block of a longer message independently is called ECB mode, and it leaks the message’s structure: equal plaintext blocks give equal ciphertext blocks. ecb_patterns.c encrypts a 48-byte message:

"PAY 100 DOLLARS " -> ed4988955ee313c89e8f9bda7616de2c
"PAY 100 DOLLARS " -> ed4988955ee313c89e8f9bda7616de2c
"PAY 999 DOLLARS " -> a3eb5fbd2bfad2e89e8f74b1a89bfd76

Without the key, an observer still learns that the first two blocks are identical and the third is different. Real systems wrap a block cipher in a mode such as CTR or XTS, and add authentication so tampering is detected; the safest choice is an authenticated mode from a maintained library, covered below.

How Fast Is It?

How This Was Measured

All figures come from one machine in one session. No CPU pinning or frequency-scaling control was applied, and no percentiles were captured. The benchmark measures single-threaded encryption of one block after another and says nothing about decryption, key setup, or a cipher mode.

The four compiled implementations land within 20% of each other, so the language matters far less than the algorithm’s form. Python is more than 200 times slower than they are, at roughly 19 KB per second. libgcrypt is about 20 times faster than the best of them, for a reason visible in its source: it computes each S-box as a fixed sequence of logic operations on whole 32-bit words, so one sequence handles all 32 bit positions at once, instead of the table lookup per bit position that the code on this page performs. That is bitslicing used for speed; the code here uses the bitslice layout for clarity. libgcrypt also has SSE2, AVX2 and ARM NEON versions that encrypt several blocks in parallel.

Is Serpent Still Secure?

As a cipher, yes. The best published key-recovery attacks reach 11 and 12 of Serpent’s 32 rounds. The 11-round attack by Nguyen, Wu and Wang (ACISP 2011) needs about 2¹¹⁶ known plaintexts, and the 12-round attack in the same paper needs more time still. No attack on the full 32 rounds comes close to being practical.

The code on this page is a different matter, and it is written for learning, not for protecting data:

• It has not been checked for timing side channels. Its S-box step indexes a 16-entry table with bits that depend on the key and the data. Such lookups can leak information through cache timing on some hardware, and nothing here measures whether they do.
• It is only the block cipher. It has no mode, no authentication, no key derivation and no safe way to generate keys (never rand(); see random number generation in C and C++).. Each of those is where real systems usually fail.
• It is slow. At 4 MB/s, it is a teaching tool.

If you need Serpent itself, for compatibility with an existing format, use a maintained library: libgcrypt, Botan and Bouncy Castle all include it. For new designs, the usual choice is AES in an authenticated mode such as AES-GCM, or ChaCha20-Poly1305. Both are standard, widely reviewed and available in every major language, and most current x86 and many ARM processors have AES instructions, which Serpent cannot use.

Serpent vs AES

Serpent’s extra rounds buy a larger safety margin at the cost of speed in software. AES, standardized as FIPS 197 in November 2001, won the competition and the hardware support that came with it, which is why AES is the default and Serpent is a choice made for specific reasons. Public-key systems such as RSA work differently again: they rest on number theory, such as products of large prime numbers, while a block cipher like Serpent uses nothing more than XOR, rotations and table lookups.

Key Takeaways

• Serpent is a 128-bit block cipher with keys up to 256 bits and 32 rounds, an AES finalist that took second place, 59 votes to Rijndael’s 86, and is in the public domain.
• Each round is subkey XOR, S-box, linear transformation; the last round replaces the linear transformation with a final subkey XOR.
• The bitslice layout treats the block as four 32-bit words, so the 32 S-box lookups of a round can be computed together with logic operations.
• Two byte-order conventions exist. NESSIE order and the AES submission’s order are byte-reversals of each other; check which one a vector uses before debugging.
• Round-trip tests prove nothing about correctness. A one-bit key-schedule bug round-tripped all 1,043 vectors and matched none of them.
• Test every key size. A missing padding bit passed every 256-bit vector and failed every shorter one.
• Use a maintained library for real data. The implementations here run at about 4 MB/s, libgcrypt at about 89, and none of the code on this page has been reviewed for side channels.

Frequently Asked Questions

Conclusion

Serpent is a good cipher to learn from because nothing in it is hidden. Four words, a 4-bit table, ten lines of rotations and XORs, and a key schedule built on the golden ratio are the whole design, and 32 rounds of them produce a cipher that has outlasted every published attack. The byte-order split is a useful lesson too: a cipher is defined by its test vectors as much as by its specification.

The other lesson is about testing, and it applies well beyond cryptography. Code that agrees with itself proves nothing; code that agrees with someone else’s published answers proves a great deal. More algorithms implemented and tested this way are in the algorithms section.

Source Code and Tests

All the code on this page, with its tests, is in the cryptography/serpent directory of each language’s MYCPLUS examples repository:

Language	Repository directory	Build
C	mycplus/c-examples › cryptography/serpent
C++	mycplus/cpp-examples › cryptography/serpent
Java	mycplus/java-examples › cryptography/serpent
Python	mycplus/python-examples › cryptography/serpent
C#	mycplus/csharp-examples › cryptography/serpent

Build and test the C and C++ versions:

cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure

Java: javac -d build src/*.java tests/*.java bench/*.java, then java -ea -cp build SerpentTest tests/vectors/serpent_nessie.txt. Python: python3 -m unittest discover -s tests -v. C#: dotnet build Serpent.slnx -c Release, then dotnet run --project tests/SerpentTests -c Release --no-build.

What the build checks. In every language: all 1,043 NESSIE vectors with 128-, 192- and 256-bit keys, encrypting and decrypting; rejection of 0- and 33-byte keys; and the demo printing exactly the output on this page. C and C++ compile with GCC and Clang (C11 and C17; C++17 and C++20) and with MSVC /W4 /WX, warnings as errors, and run under AddressSanitizer and UndefinedBehaviorSanitizer. The C build also checks that the three pitfall programs print the outputs shown above and, on Linux, compares 9,000 random keys and blocks with libgcrypt. Java runs on 17, 21 and 25, Python on 3.10 to 3.14 with ruff and mypy --strict, both on Linux and Windows, and C# on .NET 10 on Linux, Windows and macOS. The builds do not run the benchmarks, whose figures came from one machine, and do not test for timing side channels.

What I Could Not Verify

• Side channels. No implementation on this page was tested for timing or cache leakage, on any hardware.
• Other platforms. Everything was run on one x86-64 Linux machine. The MSVC, Windows, macOS, Java 17 and 25, and Python 3.10–3.12 and 3.14 jobs run only in the repositories’ CI.
• Which libgcrypt code path ran. The 89 MB/s figure is for the Ubuntu build of libgcrypt 1.10.3 on this CPU. Its SIMD versions are described from libgcrypt’s source code; I did not check which one the library selected at run time.
• The AES submission package itself. The submission-order check uses the 13 vectors in Bouncy Castle’s TnepresTest, which states that they come from the submission; the submission’s own files were not downloaded. The NESSIE vectors come from the copy distributed with Botan.
• Cryptanalysis. The attack figures are taken from the published paper and were not reproduced.