Implementation is not AES

[TiberiumFusion]

It seems like the the AES implementation in AES.lua is not actually AES. This library is consistent with itself, but not with the rest of the world.

Comparison against .NET's RijndaelManaged and OpenSSL

ECB-256 Encryption

Key: 68C756C6C186436C9EC51C174C32AE81761389B5E5904E30BA57CCD911290ECC
Plaintext: my secret, with extra length so that it is waaaay over 16 bytes. (64 chars)
Ciphertext:

• Lua_AES: 5281a86fc7c4dfb409fcfc937a644912beed7732ad3bd09d540f36926abbe445d4aa6f90ccc075ce0fdbe968ddb1b4e9257c536c55c3153a45e8b30e23f961d7
• .NET & OpenSSL: 375b2fcf832d21eec0baa29c145872ee8b3228a2d26e221a9647e42993e3f8584a26862cc40ea90129a7ef3462fa0fe7a865d8404bfd83f7ba60cf470e363f8a

ECB-256 Decryption

Key: 68C756C6C186436C9EC51C174C32AE81761389B5E5904E30BA57CCD911290ECC
Ciphertext: 375b2fcf832d21eec0baa29c145872ee8b3228a2d26e221a9647e42993e3f8584a26862cc40ea90129a7ef3462fa0fe7a865d8404bfd83f7ba60cf470e363f8a
Decrypted plaintext:

• Lua_AES:

 t▒▒]`▒)>▒▒D▒▒▒▒(▒nnD▒▒▒zYc▒k`▒▒▒Y▒▒5`I▒W▒=
▒▒▒▒▒\t▒▒@▒▒E▒▒_▒2n,▒M▒̧▒?$▒L
                            ▒_Z▒*▒W<▒5▒Ǔ▒jO▒▒▒<#ħ▒▒D▒▒t▒▒f_▒<R

• .NET & OpenSSL: my secret, with extra length so that it is waaaay over 16 bytes.

---

Tested with LuaJIT.
I had to modify schedule256() to accept a hex string as a key, since I don't know of any Lua distribution that has native 256-bit integers. (see: main.lua line 2)

[idiomic]

I remember looking into this a couple years ago when the comment was made, but couldn't identify the discrepancy. I'll soon tear it apart and add some debug statements to make sure that the internal state is identical at key points. I'm thinking that it has to do with the endianness of the key. The OpenSSL version uses an array of many 32 bit unsigned integers, which means that every 4 bytes may be reversed (depending on if the command line parser reads numbers and assigns them, or performs a memory copy while converting the hex).

[wangpan2333]

AES.encrypt uses padding that is not removed when decrypting. The original string is 64chars, and after AES.decrypt it is 80chars.

[mullerdavid]

I dissected it, and fixed it for myself. I use a slightly different key format, just a plain zero indexed table, so it might be different here and there.
I used this as a basis to track the vector/state: https://csrc.nist.gov/files/pubs/fips/197/final/docs/fips-197.pdf paragraph C.3

The main differences I found were the following:

• The rcon indexing is off by 1
• The shiftRows and mixColumns are transposed

Also restructured/modified the key expansion so I can understand/debug that part better, but I think that was okay as is.

See my modified version: crypto.txt