Corsix.org

I like Lua as a programming language for its beauty and simplicity, but sometimes you have to defile a language with horribly obfuscated code:

     L=           {}           for          k,v
     in          next          ,_G          ._G
     do          L[#k          ]=v          end
    L[10        ](([[p        =prin        t;for
   'q=99,      1,-1'do'      gg'q>1'      th{n'p(
  q.."'Bs'    {f'!:::'{n    'th{'<114    ,'"..q.."
 'Bs'of'!::  [.")gg'q>2't  h{n's=(q-1)  .."'Bs'{f'[
 !::'{n'th{  '<!+$."{lse'  s="1'B'{f'$  !::'{n'th{'
 <onx."{nd;  elsegg'q==1'  then'p"1'B'  {f'x!::'{n'
 th{'wall,'  1'B'of'[!::.  "s="no'mor{  '!::''{n'th
 e'<'!4!"en  d;p("Take'{n  {'down,'pas  s'it'around
 ,'"..s)p"-  "{nd]]):gsub  ("["..[==[$  4[]==]..[[x
 ]].."]",""  ):gsub([[B]]  ,"bottle"):  gsub("''?",
 " "):gsub(  "!:+","beer"  ):gsub("gg"  ,"if"):rep(
 3-2):gsub(  "<..","wall"  ):gsub("{",  function(_)
 B=((B)or(3  ))+1;return(  "eooe"):sub  (B%4+1,(B+1
 +2+9)%4+(#  L[1]-13),(B*  2)%7)end))(  L[#L]or...)

Despite what it looks like, it is valid Lua 5.1 code. If it isn't immediately obvious what it does, then you can see the output.

The end product of a few days of hacking together code: an interactive Lua terminal, in the browser, via AJAX magic.

The javascript side was lovingly ripped from the TryRuby site (as it is conveniently BSD licensed). The server side is implemented using Apache, mod_lua, a custom "persistent Lua" C module and some Lua scripts.

Feel free to try it out, but remember that it is currently very crude and at the technical proof of concept stage, rather than nice user friendly stage.

mod_lua (what was mod_wombat, and is part of Apache core since version 2.3) is something which I'd quite like to work with for Google Summer of Code 2009. In order to work on it, you first need to be able to compile and run it. Doing this on linux is trivial - checkout the svn head of httpd, ./configure (with appropriate command line arguments), make and then sudo make install. Doing so on Windows is less trivial, but it can be done.

For my Windows setup, I've got a standard binary distribution of win32 Apache 2.2.9 (complete with development include and library files) running on Vista. This makes things slightly more interesting, as mod_lua targets Apache 2.3 and above, but I can't find win32 binary distributions of 2.3, nor do I feel confident in compiling it under windows. From this baseline, I did the following:

1. Checkout the mod_lua svn trunk (using TortoiseSVN)
2. Create a new Visual Studio project and add all of the mod_lua source files
3. Add additional include directories ("E:\CPP\lua-5.1.4\src";"C:\Program Files\Apache Software Foundation\Apache2.2\include")
4. Add LUA_DECLARE_EXPORT to the preprocessor definitions
5. Add additional library directories ("C:\Program Files\Apache Software Foundation\Apache2.2\lib";"E:\CPP\lua-5.1.4\Debug")
6. Add lua5.1.lib, libapr-1.lib, libaprutil-1.lib and libhttpd.lib to the library list
7. Add a post-build event to copy the resultant DLL to the modules folder (cp "$(OutDir)\$(ProjectName).dll" "C:\Program Files\Apache Software Foundation\Apache2.2\modules\mod_lua.so")
8. Fix the usage of AP_DECLARE and AP_DECLARE_LUA throughout the codebase (patch available on the httpd-dev mailing list)
9. Copy the ap_args_to_table and ap_body_to_table functions over from Apache 2.3's util_script.c into a new file, util_script_ap23.c. This is compilcated due to ap_body_to_table's usage of the request_req::body_table field, which is not present in the 2.2 version - but can be worked around via usage of apr_pool_userdata_[sg]et with the request's pool.
10. Compile the project and tweak httpd.conf as required to load and configure the module

(for comparison, similar info screenshot on the linux build)

So, time for the daily development screenshot:

I use hex editors a lot. In particular, I use XVI32 a lot. While it's a very nice hex editor, I feel that some extra functionality in many areas would be very nice. For example:

• Ability to open 'massive' (i.e. between 4 and 100 GB) files (XVI32 is limited to 2GB)
• Faster loading of 'large' (i.e. between 100 MB and 4 GB) files
• Plugin system to extend the editor in various parts
• 'Data transformation' system to facilitate data [de]compression and [de|en]cryption within the editor
• 'Structure analysis' system to facilitate viewing the structure of a known formats (e.g. Relic Chunky files) - extended to new formats using the plugin system
• More options for the data inspector (i.e. "byte: 0xf4, x86 instruction: hlt") - again, extended via aforementioned plugins
• More built-in checksums (XVI32 only calculates CRC16 and CRC32)
• 'Data visualisation' system to facilitate viewing plaintext or image data in a suitable manner

As such, I'm writing my own hex editor, HexEditIX, the first development screenshot of which is shown below. (of course, it's not much of an editor yet, but that's software development for you - you have to start somewhere)

First of all, if you're not aware of DoW2DB, it's my latest website - a database of information about the units in Dawn of War II.

One of the reasons for making the site was to see how well Google AdSense (online adverts provided by Google) works. Despite applying for AdSense 3 weeks ago, I've only just been able to put up ads on DoW2DB.

1. February 14th: Applied for AdSense. Application page stated that it should take a day or two to review the application, help pages state it may take up to one week if they are busy.
2. February 26th (almost 2 weeks later): Got email reply stating that the application was refused due to an invalid or incorrect postal address. This confused me, as the address in my application matched exactly the address on my bank statement. Hence I sent an email to AdSense support with a scan of a bank statement with address on attached, and queried why they decided that my address was invalid. Got an automated reply "We noticed that you may have contacted us without first visiting our Help Center: ... To receive faster responses to your questions, we encourage you to visit our help resources -- they're full of instant, reliable answers to many common questions and issues ...". The Help Center does not contain any entries on Google deciding that postal addresses are incorrect when they are not.
3. March 5th (one week later): Still no human response from AdSense support. Decided to re-apply to AdSense without making any changes to the application. Application page stated that it should take a day or two to review the application.
4. March 7th (two days later): Application approved.

This post addresses a few of the questions which have arisen since part 1:

"can you put file to rapidshare" (and similar requests)
I'm here to point out flaws in Relic's security chain. I'm not here to give everyone quick and easy access to the pre-order content, so I will not be posting the modified files myself.

"Well done there, though the question of how long this will be possible for, how much data the Steam client collects about the programs it runs (DLL hashes are pretty standard, or?) and what Relic's views on this might be still remain."
Hashing of DLLs is fairly standard; for example, if you modify SimEngine.dll (perhaps to add a maphack), then you can't join online games as your hashes won't match those of the other players. However, as wincrypt.dll is not a DLL which comes with the game, it is not hashed by default. Furthermore, as it is a windows DLL, it will (probably) change with each Windows version and/or service pack and/or patch, meaning that it cannot be reliably hashed.

"How do I find where this function is which I need to replace the first five bytes of?"
There are tools which list all exported functions of a DLL and their virtual positions, i.e. the position in virtual memory (relative to the DLL base) at which you will find the function when the DLL is loaded into memory. Converting this virtual position into a physical position, i.e. the position in the file on disk of the function, can be done by analysing the DLL's PE section table in a suitable tool.

"I don't see anything about UnlockMask in the content.bin, do you have any more specific instructions for this?"
The contents of content.bin can be viewed in two ways; calling dofile("content.bin") in a Lua interpreter and then printing the resulting DLC table, or running content.bin through luac: (lua and luac are both part of the official Lua.org package)

E:\Lua>lua
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> dofile "content.bin"
> print_recursive(DLC) -- implementation left to the reader

E:\Lua>luac -l content.bin
// luac output omitted here; see full post

Once you've looked at the contents of the file, it is trivial to load it into a hex editor and replace the "1", "2", "4", "8", "16" and "32" strings with "0", "0", "0", "0", "00" and "00" respectively.

Dawn of War II has unlockable content; that is certain maps, campaign wargear, army painter colours and so on are locked by default, and have to be unlocked using special codes. People who pre-ordered the game got a code or two, and codes are also available through other promotional sources.

This unlockable content is implemented as follows:

• The content.bin file in the Dawn of War 2 folder lists all of the unlockable content, in the form of a compiled Lua file.
• Each set of unlockable content has an entry in the content.bin file listing the files/colours/wargear/etc. which it unlocks, and the UnlockMask required to unlock it
• Each GFWL account has an UnlockBits field
• Registering an unlock code sets the appropriate bit in your UnlockBits field
• To decide which content to unlock, the game (DOW2.exe) iterates over each set in turn, tests to see if all the bits in the UnlockMask are set in your UnlockBits (e.g. if UnlockMask bitwise-and UnlockBits equals UnlockMask then), and if so, unlocks the things in that set
• content.bin is cryptographically signed by the content.cat security catalogue file
• DOW2.exe is cryptographically signed by the DOW2.exe.cat security catalogue file
• wintrust.dll is used by DOW2.exe, xlive.dll, and some other DLLs to verify the security catalogues

The standard approaches to unlocking all of the content might include:

• Modify content.bin and set every UnlockMask to zero (zero bitwise-and UnlockBits equals zero). This would fail as then content.cat would fail to verify content.bin and DOW2.exe would abort loading
• Modify DOW2.exe and patch the code to read zero for every UnlockMask. Again, this would fail as xlive.dll would fail to validate the signature on DOW2.exe and multiplayer wouldn't work

Neither of the above methods would work, due to the cryptographic checks done on the content.bin and DOW2.exe files. The weakpoint of DoW2's system is how these checks are done. As previously stated, wintrust.dll (a Microsoft DLL which lives in C:\Windows\System32) is used to make sure that the security catalogue files are valid and successfully verify DOW2.exe and content.bin, using the WinVerifyTrust[Ex] function. Furthermore, wintrust.dll itself is not cryptographically signed. If wintrust.dll is copied from the system32 directory to the dawn of war directory and then modified so that WinVerifyTrust always returns ERROR_SUCCESS, then the cryptographic checks are sidestepped (ERROR_SUCCESS is conveniently the value zero, so this simply means replacing the first five bytes of WinVerifyTrust with "xor eax, eax; retn 12;" or 33 C0 C2 0C 00 in machine code).

If content.bin is now modified so that the unlock masks are all zero, then when the game is run, it'll load wintrust.dll from the game directory rather than the system32 directory (DLLs in the 'current' directory override those in the system directory by default), and when it comes to verify content.bin, the patched WinVerifyTrust returns ERROR_SUCCESS, and so the game believes that the file is still cryptographically signed. It then comes to see what content the GFWL account has unlocked by doing 'does UnlockMask bitwise-and UnlockBits equal UnlockMask' for each unlock set, and as zero bitwise-and anything does equal zero, it'll unlock all of the content.

The Dawn of War II Beta is upon us, which brings with it a series of new file formats compared to Dawn of War 1 and Company of Heroes. The most important of these is the SGA format, which is the archive format used by Relic games for game assets. Dawn of War 1 used version 2 SGA files, Company of Heroes used version 4 and Dawn of War II uses version 5 (for completeness, I think Impossible Creatures used version 1, some unreleased project or The Outfit used version 3, and Company of Heroes Online used version 4.1).

I've already released SgaReader2 for looking inside version 5 (and 4.1, 4 and 2) SGA archives. In order to actually mod Dawn of War II properly, we need to be make to make new version 5 archives rather than just read existing ones, which is where sga4to5 comes in. Carry on reading for details...

I don't usually quote emails verbatim, but then again, you don't get this kind of email every day: