Implement thread synchronization and a bunch of stuff

[viroulep]

Finally I could take some time to make a proper PR...

Lets start by the bunch of related stuff:

• First commit make custom allocator be a cmake option.
• Last commit fixes a bug that occurs sometimes: if the game does not start fast enough, linTAS would fail to connect; so I implemented a simple try/wait mechanism.

About thread synchronization now.

Here is the few issues we want to address:

1. every single thread is created through pthread_create, we need to be able to differentiate them using a solution which is platform independent.
2. There may be threads we want to wait for (loading ones), and other which we don't care
3. We need a synchronization mechanism that would work for multiple games

Point 1

The first thing to test when you'll see this PR is 1!
My first try for this was to register the calling address to SDL_Init, and use a ptrdiff with the entry points given to pthread_create on thread creation, assuming it would be the same regardless of the platform (as I expected both addresses to be in the same non-relocatable binary executable).
It worked ok for SMB (which is one big executable), but games such as FEZ may not be monolithic.
The loading stuff for FEZ seems to be in a separate library, which means it's relocatable relative to the main binary, and that simply invalidate this first approach.

I managed to get better result by identifying each thread using the ptrdiff between the routine's address and the caller address.
This approach will break if the routine passed to pthread_create is not part of the same object file as its caller. Lets just hope you don't want to TAS games like that.

The value identifying each thread should therefore be only dependent on the game executable.
Two different entry points may have the same id only if their caller has the same relative position, which is very unlikely.

So when you checkout this, please activate the LCF_THREAD messages in taskflags.cpp, and run the 64bit version of FEZ (we should have the same executable).
Just open it, go to the main menu and close it, at the end of the execution you should have a summary of the run looking like this:

Record for entry point : -557

• AcwU0i2f
  1: Started
• Ac2IYj2f
  1: Started
• Ac5DZj2f
  1: Started
• AcKsZj2f
  1: Started
• Act0Zj2f
  1: Started
  These threads started before SDL init and can't be waited for :
  0x7f68f82745f0
  0x7f6907b7bda2
  0x7f6907b7ce7f
  Game has quit. Exiting

All these number can change but one: you should have "-557" or else we should find another way of identifying threads :s

Point 2

Tracking every thread creation make it easy to generate a summary as above: I register which entry points have been created under which pthread_id (which tells us the number of time a particular entry point has been called).
Playing the game through a couple of level should give a good overview about which entry points are used, and tell us which one we should wait for.
For instance during my test with FEZ, entry point with id 2003 was only started once at the beginning, so it was probably related to automatic save, that doesn't lead to desync during the game (so we don't care).
However -557 was created at every load, which means we do care about it.

Ideally we would detect that automatically, but we could also simply create a simple database for known game executables.

Point 3

Here is the fun part.
As far as I could test, games create threads and detach them at the end of the routine execution, instead of joining them.
This is probably because the main thread doesn't want to sleep and want to continue to render frames during loading phases.

So the idea is the following:

1. For every thread creation, get its id
2. If we care about this id (ie: we expect level loading), tell the main thread to sleep and wait for it
3. When a thread is detached, wake up the main thread if it is sleeping.

Here 2 is not straightforward, as we want to pause the main thread from another thread: we cannot use pthread's structures like conditions to do that.
I did this through a custom sighandler for SIGUSR1.
pthread_kill enable us to send to a given pthread_t a specific signal, so we register the main thread's pthread_t, and we send SIGUSR1 to it when we want to suspend it.
The custom sighandler is a simple spin on an int. When resume is called, the spinning variable is reset, allowing the main thread to resume its execution.

Here is the fez input file I used for this video: https://we.tl/zekoUXg748 (will disappear in 7 days).
So if the magic number matches, a good test would be to run ./run.sh -r fez.input pat/to/FEZ.bin.x86_64 and see if you can replay this :p (assuming the first save loads the same level obviously).

Misc

I tried to hook the pthread_cond primitives but couldn't do it with the last master for some reasons that I didn't look into. I removed this in 7a08a60, it may give you ideas about what else I tried to track and understand the synchronizations in the game.

I hope I described my approach clearly, please let me know if some points are still obscure.

[clementgallet]

Thanks for this PR.

I got the same value for the FEZ ptrdiff that handles loads. The outcome of the input file did differ a bit compared to your video (by about 10 frames), but it is possibly because my save is not the same as yours even if both place the character at the same spot (I collected bits and opened the top door).

I tried to launch Towerfall and it also uses -557 as entry point for loads. Actually both games are FNA ports from XNA games.

There was a problem when enabling fileio hooks (open, fopen, close, etc.) together with the thread manager, but fileio hooks are incompatible with a lot of stuff right now.

Next step would be to insert entry points automatically using a set of rules, and save them in the movie file. I will probably modify the movie file format by switching to the way BizHawk or lsnes is handling it: using several files (movie file, config, threads, etc.) and zipping them together, instead of using a single file with a header and painful parsing.