Porting 3D Movie Maker to Linux - part 1

[Update: Ben has now posted about his work at https://benstoneonline.com/posts/porting-3d-movie-maker-to-linux/]

The origin of my interest in 3D Movie Maker goes back nearly 30 years, to when I was a young student working at a Midlands school during my gap year. Since I had a keen interest in computers, I was assigned to help out during ICT lessons: I was the immediate technical support for teachers taking lessons, whilst also helping out pupils who were not familiar with computers to help them keep up during lesson.

It was here that I was first introduced to 3D Movie Maker. During breaktimes and after school hours, the pupils were allowed to access CDROMs with some more fun applications, and I soon realised that almost everyone was spending their time laughing and creating their own movies. I was amazed how it could capture their attention for seemingly hours on end, so curiosity got the better of me and I started to play around in the evenings, whilst performing the almost inevitable weekly rebuilds of Windows 95.

It’s fair to say that I was fairly awful at making movies, but the whole experience (and its impact on the pupils) was so much fun that it made a lasting impression on me.

Now we fast forward to the end of 2024. During the post-Christmas holiday I randomly came across some articles related to the open-sourcing of 3D Movie Maker, which I vaguely remember from a few years back, and I wondered what was the current status - would it run on my Linux laptop? Even better, was it possible to run on a Pi so that my 2 young boys could also have the 3D Movie Maker experience? The articles pointed me towards the 3DMMForever project on GitHub, and after looking around there I could see that the project had stalled some time ago.

As curious developers tend to do, I checked out the 3DMMForever source repository and had a look around the source to see if this was something that was even possible. From what I could see the source tree was well structured and there were 3 main parts to consider:

• kauai: the core runtime library, which appeared to contain support for running on Classic MacOS

• BRender: the Blazing Render library from Argonaut software, which was also open sourced at the same time as 3D Movie Maker

• 3D Movie Maker: the main application that runs on top of both these components

I’ve done a fair bit of C programming in my time, so the language was no problem. And I’d also dabbled with VB/MSVC calling into the Win32 API so I had a fair idea what bits needed to be changed, and how to change them. So why not give it a try and see what happens?

Finding 3DMMEx

The 3D Movie Maker codebase makes use of a number of application classes which are represented both as 4 letter codes e.g. TATR is the theatre class. Whilst searching for information about these classes on the internet, I came across Ben Stone’s excellent set of blog posts about reverse engineering 3D Movie Maker and a list of class codes and their proper names. After a little more exploration, I found that Ben had his own 3DMMEx project containing a copy of the 3DMMForever source code, along with some bugfixes that had been committed within the last week!

Happy to have found an active fork of 3D Movie Maker, I emailed Ben to introduce myself along with an outline of my plan to get 3D Movie Maker running on Linux, and asked if he would be interested to take any patches that I created. Ben immediately replied and said that the plan looked good, provided a few hints, mentioned that he’d been toying around with SDL on Windows (yes!) and finished up saying he was looking forward to seeing my pull requests. So with all the basics in place, it was time to start work.

Modernising the code

When working on a project such as this, I find that it’s almost impossible to know at the start of the process exactly what issues are waiting to be discovered. My normal strategy is to create a “dev” or “hack” branch where anything goes: I don’t care about whitespace, breaking the build for others, naming conventions - the aim is just to get something that works. Using this approach I started to stub out the various Win32-only functions in kauai/src, working my way through each file in turn to get a library binary.

Using this approach, I was soon able to pick out some of the more obvious problems:

• Use of Win32 file APIs in fniwin.cpp
• Use of Windows-specific types (ulong, DWORD, BYTE)
• Many warnings about multi-character constants e.g. #define kclsFOO 'CFOO'

The main problem I discovered with the Win32 file APIs is that certain path resolution functions do not require a file to be present on the filesystem, and that behaviour was used by the FNI class. After a bit of searching I found that C++17 implements a std::filesystem class whose behaviour is almost identical to the Win32 file APIs in this way! But when I tried to use std::filesystem in my local build, the functions did not exist: and that turned out to be because cmake was set to use C++14.

I proceeded to updated CMakeLists.txt to use C++17 and then suddenly my stub-compiling library turned into a mess of compile errors complaining about ‘size’. With some more debugging I worked out that the problem was because kauai defined its own size() macro which clashed with the size() method being used by C++17. After a brief email thread with Ben, we decided that I would use a simple search-replace to replace all instances of the size() macro with SIZEOF() for now, and consider later if we can use the standard C sizeof() function. With that change I was able to continue my stub library once again.

Next I decided to tackle the Windows-specific types. In Linux (and almost all POSIX OSs) projects will use the stdint.h types to ensure that variable sizes are consistent across architectures. Knowing that 3D Movie Maker was a 32-bit Windows application meant that it was possible to come up with a set of mappings that would work on both 32-bit Windows and 64-bit Linux:

It seemed to me that it was a no-brainer to switch everything over to use stdint.h types since they would both a) work in MSVC and gcc and b) gcc would detect any instances where a pointer was being truncated by being cast to a ulong on a 64-bit architecture. At this point I realised that I would need to test the resulting changes on Windows, so it was time to set up a Windows 11 VM along with VS CE 2022 to make sure that everything still worked.

The switch ended up only taking a couple of evenings, primarily using sed to perform bulk replacements on the type names on Linux, checking the results and then pushing the results over to the Windows 11 VM to manually fix up any compilation errors.

The final issue to solve was that of the multi-character constants: when attempting to build kauai on Linux, gcc emitted a huge number of warnings of the form:

kauai/src/fni.h:50:21: warning: multi-character character constant [-Wmultichar]
   50 | const FTG kftgDir = '....';

Looking at the code showed a large number of such constants, in particular being used to define the kauai core classes in the form #define kclsFOO 'CFOO'. After a bit of experimenting, I could see that the issue was because these constants were being cast to a 32-bit ulong for use as magic values, so my solution was to devise a new set of KLCONSTX() macros to manually convert each letter of the constant “string” to a 32-bit value:

#define KLCONST4(a, b, c, d) ((a << 24) | (b << 16) | (c << 8) | d)

...
...

const FTG kftgDir = KLCONST4('.', '.', '.', '.');

I then went for the same approach as I did before with the Windows-specific types, and use a script with a handful of regexs to do the majority of the conversions in the src directories, and fix up by hand any remaining issues. After the conversion passed a few basic tests on my Linux stub setup, I attempted to build 3DMMEx with the same changes on my Windows VM and to my dismay, a whole new set of errors appeared whilst running chomp to generate the .chk datafiles containing the binary resources.

Checking with grep I discovered that the original authors had been quite cunning here: not only had they used some of the multi-character constants as a 32-bit ulong, but also as strings in the C++ preprocessor when processing the .cht input files to chomp! With this new knowledge I realised that within the preprocessor I couldn’t use the same #define for both the 32-bit ulong and the string version of the constant - but was there a better way than simply duplicating each constant?

As I was currently examining the chomp pre-processor code in cmake, it dawned on me that I should be able to use cmake to take an input file of constants and generate both the 32-bit ulong AND the string constant from a single source file. I therefore took the duplicated constants and added them (without the kctg prefix) to a new kauai/src/framechk.in file, and then added a new GenerateChunkTags cmake target to generate both constants: the 32-bit ulong constant with a kctg prefix in one header file, and the string constant with the kcctg prefix in another header file.

These files were then generated as part of the build using cmake, and then the .cht files updated to use the kcctg prefixed string constant: at this point the Windows build was able to complete successfully, and a large amount of compiler noise magically disappeared from my Linux library stub.

Moving to a 64-bit world

During the stdint.h conversion it became clearer that there were actually 2 separate tasks that were required to get 3D Movie Maker running on Linux. The first was replacing any Win32-specific APIs with ones that would work on a POSIX OS, whilst the second was ensuring that the software would run on a 64-bit OS.

In particular I noticed there were a number of places in the code whereby a pointer would be cast to a 32-bit value and then back to a pointer, which would just not work on a 64-bit OS. My worry was that I would experience random crashes and then spend a huge amount of time trying to find all the instances where this occurred. I figured out that the next step would be to get 3D Movie Maker running on x64 Windows, since if I could get it working there, I could guarantee that any crashes/bugs found on Linux would be due to using a POSIX OS and not a 64-bit OS. And even better, I could debug 32-bit and 64-bit 3D Movie Maker applications side-by-side if needed to find those elusive bugs.

It was time to attempt to build 3D Movie Maker on x64 Windows. The first problem to solve was to find a version of BRender that would compile on x64 and Linux (and ideally macOS). At the time 3D Movie Maker had been open-sourced, a number of BRender forks had sprung up and so myself and Ben did some testing: we needed a version of BRender that supported OSs other than Windows, non-x86 architectures, and fixed point arithmetic. He eventually discovered that @prettytofugirl’s 3DMM-Brender fork met all these criteria, and worked as a drop-in replacement for the pre-compiled BRender libraries for an x86 32-bit build… except that Actors were rendered in a lovely shade of rainbow stripes.

For now I could at least use the BRender fork as part of my x64 build, even if the rendered colours were incorrect - that could be fixed later. And Microsoft provide some fairly good documentation for converting Win32 applications to x64, so it was a fairly straightforward exercise to work through those. However when trying to run 3D Movie Maker on x64 I kept hitting a static_assert() failure at compile time.

As a bit of background, when I mentioned to Ben that I was going to try and get 3D Movie Maker to build on x64, he had gone through the various serialisable C structures in the codebase and added C++ static_assert()s so that any changes would generate a compile-time failure. The main failure I was seeing was due to the TAG structure, which is embedded within a number of 3D Movie Maker on-disk structures:

On x86:

struct TAG
{
    int32_t sid;       // size 4, align 4
    PCRF pcrf;         // size 4, align 4
    CTG ctg;           // size 4, align 4
    CNO cno;           // size 4, align 4
};                     // Total size 16 bytes

On x64:

struct TAG
{
    int32_t sid;       // size 4, align 4
    PCRF pcrf;         // size 8, align 8
    CTG ctg;           // size 4, align 4
    CNO cno;           // size 4, align 4
};                     // Total size 24 bytes

Yikes. So the problem is that a PCRF pointer had been embedded in the TAG struct, which of course is different in size between x86 and x64 Window. From what I could see the pointer was being used as a cache once the TAG had been loaded, so its value was actually being ignored during serialisation but had the side effect of changing the size of the structure being written to disk.

Looking at the TAG structure it seemed the best solution would be to replace pcrf with a dummy int32_t value and then move it to the end of the structure, outside of the serialisable section so it would look something like this:

struct TAG
{
    int32_t sid;       // size 4, align 4
    int32_t _pcrf;     // dummy: was pcrf, size 4, align 4
    CTG ctg;           // size 4, align 4
    CNO cno;           // size 4, align 4
    PCRF pcrf          // who cares?
};

It seemed a daunting task to try and locate the serialisation code for all serialisable structures that embed a TAG, until Ben happened to mention that 3D Movie Maker supports saving in both big-endian and little-endian formats, and byte-swapping was handled by the SwapBytesBom() function.

Armed with this knowledge I then came up with a simple plan: introduce a new TAGF stucture that is exactly the same size of the TAG structure on 32-bit Windows, and ensure that it is used in all serialisation and deserialisation routines. To make this even easier I also added a couple of functions DeserializeTagfToTag() and SerializeTagToTagf() that I could drop to convert between one and the other.

With this done all I had to do was ensure that any project files loaded and saved were identical on 32-bit Windows, at which point I could add a single commit to move PCRF to the end of the TAG structure which should then allow it to work on x64 Windows. And so with this work complete, I was able to follow through the standard guides for porting 32-bit Windows applications to 64-bit Windows applications, and suddenly after a few more tweaks, I was staring at the 3D Movie Maker menu running on x64 Windows!

I did some quick tests, and from what I could tell, everything was working as expected apart from as noted earlier, most of the Actors being rendered in rainbow stripes. Ben spent some time looking at this, and with luck found a patch in @jayrod246’s BRender fork that appeared to fix this issue! This pointed towards a problem with the triangle rendering functions, and with a bit of trial and error I was able to isolate the problem and file a GitHub issue with the fix at https://github.com/prettytofugirl/3DMM-BRender/pull/3.

Now we had a fully working x64 version of 3D Movie Maker, the next task was to switch back to trying to get 3D Movie Maker running on Linux. With a young family I wasn’t able to spend much time working on this over the summer, however with a huge effort over many commits, Ben had managed to get his SDL rendering experiments into proper shape and working on x64 Windows. This meant that I now had all the dependencies required for 3D Movie Maker to run on Linux.

Next: Porting 3D Movie Maker to Linux - part 2