Hopefully final progress report on v4.0c, playing around with new site

So, hopefully this is the final progress report for v4.0c.

What's changed? A handful of things:

  1. GUI and CLI executables will be in separate packages. cx_freeze doesn't have any real way of having the two modes in the same package (or indeed, the same executable), so it's going to be 2 packages available for download. On the plus side, having to do this force me to sort out some long-standing source code organisational issues, and then implement hacks around Python bugs to get it to work on Windows...
  2. Changing the way data files are loaded - again: My previous solution was to load a shim RPGMaker compatibility layer and then load the scripts of the game into Ruby so that Ruby could unmarshal all the classes. This turned out to be a bad move because many games do processing on script load (rather than on game start), and so many games would crash the Ruby interpreter during this stage. The new approach is something I implemented nicknamed "unmarshall", and basically uses the errors generated when failing to unmarshal something to generate a shim to enable unmarshalling... if that makes any sense.
  3. Move from Ruby 1.9 to Ruby 2.1: because one thing I found out during implementing unmarshall is that under certain circumstances Ruby version < 2.1 will quite happily unmarshal a data structure to completely the wrong class, without even a hint on an error message.
  4. Hidden support for translating script labels: My test case game requires this. However, to stop labels polluting the majority of patches which don't require it, it's disabled by default. And has no option to enable it - although there will be one, probably in the next semi-major version.
Things that need to happen before v4.0 final:
  1. More testing
  2. UI bug when unpacking a game (progress bar stuck at 100% during patching) 
  3. Release channels
And the final thing in this post... It turns out that I hate HTML. Unfortunately, that's what my website is written in. I'm currently experimenting on a replacement website, which will be powered either by Nikola or Urubu. I'm not sure what I'll end up with, but whatever it is there will be pretty big changes on the website in the near future.

More Progress on V4

This is just an update, considering 3 weeks ago I said 'expect something soon' and then nothing on this blog.

Basically, my code has inexplicably stopped working on Windows. I am genuinely confused as to what is going wrong here, because I have no reason to believe that the code shouldn't work with Windows, and yet it does not.

For those technically minded: after this commit, things don't work on Windows anymore. There seems to be some kind of deadlock, but I'm still no closer to figuring out where it is or why it happens. All I know is that patching jobs get submitted to a multiprocess Pool, but they never start. My concern is that there is a problem within multiprocessing, because the aforementioned commit does not do anything that should affect this, and the problem persists even when I make the jobs modal (which is definitely worrying...)

So there'll be a new release when it's ready, basically. I'm hoping this is sooner rather than later, but as I'm genuinely lost on this issue, no promises.

Progress on V4

So, what's the status of V4? Well, here's a list of what I know to be not working in the 4.0b release.

  • CLI mode is executable is broken. It's actually been broken since release, which suggests it isn't being used much. In any case, this should be fixed soon.
  • XP support was broken. Hopefully this is fixed in current code (not released)
  • Ruby 1.8/1.9 differences are causing problems. I'm investigating ways to fix this, but it's looking tricky.
  • Anything using non-standard DLL extensions doesn't work. 
  • And some problems to do with bad emulation of the RGSS scripts.
The last three issues are caused because I'm just loading the Scripts file, and a lot of games appear to do loading and processing on load (rather than on game start). While I'd call it sloppy programming, not much that can be done about it, unfortunately. Hence, I'm considering another new loader. This will either be doing some very lightweight processing on the scripts to figure out the structure (without doing any processing - something that may work as I only require module/class names), or brute forcing the structure from Ruby error messages.

Expect a v4.0c release shortly to fix the first 2 issues, then something else later to fix the others.

RPGMaker Trans v4.0 Prerelease

So I'm uploading this as a prerelease first, because of the massive and extensive changes to RPGMaker Trans. Seriously, there's a lot that happened since the v3.07 release. Excluding bug fixes, here are the big ticket items:

  • RPGMaker XP and VX Ace are now supported. Although XP support is somewhat untested at present.
  • Translated scripts now have some validation done to them. The validation is not perfect, but it'll certainly stop you from accidentally leaving a closing quote off a string (which is the most common error I've made).
  • Heavily customised games should now be supported. Scripts are now loaded into RPGMaker Trans, which should allow even the most heavily customised game to be translated.
  • v3.2 Patch format simplifies contexts by removing class names, correcting the order of things to be more logical, and fixes a bug where InlineScripts didn't have their position in the event page displayed.
  • And (hopefully) games residing in a non-ASCII folder will now work properly.
The reason for announcing this as a prerelease is there is one thing which needs more broad testing than I can realistically do by myself, which is the method for handling heavily customised games. As this loads in scripts, I'm a little concerned that the incompatibilities between Ruby 1.8.7 and 1.9.3 might mess up some XP/VX games. If anyone encounters any such game, or any game which doesn't work, please file a bug report.

The prerelease can be found on the RPGMaker Trans website.
EDIT: Prerelease updated to 4.0a to fix a crash when choosing a game through the file picker.
EDIT2: Prelease updatead to 4.0b to fix a crash when launching a Ruby process through GUI.

RPGMaker Trans v3.07 Released

So... the number and severity of bugs appears to be going down. Here's a quick release, which contains the following fixes:

  1. Correct a bug where multiline inline scripts would not parse
  2. Include the README and LICENSE files in the distributed binaries
Assuming that's the last of the bugs in the v3.0 series, the next release will probably be a major version bump due to adding in support for VX Ace (and hopefully XP), as well as the new script loader and minor patch version.

Revisiting Contexts

So... the v3.1 Patch Format has only been in public for, well, a few weeks, and I'm already thinking of upgrading it. The reason? Again, it boils down to trying to get support for VX Ace and custom scripts shoehorned in.

Now, it's important to preface this with the following: v3.1 patches will have a transparent upgrade to v3.2. Unlike the transition from v2.x to v3.x, which involved the creation of entirely new classes of context (as well as syntax, persistence etc), the v3.1 to v3.2 transition is much milder, and only involves rewriting contexts. The absolute worst case is that some of the older style contexts will remain, but that's it.

The reason for this is that there's a tiny corner case in how Ruby files were processed. The corner case meant that it was impossible for certain classes to be seen by the patcher as an entity in and of themselves (although the patcher did see their constituent parts). This meant that the newer, adaptable detectors never saw what they needed to detect, and... well, it didn't go well.

What's the benefit for this? Well, ignoring the addition of VX Ace support, my test VX Ace game includes some custom scripts for what I'd call 'inter-party recap of goal events'. These are simply common events, but they're in a non-standard file. The older, fixed structure detectors missed them completely, and would require some custom code adding to the patch to make it work. The newer adaptable detector finds it without issue.

Downsides? Well, it's a little slower, and I'm still trying to weed out some of the things that don't need translation. For example, I thought it'd be reasonable to translate any name attribute as a string, but it turns out there's a lot of names in a game which are only for the developer to see. That's slightly annoying.

Anyhow, I'm hoping for VX Ace + custom data support early next month, once the v3.2 patch format is ironed out. Hopefully that will go well.

Revisiting Script Loading

Well, I do know that RPGMaker Trans's VX use hasn't been widespread, at present - far too many critical bugs which weren't found. But oh well. In any case, much more interesting is adding VX Ace support, given that's where all the new games are at.

In any case, I've picked a test game, and then found that my current method, a generic version control script for RPGMaker, doens't work. Custom data types, which Ruby doesn't like unmarshalling without a definition. Of course, a sane person would have picked a new test game, but it's far more fun to get custom data types working. Also more useful in the long run.

Therefore I'm migrating away from the older approach of assuming the structure and onto an approach where the structure is loaded from the Scripts file, as happens in the RPGMaker runtime. There's a couple of downsides to this, unfortunately. The first is negligble - the scripts get a chance to run, which leaves you open to malicous scripts. While this sounds bad, it leaves you no less vulnerable than if you just ran the RPGMaker game, so I'm not concerned here.

The second is somewhat more annoying: XP/VX use Ruby 1.8, XV Ace uses 1.9, and these have different syntax's. This probably means that on Windows I'll end up shipping multiple Ruby binaries in the pruby folder. On other platforms... it's less clear what I'll do, but probably the onus will be on the user to provide an appropriate Ruby interpreter (perhaps using RVM). That said, most of the time it should be perfectly fine to use a different Ruby interpreter, it's just the little corner cases.

There's also going to be an overhaul of how things are checked for if they're translateable - probably along the lines of Ruby files which inform the patcher - but before I can comment on how that's going to work I need to get a better hang of Ruby metaprogramming so I can provide a nice, clean, interface.

On a side note, does anyone know how to programmatically call all the methods on a Ruby Module? I know how to do it with a Class, but finding out how to do it for a Module is driving me nuts... EDIT: Think I've got this now...

Prerelease RPGMaker Trans v3.04

So there's still bugs in v3.03. v3.04 is available on the download page, but isn't officially being released - it still counts as beta software.

Here are the bug fixes:

  • Fix for potential (although obscure) confusion with Patch/Asset files/folders in v3 patches
  • Fix for patched inline comments crashing the game
  • Fix for crash on exit (temporary)

RPGMaker Trans v3.03 Released

Probably should do a release announcement for this. New version fixes 2 bugs:

  • Comments in v3 patches could modify the string inserted
  • Crashes with version controlled games/patches

Also doing VX Ace support is much harder than I at first anticipated.