Just a short update: normally I'm pretty responsive about answering questions etc. However, my smartphone has just died - and hence I'm not getting notifications of questions here at all quickly.
So if I'm late answering a question, it's not because I'm neglecting my userbase or abandoning the project, it's more likely that I just don't know there's a problem. I'll try and get responses within a reasonable time, but don't be surprised if it takes a few days.
OK, so this release represents most of my work in the "speedy" line. It isn't all of my work, because I've only dropped the speedy patching engine into the old framework; there's other improvements I'm making in the framework which should make things faster, but they're not ready and I don't want to delay getting this new release out any more.
I also have an apology to the Korean team who wanted to use RPGMaker Trans: testing revealed another problem when changing the locale of the game, specifically that files named in Japanese can no longer be used - with this breaking most of my test games. Automatic file renaming cannot be handled by the current framework, so multi-locale support will have to wait until the next version. Sorry.
Changes
So I couldn't quite keep meet my target...
Basically, lack of time and a few problematic bugs stopped me. I'm finally at the stage where the parser/assembler works without issue, but it isn't yet integrated into the translator or GUI - and nor are the changes to these two parts finished yet either.
I'm very disappointed that I couldn't make it in time for an ironic April fools release though...
Well, new version within the week, I suspect.
Well, I've been investigating stuff, and turns out things aren't perhaps as rosy as I thought.
Turns out things are more complex than my initial survey of hosting providers suggested. Long story short, to actually run a website I'd need about $10 a month all told. So, whilst I've very grateful that just over half the respondents to the little survey said they would be prepared to fork out some money to move RPGMaker Trans onto a dedicated site, the finances don't work out.
Damn, that's a shame. Oh well, maybe if I can get some other projects going I'll be able to work something out... but for now, a dedicated site will remain a pipe dream.
Target is for the new version of RPGMaker Trans, with speedy 2k support, to be out by the end of the month. Then I'll move onto either automatic machine translation or XP/VX support.
Well, after publishing my roadmap, and making some progress with it, I decided to essentially ignore it and do something else. So, here I'll document exactly what I did with the only time I had to work on RPGMaker Trans recently.
I made a Python binding onto Fujitsu's ATLAS software.
Now, anyone with two brain cells should be able to work out that this means I intend to bring automatic machine translation directly into RPGMaker Trans.
To be clear, I'm not sure when this feature will land. It still isn't on my Roadmap, although logically it shouldn't be that difficult to implement - at least in a primitive form. I'm looking to "do things right" though and enable the creation of some small scripting component which allows scripting around the types of things that ATLAS doesn't translate well. For example, for some games which "announce" character names, ATLAS will decide to merge the character name into the characters dialogue with bad consequences.
But this type of scriptable translation could also be useful for other games, for pretty much the same reason. Hence I'll also be working on a "Scriptable QuickATLAS" style program - name to be decided. There's probably quite a few uses for such a tool, like working around "triangle-ised" text from an AGTH hook (you'll know it if you've seen it), other hook based nonsense, or just generally improving ATLAS translations.
Again, no ETA on this.
Oh, and before I forget, the Python/ATLAS bindings will be released separately under a Public Domain license, once I do a little stress testing on them to make sure they don't fall apart under non-trivial workloads.
And finally, before anyone points it out: I am aware of Vkozyrev and his machine translator for RPGMaker Trans patches. I think, however, that using ATLAS has substantial benefits over his solution of online translators, partly because of speed but also because a lot of people use ATLAS, and hence a lot of people are accustomed to ATLAS-speak. Also, I tend to find that ATLAS with a good custom dictionary simply works better than the online translators. Finally, by integrating machine translation into RPGMaker Trans, Machine Translation becomes much more available (i.e. I'm pretty confident that not many people actually download Vkozyrevs machine translator, otherwise I think we'd be seeing a few more actual patches/pirated games based on it online rather than just the ones he maintains).
Well, Megaupload died. Everyone should know about it by now.
Problem though is that the cyberlockers are now running in fear. Filesonic is no longer offering any services - problem as they're one of my mirrors. Also uploaded.to is blocking the US - and whilst I don't use them, considering that blocking the US is probably a correct way to remove yourself from the jurisdiction of the FBI, I can see other cyberlockers following suit.
Now small rant upcoming (don't want to read, skip to the next paragraph): I think the US exceeded its jurisdiction (getting a German national arrested in New Zealand seems a little over reaching). However, it seems Kim Dotcom (probably - pending trial to establish facts) deserved it. By all accounts it does seem that Megaupload was not responding to requests by copyright holders to remove files, so he was (probably) one of the "bad guys" in the copyright debate. I'm definitely against piracy, as I am a content creator under other names, and if the reports are true on Megaupload then it probably deserved to be shutdown. But arresting a non-national in a foreign jurisdiction seems excessive. It isn't as if New Zealand doesn't have anti-piracy rules, and there shouldn't have been any problem in arresting and trialling him in either New Zealand or Germany.
Needless, regardless of any rants or whatever, everyone has to deal with the aftermath. It seems pretty likely that at some point in the future most cyberlockers will become unreliable (perhaps Mediafire/Rapidshare will stay above board, but the rest, I doubt it). Hence I'm looking to perhaps take RPGMaker Trans onto private hosting. The catch is that this costs money, and I'm pretty unwilling to actually spend money on distributing RPGMaker Trans. That said, private hosting would enable a few neat things to go ahead that up till now haven't been possible.
So I'm putting out a poll on if people would donate to the RPGMaker Trans project. If there's enough people backing me, I'll look to setting up a proper website for RPGMaker Trans and whatever other projects I make. I'm not looking for a huge amount of money - on low end VPS's the price would probably be about $50 a year to cover my costs. So basically, if I get say about 20-30 backers, there'll be a dedicated website. If not.... ah well. What could have been, I suppose.
And so the new speedy parser is done. Well, done enough that I can get useful performance data out of it. Technically, it needs integrating into the translation engine, but it works pretty well. Also LMU files, but that should be less than 20 minutes work once everything else is ready.
So, the summary of improvements: Much faster (10 times, at least), and now picks out "Change Hero Class Name" (thanks shadepariah for pointing this out and providing a sample of it). Also, (untested) should pick out if a face is being displayed on the dialog - this is useful for translators as a face decreases the amount of space available for text. Finally, the script parser should be much more robust than it was before.
Also note: as this parser uses a different internal representation, there won't be a release until a new serialiser has been written.
The rest of this post is a programming type post, for people interested in the intricacies of programming high performance Python. OK, a lot of this will also apply to many other languages, especially interpreted ones, but mainly Python. If you're interested, look after the break.
1) Lambdas are bad
For functional programmers, Lambdas are really useful tools, but in Python they are bad news. Why? They're slow, and they all show up as "lambda" in a profiling. That means it's impossible to tell where your bottlenecks actually are, and hence you can't improve performance easily.
If you need something like lambdas, I'd recommend a function factory now. Python lets you do them easily (try defining a function inside a function; every time the function is called, you'll get a new function. Now put variables from the parent functions local scope into the child function, have the parent return the child function, and you have a function factory).
2) Use language features to do as much heavy lifting as possible
Old RPGMaker Trans used a very verbose representation for constructing RPG Ints, by converting characters to bitstrings to ints. Now, whilst I did have a good reason for this in reverse engineering it (specifically, that I needed to see what the RPG Ints looked like), this hammered performance badly. The new version uses Pythons struct library for unpacking the file into ints, then bitwise math for the rest. Much better.
Another things to point out: don't use a custom object when a language native one will do. Initialising objects is much more expensive if the interpreter has to think about it, so stick to language types (like lists, dicts etc) as much as you can.
3) Data copying is bloody expensive
Old RPGMaker Trans copied data around in memory like nobodies business. Not a good idea, as it turned out. Compounded by the fact that Pythons string construction is not terribly efficient
4) It's cheaper to do one big operation than hundreds of small operations
In an interpreted language, function calls have high overheads. In fact, any operation has a high overhead. Simple message: if you can at all help it, batch up operations and do them all in one go.
This is just a post to say "I'm not dead". True, I've been working on RPGMaker Trans a lot slower than usual, but I'm not dead. So, with that, here's a little breakdown on what's been going on:
1: Bugfixes
I've found the cause of the bug involving the GUI build and non-ASCII directory names; a bugfix for this should be out soon. There also seems to be a more obscure bug floating around regarding conflicts with existing Python installs - I'm not sure what's going on there, and can't reproduce it. There may be an attempt to fix this, but I'm not sure.
2: Improvements
I've got to say: most of the parsing engine that's presently in use is pretty old code - and a lot of it is from when I was trying to figure out the RPGMaker 2k format. Whilst it's quite easy to work with from the reverse engineering point of view, as I'm sure everyone will have noticed, it isn't fast. I'm presently working on a new parser which is much, much faster - though only about 70% complete so far. It will also need a new rebuilder as well, but this should be fairly straightforward to make once the new parser is done. So expect a speed up in the near future.
3: XP/VX support
The big one, really. I'm still working on memory usage with this; there's some complex issues with regard to how Windows handles pipes (poorly) and Ruby 1.8s memory model (bad). Net result is that when I tried my current build of XP support on Windows, I used all 6GBs of memory before it blew up. I'm working on a new system which should solve the memory issue. So support for newer games is still on the menu, but will take a while.
It seems that some people are distributing prepatched copies of a certain game. I'm sure everyone involved knows who they are.