the updating problem :)

[jack1974]

Warning: some technical stuff and ramblings. You can ignore this if you just want to play games is mostly for _renke haha

As some of you know if you follow me on twitter, yesterday did 3 updates. Now the game is getting big, 146mb, so basically I've uploaded 450mb for an actual change of ... maybe 50kb?
This is a big unsolved problem and that's why I might use Steam Early Access for the final testing of the game. I'm not eager to do that and I would still provide updates the "normal way" anyway, but for example I could have done yesterday testing much faster.
I've spoken with other indies asking what they use, and everyone is in my same situation.
I've tried to use rsync but the fact is that the amount of time to calculate the differences was higher than... just uploading the file with my connection takes 15-20min to upload to a server nearby, and then I copy the file with ssh to my CDN server (that is even faster, so overall the process is around 20-25min max).
With rsync was taking 10-15 min so was quicker on itself, but then I had to .zip the folder, and at that point I wasn't even sure if would work or not (I've never tried this before). And I had to use cygwin, because the rsync implementation on Windows aren't exactly good IMHO. But in general, is more time consuming for me because I have to remember even a series of steps, and I don't want to stress myself even more. Every new step in building the beta = more chances of failing = more chances of me wasting days doing nothing instead of finishing the game.

Comparing this to steam where you just launch a batch file and done...

[renke_]

ah yes, I think I see the issue with your rsync approach - give me a couple of minutes to prepare an example to show you how I would do this.

[jack1974]

OK, I've read tons of tutorials but I lost it at the 30th commandline command to type just to setup the thing

[renke_]

say this is your situation:

on the remote box (your webserver) the file SOTW-0.8.24.14-all.zip is stored, locally on your win box you have the update SOTW-0.8.24.15-all.zip - nearly identical and it would be nice to copy only the diff to the remote server :)

rsync should be quite effective in reducing the amount of time, but to calculate the differences the tool needs a basis for comparison.

I would do this:
1. ssh to your webserver (with putty probably) and copy the last upload

Code: Select all

cp SOTW-0.8.24.14-all.zip SOTW-0.8.24.15-all.zip

the copied file is (of course ^^) not the real .15 but a good basis for rsync's algorithm
2. run rsync on your local machine, and copy the actual SOTW-0.8.24.15-all.zip to the fake one on the webserver

Code: Select all

rsync -av SOTW-0.8.24.15-all.zip <user>@<webserver>:/var/www/downloads/SOTW-0.8.24.15-all.zip

(or similar, so far I didn't open your box and don't know the real directory structure :P)
when you add the --progress option rsync is a little bit more verbose, but because of the diff calculation the ETA is a lie ;)
3. from your remote box push the .15-release archive file (as it's now the real thing) to your cdn

With 2 mbit bandwidth a direct upload of .15 takes 10 minutes, the process outlined above (copy .14 archive to a fake .15 zip and rsync the diffs) 5 mins.

I think the stuff above is rather straight-forward (remote: cp <old-file> <new-file>; local: rsync <actual-new-file> <remotebox>:<fake-new-file>; remote: push to CDN) but I do this shit for living, your mileage may vary :)

And I had to use cygwin

cygwin IS the way to use rsync in windows, the for-profit project cwsync (or how it's called?) is not a good port

but then I had to .zip the folder

rsyncing a folder of files is more effective, but I don't think you would benefit from this - much more stuff to do to create the release archive.

oh, and if you're already using cygwin: learn a little bit of bash scripting, place ssh keys w/o password everywhere and automate the whole thing :D

a script could look like this:

Code: Select all

#!/bin/bash
if [ "$1" == "" -o "$2" == "" ] ; then
  echo "usage: $0 <old archive> <new archive> (e.g. $0 SOTW.0.24.14.zip SOTW.0.24.15.zip)"
  exit 99
fi
ssh <user>@<remotebox> "cp /path/to/download/directory/$1 /path/to/download/directory/$2"
rsync -av $2 <user>@<remotebox>:/path/to/download/directory/$2
ssh <user>@<remotebox> "scp /path/to/download/directory/$2 <user>@<cdnuploadserver>:/path/to/upload/folder/"

[jack1974]

Thanks quite useful, I think my mistake was indeed to try rsync the "unzipped archive" while is probably better to use it to check the differences between the two .zips as in your example.
And I know a bit of linux I have made some simple shell scripts to upload files with FTP and such, but the one above would be more useful. Actually I didn't explain well, the CDN grabs the files from my server automatically, so I don't even need the last line.

Will make some tests with the new updates, thanks for now!

[renke_]

you're welcome. I forgot one exit and a single colon, but I'm sure you will notice it ;)

It seems rsync is much more effective when you sync the directory tree instead of the archive - here are some ideas for creating the zip file on your webserver.

Assumption: Your IDE creates a folder called like your ZIP archives (SOTW-<version>-all)

Code: Select all

#!/bin/bash
if [ "$1" == "" ] ; then
  echo "usage: $0 <source dir>"
  echo "e.g. $0 SOTW-0.8.24.15-all"
  exit 99
fi
RELEASE=`basename $1` # only the name, no path separators 
rsync -av --delete --progress $RELEASE/* <user>@<remotebox>:/path/to/download/SOTW-rsynctarget/ # the target dir is used for all releases
ssh <user>@<remotebox> "cd /path/to/download/ ; ln -s SOTW-rsynctarget $RELEASE" # path in zip must be identical to source
ssh <user>@<remotebox> "cd /path/to/download/ ; zip -r $RELEASE.zip $RELEASE/" # zip creation with root path according to source name
ssh <user>@<remotebox> "rm /path/to/download/$RELEASE" # delete the symbolic link

This does only work if you call the script from the parent directory of the release folder ;)

The idea is to use on the remote box only one folder for every release, updated by rsync with the --delete option (quite important, normally rsync never deletes files, even if they are not available anymore in the source). As the zip cmd tool is rather unflexible with folder names a symbolic link is used to fake the correct directory.

[jack1974]

Cool will make some tests with it. When I tried the first script before it was slow but because the closest server doesn't let me use rsync for some reason (is a shared one so probably has some limits). So I had to use the main one in USA which is slower. But if works will get another one to use this system

[renke_]

I fooled around with bsdiff (a tool for creating binary diffs) - forget it, the performance is abysmal :D

You could propably streamline your upload process even more if your remote box supports python: Place parts of the renpy SDK on the server and create the archive.rpa on the server - no more copying the whole blob because of one added icon :)

[jack1974]

I don't think the SDK works in command line, but I tried a few time ago using VNC / TightVNC to connect to a linux box and if I remember well was working. Problem is that I don't want to install that on my server for security reasons, and getting a secondary VPS just for that is a bit silly (now I use it often but there could be months in which I never use it).
For now the rsync system seems good enough!