I’m pleased to report that since the release of version 0.8 of Lightning, and the corresponding update of the Google Data Provider add-on everything seems to be working well. I can now update calendar data in both directions.

This, in combination with GooSync, which syncs the Google Calendar to my mobile phone (a Nokia N95), means all my calendars are now easy to keep in sync.

For work I use a Blackberry 8300, and I’m pleased to say there’s an official Google application to sync the calendar.

Phew! Synchronising calendars has been a surprisingly difficult problem to solve, but thankfully we seem to have licked that particular problem now

Unfortunately most of these data were in small files - the largest probably a megabyte or so.

I ran some speed tests. We had gigabit ethernet connecting the 2 boxes on a little used LAN.

I wrote a small script to start 6 copy processes in parallel and another to periodically measure how much disk space had been used on the target machine.

I found that 6 simultaneous scp commands ran at about 15Mb/s (Megabytes per second). scp has the overhead of encrypting the data stream before it is transferred, so I tried plain old rcp as well. That gave me worse results than scp.

Thinking that the large number of small files was slowing things down I tried using tar to batch up the files. I wrote a tiny script to pipe the output of the tar command across to the target machine directly in to a receiving tar process which unbundled the files.

The tar approach almost doubled the rate of transfer to about 30Mb/s.

Here are the tar commands. Note that you’ll need to enable r-commands by creating .rhosts files in the home directories of each machine (remove these after they copy is complete - they are notorious security problems). Note also that, as usual, HP-UX is awkward - whereas the rest of the world uses ‘rsh’ for the remote-shell command, HP-UX uses ‘remsh’. ‘rsh’ is some kind of restricted shell in HP parlance.

box1> cd source_directory; tar cf - . | remsh box2 "cd target_directory; tar xf - "

The first tar command creates a file called ‘-’, which is a special token meaning ’standard output’ in this case. The archive created contains all the files in the current directory (.) plus all subdirectories (tar is recursive by default). This archive file is piped into the remsh command which sends it to the box2 machine. On box 2 I first change to the proper receiving directory, then I extract from ‘-’, or ’standard input’ the incoming files.

I had 6 of these tar commands running simultaneously to ensure the network link was saturated with data, although I suspect that disk access may have been the limiting factor.

Don’t think that because you’re more technical/brighter/don’t really care about the existing data that you’ll be OK.

If you don’t have a back up Murphy’s Law dictates that something disastrous will happen and you’ll lose everything.

If you do have a backup, by the same logic you’ll be fine. Think of it as an insurance policy