Quest for the perfect file syncing solution

Category: Linux

Attempts at getting ownCloud and Seafile working on a Raspberry Pi

Lately I've been playing with owncloud and Seafile, both open source self-hosted file sync server software. I've used Dropbox for several years and don't feel like trusting them with my data anymore, mainly because a load of files mysteriously went missing at one point (although there were recovered), and of course after their reputation has been tainted since taking on Condoleezza Rice.

My requirements were quite simple:

  1. Must run well on a Raspberry Pi
  2. Have Windows and Linux clients (Android is a bonus)
  3. Be able to sync my entire photo collection which currently occupys 210Gb on my Windows machine

ownCloudownCloud

Version: 6.0.3

After hearing good things on the FLOSS Weekly podcast, I thought I'd check out ownCloud first. There are posts that guide you through installing it on the Pi, which would imply that some people are using it successfully. I never got it to sync my large photo collection, it was just too slow and never finished even after a day. The UI was painfully slow as well, despite doing everything I could to get the maximum performance - installing PHP 5.5 with OpCache and using Nginx instead of Apache. I guess that people using it on a Raspberry Pi don't expect too much more than syncing a relatively small stash of documents across their computers, which it could do just fine without relying on the UI too much.

SeafileSeafile

Version: 3.0.2

Written in Python, the Raspberry Pi's language of choice, is Seafile. I'm happy to say that there were no performance issues here, the UI (both web and client) performed without issue, and the file syncing itself was reasonable. One of the things I liked over ownCloud is that you can create many libraries and choose which ones you sync, instead of having one big library. One downside to Seafile though is that there are no packaged installers for Linux, the installation uses shell scripts and installs to your home directory. Also it was not a straightforward process, mainly because there are a variety of ways to set it up.

Large library sync

After leaving the photo library syncing overnight, I got the 210Gb library uploaded to the Pi, which showed a green checkmark on the client, nice! Things were going very smoothly until I realised that in the seafile data folder, the disk usage was through the roof! It turns out that there is some intensive caching going on during the sync which can consume almost the same size as the library. The "fix" for this is that once the sync has completed, you must restart the client, and it will slowly release the cache.

Not so perfect

Problems soon arose as a result of this caching, on my laptop, running Arch Linux, the sync failed when the disk became full, and no amount of restarting the client after de-syncing the library would free it. My only option was to delete the entire seafile data folder and sync all of the libraries again. 

It's a shame but this type of software is still relatively new in the open source community, Dropbox first came about in 2008, and since then domestic internet speeds have improved greatly making own hosted solutions viable. Both projects are under active development, so I'm hoping that things will improve.

In the meantime, I'll keep using Seafile, but not for my photos.

Honourable Mentions

Some other projects that offer similar file sync functionality but didn't fit the bill for one reason or another.

Sparkleshare

BitTorrent Sync

SpiderOak

git-annex

Wednesday April 30, 2014

comments powered by Disqus