Roll your own cloud storage

Cloud storage has become pretty common-place. With services like Dropbox, OneDrive, Google Drive, etc. It's a good place to put your stuff across your multiple devices, phones, tablets, laptops, desktops, and more. Most of these services run on other computers, in the cloud, hence the name. This is fine for most people, however it does have some downsides.

Downsides such as

Pricing

For common 2TB storage plans, the pricing is about AU$12-25 per month. Over a year, this is about the cost of an actual 2TB hard drive. It's not that bad but not like a one-off cost in most cases.

Privacy

While many services claim good privacy, some do scan your drive for md5 signatures, or fingerpints of files. Some files have certain recognisable fingerprints that can make them get removed. It is usually very unlikely for the common user, but it is possible.

Security

Being popular services, they are vulnerable to being hacked. If you password has leaked, you may not know it and you may also not know that someone else can log into your files. Using an alternative service, such as Nextcloud, it is a much more obscure target. Even if someone has your password, they might not find your Nextcloud. Of course two factor authentication can resolve this, but even that can be hacked if someone is determined enough.

Proprietary

Why use proprietary closed-source services? Open source alternatives exist for many things. Web browser, office suite, image manipulation, etc. Why not cloud as well? It seems like a natural fit, being less user-interface oriented, where many open source projects have issues.

Data corruption and downtime

Being someone elses computer, you are at the mercy of someone elses ability to maintain their service. I have heard cases where OneDrive has corrupted and deleted users files in the past. And if there is an outage of any kind, there is nothing you can do to resolve it. With your own server, you have the ability to switch it off and on again if it stops working.

My Setup

I am currently using Nextcloud for larger files, and SyncThing for projects with many files. This has been mostly pretty solid over the last 2 years or so. I wanted to use exclusively Nextcloud, but I have had issues with git source code repositories causing all kinds of sync issues, due to having many files that change often. Maybe that issue has been resolved now, but to be safe I'm using SyncThing for my projects, which seems to handle many rapidly changing files a lot better.

What is Nextcloud?

Nextcloud is cloud storage service you can install on your own system which acts a lot like Dropbox, OneDrive, etc in that it syncs your files and photos from your devices. It is open source and you can install it on your own computers at home or in the cloud. It has a web interface and mobile and desktop clients that synchronise your files with a central server.

What is Syncthing?

Syncthing is a unique piece of software that is similar to cloud storage, but it is decentralised. This means that there is no real central server that everything runs off. Each device syncs with other devices, so laptop A can sync with laptop B and server A for example. Like a mesh of connection pointing to different locations. I like syncthing because if my server happens to go down, I can quickly synchronise files to another device. If I have two devices in one location, and my server is far away behind a slow connection over the Internet, it makes sense to have the two devices on the same network sync with each other, it is much faster. SyncThing is also open source and can be installed on Linux, Mac, Windows, and there are also mobile apps available however I do not often need them as I mainly store project files which are used with desktop applications.

Backups

Everyone should have a backup strategy, and the cloud is certainly a simple and easy one. For when you run your own cloud service, if you have a hardware failure, you ought to have had backups. Also, if you want to roll back to previous versions, you want that ability as well.

Offsite backups

Admittedly I have not set up remote backups myself, but I intend to. The well known 3 2 1 backup strategy states you should have one copy off-site, ie a full copy of your data in two permanent locations. Say your home server and another physical location. For my case, I might set up a Raspberry Pi at my parents place.

How to

I won't go into the full details here, there are plenty of resources online. There are however many different ways to set up Nextcloud and Syncthing. My solution is below

• Custom-built server with Intel Core i5 6400, 32GB DDR4 and plenty of storage (including SSDs for content as well as VMs)
• Proxmox virtual server for virtual machines and containers
• I used a virtual machine running the Ubuntu LTS release at the time. I retrospect, I have been using proxmox containers more recntly, and they have been a quicker way to set up new headless servers with just as much functionality
• I installed Nextcloud with snap install nextcloud. You can follow a guide such as https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-nextcloud-on-ubuntu-20-04
• The install guide sets up HTTPS certificates, for encrypted connections. However for my server I have multiple services I run, and I have set up Caddy which is like nginx reverse proxy that also sets up HTTPS certificates. Allowing a very easy way to run multiple HTTPS servers on one Internet address. My Caddyfile is this

  <nextcloud URL> {
      reverse_proxy <local nexcloud IP>:443 {
              transport http {
                      tls_insecure_skip_verify
              }
      }
  }

• For Syncthing, I run it on the same server as Nextcloud. It acts like a central server, but to my other devices it is just like any other machine effectively. I also point Caddy to the Syncthing web address so I can add new devices while I'm away from home.
• As a rolling backup for Syncthing, so I don't lose changes, I run a crontab with entries like this:

  00 4 * * * tar -czvf /home/user/SyncthingArchive/archive_CurrentPersonalProjects_$(date +\%Y_\%m_\%d-\%H_\%M_\%S).tar.gz /home/user/CurrentPersonalProjects && find /home/user/SyncthingArchive/archive_CurrentPersonalProjects_* -mtime +7 -exec rm>

  Which backs up to an archive file every day, and deletes backups older than 7 days.

• Syncthing is a little tedious to set up on new Linux machines, you have to run systemctl enable syncthing@<username> systemctl start syncthing@<username> after installing. But for Windows and Mac it is much simpler.
• After installing Syncthing, you need to go to the web interface on a browser, http://localhost:8384/ or http://localhost:8080 and then you can add devices, and share folders with other devices. Devices should show up on the local network, otherwise you have to copy the devices ID, which is a long set of letters and numbers, easiest to save to a file.
• You of course also need to purchase a domain name and point your server's (static) IP to it with a DNS record.

Conclusion

With Nextcloud for larger files, and Syncthing for smaller, more frequently changing files, this solution has stuck for me. I have used Google Drive with Insync, OneDrive when I was using Windows a lot, pCloud for a bit - as it works well in Linux devices with mounted drives. But Finally settling on my own cloud storage solution has been perfectly usable and stable for a long period of time. It is not super simple to set up, and does require a bit of technical proficiency. One day I might write a guide for absolute beginners, such as installing very quickly on a cloud server (Linode?). However the more information about how to do so gets out there, the more people might start switching over!