Happy 2018! A mixture of things have kept me preoccupied until now, including, but not limited to, a hand injury stopping me from typing properly, a holiday period, and of course, my favourite pastime of all, yak shaving (of both varieties). This post is largely borne from my efforts in yak shaving of a certain problem, motivated by a desire to get more organized in 2018 - specifically, the problem of having a synchronized list of things to do.

More specifically, the problem is thus: I use a lot of devices. I have a desktop computer, a laptop, a microserver, a tablet and a smartphone, and on top of that, I also have a computer assigned to me in my office at uni. For a to-do list to be useful, I have to be able to:

• Have the list available no matter which device I have close to hand;
• Be able to modify and delete entries from all these places; and
• Ensure that the entries are kept consistent across all devices.

This, by itself, could be done using some kind of manual synchronization method (such as Git), but that would be both too much work and (strangely) too easy. Something as boring as keeping files synchronized across multiple locations is a job for a computer, not a person. Additionally, I disdain 'cloud' 'services' with a fiery passion, as they're all basically surveillance capitalism, whether you pay for them or not, and the inability to know or modify what they're running also offends my inner engineer and perfectionist. Lastly, I would much prefer a replication-based approach, because relying on an Internet connection at all times is asking for trouble. That the solution needed to both be open source and work from a command line where possible should go absolutely without saying.

It turned out that all of these criteria can be met, and the result brings with it a considerable number of additional helpful things. However, setting everything up is not simple; you have to do it in an oddly specific order, and the documentation for each of the pieces of puzzle seems strangely unaware of each of the others, requiring no small amount of guesswork. Thus, this post is meant to be a guide for anyone who wants to do this all themselves, and benefit from the (arguably extremely cool) outcomes of all this without having to go through the same frustrations I did.

The solution I found

The core of this whole bag of tricks is CalDAV - an open standard for remote storage and access of calendars and contacts. As it is an open standard, there are a whole bunch of implementations of both servers and clients for it, which work on pretty much every platform you can think of. The main thing for me, though, was that it can also synchronize to-do lists, even thought that's not its primary purpose. The fact that I also get contacts and calendar synchronization on top of that is just gravy.

CalDAV relies on a client-server model, and so to use it, you need a server. The one I went with was Radicale, as it's written in Python (so no need to run a disgusting slurry), is open source, and will happily run on a smaller device. It also has authentication capabilities on a per-user basis, and lets each user have multiple contact lists, calendars and to-do lists, so if I ever want to share the server with others in a secure way, I can. From the point of view of client programs, I basically only need two: one for my Linux-based machines, and one for my Android devices. For the former, I decided to go with todoman, as it's a simple command-line manager of to-do lists, while for the latter, I went with OpenTasks mostly by process of elimination. Lastly, both clients need a separate synchronizer program, which for me ended up being vdirsyncer for todoman and DAVDroid for OpenTasks.

What you will need

Obviously, this is not easy to set up if you haven't self-hosted before, and you're going to need certain basics in place. In particular, the following are necessary:

1. A static IP address (either via your ISP, a VPS, or something else)
2. A machine to act as a server (any of the cheaper ARM-based SBCs will do)
3. A domain name
4. A web server set up to serve to the domain name you own (I use Nginx)
5. A Let's Encrypt certificate for the domain name you own, with appropriate configuration for your web server and auto-renewal

Setting up all of this is outside the scope of this particular article, although I may write something for each of these things one day. If you already have a self-hosted blog with HTTPS, you're all set. I will use Nginx as the web server throughout this writeup, but if you use something else, it shouldn't be vastly different.

What to do

For reasons that completely elude me, each of these tasks must be done in exactly the order I'm presenting them in. If you don't do them in this order, you'll either end up having to repeat work, or, in the worst case, your stack won't work and you won't know why. I discovered this order through a lot of trial and error, so spare yourself the pain and just follow it.

1. Set up Radicale

Installing Radicale is basically the same as setting up any Python package. You can use pip if that's how you want to do things, or you can check if your distro packages it. Fortunately for me, Radicale was packaged for Arch Linux (in the AUR, of course), so this part of the setup was fairly painless.

We also want to make sure that we have authentication support for users. This is a good policy even if you don't plan to share, as it means that random drive-by connections to your server won't make it off with your contacts or calendar. To do this, you'll need a tool called htpasswd, which is packaged with the Apache server. If you don't have it (and I didn't, because I use Nginx), the easiest thing to do is just to download it, although you can do without. Additionally, you want to make sure you have Python's passlib and bcrypt modules available, so go ahead and install those too. The AUR package for Radicale even helpfully tells you that they're optional dependencies.

If you are using a distro package, this next step might be optional, depending on how your package is configured. In order to ensure good privilege containment, Radicale should run as its own user, with its own directory, without being able to write to arbitrary places. The AUR package already does this for you, but if you installed it using pip or through a different package manager, you may have to do this step yourself. We'll assume that Radicale's home directory is /home/radicale, and its user is named radicale for the rest of this writeup.

We need to create a users file which Radicale has access to, which we'll put in /home/radicale/users. You can create as many users as you want, but you'll need at least one. The command in my case was:

[koz@banana ~]$ sudo -u radicale htpasswd -B -c /home/radicale/users koz

You'll be prompted to create a password for this new user, then confirm it. Repeat this as many times as desired.

Next, you need to configure Radicale itself. There's really only two things we need to do - specify what port Radicale will listen on, and tell it that we'll be using our brand-new users file for authentication. This is quite straightforward: edit /etc/radicale/config, and ensure it contains these lines uncommented:

[server]
hosts = 127.0.0.1:5232

[auth]
type = htpasswd
htpasswd_filename = /home/radicale/users
htpasswd_encyption = bcrypt

Now, all of this bears some explanation before we continue. Radicale normally runs only locally; it'll be the job of our friendly web server to forward it external requests, with HTTPS authentication thanks to our Let's Encrypt certificate. We're using bcrypt for password hashing (hence the -B flag to htpasswd and the setting in the config file) because that is what you should do no really. With this, Radicale will nicely manage our contacts, calendars and (most importantly for us) to-do lists, with authentication so that our data doesn't get stomped on by randos.

At this stage, it's a good idea to test-start Radicale's server. You can do it by invoking sudo -u radicale python radicale -f --debug, then checking localhost at port 5232. If you get a login page, success! If not, check the log spew you're getting in your terminal to deduce what's up.

Once you have everything working, start the daemon, either manually or using whatever init system you favour. The AUR package provides a systemd service file, which works well enough, but needs a few edits to match your directory, and also to provide it the --debug flag.

2. Set up your web server to serve Radicale

Running your Radicale locally isn't going to get you very far on its own, so now we need to set up your web server properly. Once again, the examples given here use Nginx, so if you use some other web server, adjust appropriately.

In your Nginx config, you'll have an entry for your HTTPS server. In that server block, you want to add something that looks like this:

location /radicale/ {
  proxy-pass         http://localhost:5232/;
  proxy-set-header   X-Script-Name /radicale;
  proxy-set-header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy-pass-header  Authorization;
}

This will allow you to access your Radicale server at https://your-domain.coffee/radicale/. Restart your web server and check that you can using an ordinary browser. If all went well, you should be able to get a login page. Try using the credentials you created previously to log in - you should be able to do so, but there won't be much there to look at yet.

3. Download and set up OpenTasks

I would suggest using F-Droid for this. Simply install the app, run it, and give it appropriate permissions. You won't have anything useful there right now, and thus, this step seems a bit silly, but as you'll see, this is essential.

4. Download and set up DAVDroid

Again, F-Droid is recommended. Once you've installed DAVDroid, you'll need to point it at your now-accessible Radicale instance. You'll need to select 'Login with URL and user name', and enter the link you tested with above, followed by your credentials as generated in the first step.

If successful, you'll land on a screen asking you to create a user. Now, you're likely a bit confused at this point, because you're not Xzibit, and you've already created a user, but this is different. In the context of Radicale, 'users' are authentication groups to the server; in the context of DAVDroid, 'users' are individual instances of collections of contacts, calendars and to-do lists in the same authentication group. Why these two terms were given such similar names will always be beyond me. Additionally, DAVDroid will not inform you what 'users' (from the DAVDroid point of view) exist on the 'user' (from the Radicale point of view) that you've just logged into. It suggests using your email address for the username, and I strongly suggest doing the same, as it'll allow you to keep things nice and consistent if you have multiple Android devices.

After this, you'll be on a screen with a CardDAV section and a CalDAV section. The former is where you can create and store contact lists, while the latter is where you can create and store calendars and to-do lists (or combinations of both if you choose). For myself, I created one of each, but for the purposes of this write-up, it's sufficient to just create a to-do list with an appropriate name. Ensure that you synchronize it to the server before you continue. You may also want to tell DAVDroid to only sync manually, to avoid draining your battery and needlessly using mobile data when not appropriate.

5. Set up OpenTasks, again

You're probably wondering at the ordering, but unfortunately, it's a known bug that installing DAVDroid first will break OpenTasks. This fact is hard to find (in fact, I'm not managing to find out where I found out), but unfortunately, them's the breaks. Add it to the list of things which make Android weird, I guess.

Here, you just have to re-open OpenTasks, and delete the local task list which you no longer need. Task lists are OpenTask's terminology for to-do lists; to remove the local list, go to 'Displayed lists' in the corner menu and then follow on from there. After you're done, you should only have the synchronized task list you created earlier, which is empty.

6. Set up vdirsyncer

You can install vdirsyncer using either pip or your package manager as you prefer (assuming your distro bundles it). After this, we need to set up its configuration file, which normally resides in ~/.config/vdirsyncer/config. The file should look something like this:

[general]
status_path = "~/.vdirsyncer/status/

[pair]
conflict_resolution = "b wins"
a = "todo_local"
b = "todo_dav"
collections = ["from a", "from b"]
metadata = ["color", "displayname"]

[storage todo_local]
type = filesystem
path = "~/documents/todo"
fileext = ".ics"

[storage todo_dav]
type = "caldav"
url = "https://your-domain.coffee"
username = "Auser"
password = "this is a secret"

This bears some explanation too. The way vdirsyncer works is by using 'pairs' of local and remote documents. In this case, the local document is this machine's replica of your to-do list (which is going to be stored in ~/documents/todo), and the remote document is the replica living on your server machine, managed by Radicale. You can have as many of these as you like, but in our case, this will do. We do this after setting everything up for DAVDroid because it's the easiest way to do this.

You then need to run vdirsyncer discover pair, followed by vdirsyncer sync. This will transfer a directory with a name that looks like a hash from your Radicale instance, which will go into ~/documents/todo. Note what this is - I will use deadbeef as the name for typing simplicity. From this point onward, every time you make any changes to your todo list that you want synchronized, just run vdirsyncer sync and they'll be matched. You can automate this if you want, but I didn't bother.

7. Set up todoman

Like vdirsyncer, todoman is a Python program, so install it as appropriate. We need to set it up so that it sends your to-dos to the folder we've set vdirsyncer to synchronize. For that to work, todoman's configuration file should look like:

[main]
path = ~/documents/todo/*
date_format = %Y-%m-%d
time_format = %H:%M
default_list = deadbeef 
default_due = 1

The other settings are mostly self-explanatory if you're familiar with Python's format strings. default_due is the default number of hours from the point of a new to-do item being created that it will be due - adjust as appropriate to your needs.

With that, run todo from your command line, which shouldn't produce any output (as our to-do list is currently empty). To create a new item, run todo new from the command line, and use the TUI provided. Create a test item, and check that you can see it with todo.

8. Profit!

You are now ready to check synchronization. Try running vdirsyncer sync, and then when it completes, try refreshing in DAVDroid, then checking OpenTasks. If your task appears there successfully - well done, you are well on the way to being more productive. If not, work backwards and check that you've completed all the steps along the way. You can delete the sample task if you want - you can do this using todo delete x, where x is the number that appears next to the task when you run todo.

How to find out more

This has been a pretty 'quick-and-dirty' rundown of how to set all this up. Obviously, your needs might be very different, and even if they're not, you might at least want to know what your options are. Here is the relevant documentation:

• Radicale
• todoman
• vdirsyncer

For DAVDroid and OpenTasks, there's not much out there, but they're fairly self-explanatory as long as you install them in the right order.

Deficiencies

Despite this scheme being rather brilliant, there are a few missing pieces:

• todoman is a CLI client. If GUIs are more your thing, you can check out your options here.
• No current way to set up recurring tasks. todoman devs are working on it, but I am unsure if OpenTasks devs are.

These are not deal-breakers for me, but you may disagree. As always, think hard, have fun, and happy to-do listing.