Happy 2018! A mixture of things have kept me preoccupied until now, including,
but not limited to, a hand injury stopping me from typing properly, a holiday
period, and of course, my favourite pastime of all, yak shaving (of both
varieties). This post is largely borne from my efforts in yak shaving of a
certain problem, motivated by a desire to get more organized in 2018 -
specifically, the problem of having a synchronized list of things to do.
More specifically, the problem is thus: I use a lot of devices. I have a
desktop computer, a laptop, a microserver, a tablet and a smartphone, and on
top of that, I also have a computer assigned to me in my office at uni. For a
to-do list to be useful, I have to be able to:
- Have the list available no matter which device I have close to hand;
- Be able to modify and delete entries from all these places; and
- Ensure that the entries are kept consistent across all devices.
This, by itself, could be done using some kind of manual synchronization method
(such as Git), but that would be both too much work and (strangely) too easy.
Something as boring as keeping files synchronized across multiple locations is a
job for a computer, not a person. Additionally, I disdain 'cloud' 'services'
with a fiery passion, as they're all basically surveillance capitalism,
whether you pay for them or not, and the inability to know or modify what
they're running also offends my inner engineer and perfectionist. Lastly, I
would much prefer a replication-based approach, because relying on an
Internet connection at all times is asking for trouble. That the solution
needed to both be open source and work from a command line where possible
should go absolutely without saying.
It turned out that all of these criteria can be met, and the result brings
with it a considerable number of additional helpful things. However, setting
everything up is not simple; you have to do it in an oddly specific order, and
the documentation for each of the pieces of puzzle seems strangely unaware of
each of the others, requiring no small amount of guesswork. Thus, this post
is meant to be a guide for anyone who wants to do this all themselves, and
benefit from the (arguably extremely cool) outcomes of all this without
having to go through the same frustrations I did.
The solution I found
The core of this whole bag of tricks is CalDAV - an open standard for
remote storage and access of calendars and contacts. As it is an open standard,
there are a whole bunch of implementations of both servers and clients for it,
which work on pretty much every platform you can think of. The main thing for
me, though, was that it can also synchronize to-do lists, even thought that's
not its primary purpose. The fact that I also get contacts and calendar
synchronization on top of that is just gravy.
CalDAV relies on a client-server model, and so to use it, you need a server. The
one I went with was Radicale, as it's written in Python (so no need to run
a disgusting slurry), is open source, and will happily run on a smaller device.
It also has authentication capabilities on a per-user basis, and lets each user
have multiple contact lists, calendars and to-do lists, so if I ever want to
share the server with others in a secure way, I can. From the point of view of
client programs, I basically only need two: one for my Linux-based machines, and
one for my Android devices. For the former, I decided to go with todoman, as
it's a simple command-line manager of to-do lists, while for the latter, I went
with OpenTasks mostly by process of elimination. Lastly, both clients need a
separate synchronizer program, which for me ended up being vdirsyncer for
todoman and DAVDroid for OpenTasks.
What you will need
Obviously, this is not easy to set up if you haven't self-hosted before, and
you're going to need certain basics in place. In particular, the following are
necessary:
- A static IP address (either via your ISP, a VPS, or something else)
- A machine to act as a server (any of the cheaper ARM-based SBCs will do)
- A domain name
- A web server set up to serve to the domain name you own (I use Nginx)
- A Let's Encrypt certificate for the domain name you own, with appropriate
configuration for your web server and auto-renewal
Setting up all of this is outside the scope of this particular article, although
I may write something for each of these things one day. If you already have a
self-hosted blog with HTTPS, you're all set. I will use Nginx as the web server
throughout this writeup, but if you use something else, it shouldn't be vastly
different.
What to do
For reasons that completely elude me, each of these tasks must be done in
exactly the order I'm presenting them in. If you don't do them in this order,
you'll either end up having to repeat work, or, in the worst case, your stack
won't work and you won't know why. I discovered this order through a lot of
trial and error, so spare yourself the pain and just follow it.
1. Set up Radicale
Installing Radicale is basically the same as setting up any Python package. You
can use pip if that's how you want to do things, or you can check if your
distro packages it. Fortunately for me, Radicale was packaged for Arch Linux (in
the AUR, of course), so this part of the setup was fairly painless.
We also want to make sure that we have authentication support for users. This is
a good policy even if you don't plan to share, as it means that random drive-by
connections to your server won't make it off with your contacts or calendar. To
do this, you'll need a tool called htpasswd, which is packaged with the
Apache server. If you don't have it (and I didn't, because I use Nginx), the
easiest thing to do is just to download it, although you can do without.
Additionally, you want to make sure you have Python's passlib and bcrypt
modules available, so go ahead and install those too. The AUR package for
Radicale even helpfully tells you that they're optional dependencies.
If you are using a distro package, this next step might be optional, depending
on how your package is configured. In order to ensure good privilege
containment, Radicale should run as its own user, with its own directory,
without being able to write to arbitrary places. The AUR package already does
this for you, but if you installed it using pip or through a different
package manager, you may have to do this step yourself. We'll assume that
Radicale's home directory is /home/radicale, and its user is named
radicale for the rest of this writeup.
We need to create a users file which Radicale has access to, which we'll put
in /home/radicale/users. You can create as many users as you want, but
you'll need at least one. The command in my case was:
[koz@banana ~]$ sudo -u radicale htpasswd -B -c /home/radicale/users koz
You'll be prompted to create a password for this new user, then confirm it.
Repeat this as many times as desired.
Next, you need to configure Radicale itself. There's really only two things we
need to do - specify what port Radicale will listen on, and tell it that we'll
be using our brand-new users file for authentication. This is quite
straightforward: edit /etc/radicale/config, and ensure it contains these
lines uncommented:
[server]
hosts = 127.0.0.1:5232
[auth]
type = htpasswd
htpasswd_filename = /home/radicale/users
htpasswd_encyption = bcrypt
Now, all of this bears some explanation before we continue. Radicale normally
runs only locally; it'll be the job of our friendly web server to forward it
external requests, with HTTPS authentication thanks to our Let's Encrypt
certificate. We're using bcrypt for password hashing (hence the -B flag to
htpasswd and the setting in the config file) because that is what you
should do no really. With this, Radicale will nicely manage our contacts,
calendars and (most importantly for us) to-do lists, with authentication so that
our data doesn't get stomped on by randos.
At this stage, it's a good idea to test-start Radicale's server. You can do it
by invoking sudo -u radicale python radicale -f --debug, then checking
localhost at port 5232. If you get a login page, success! If not, check the log
spew you're getting in your terminal to deduce what's up.
Once you have everything working, start the daemon, either manually or using
whatever init system you favour. The AUR package provides a systemd service
file, which works well enough, but needs a few edits to match your directory,
and also to provide it the --debug flag.
2. Set up your web server to serve Radicale
Running your Radicale locally isn't going to get you very far on its own, so now
we need to set up your web server properly. Once again, the examples given here
use Nginx, so if you use some other web server, adjust appropriately.
In your Nginx config, you'll have an entry for your HTTPS server. In that
server block, you want to add something that looks like this:
location /radicale/ {
proxy-pass http://localhost:5232/;
proxy-set-header X-Script-Name /radicale;
proxy-set-header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy-pass-header Authorization;
}
This will allow you to access your Radicale server at
https://your-domain.coffee/radicale/. Restart your web server and check that
you can using an ordinary browser. If all went well, you should be able to get a
login page. Try using the credentials you created previously to log in - you
should be able to do so, but there won't be much there to look at yet.
3. Download and set up OpenTasks
I would suggest using F-Droid for this. Simply install the app, run it, and
give it appropriate permissions. You won't have anything useful there right now,
and thus, this step seems a bit silly, but as you'll see, this is essential.
4. Download and set up DAVDroid
Again, F-Droid is recommended. Once you've installed DAVDroid, you'll need to
point it at your now-accessible Radicale instance. You'll need to select 'Login
with URL and user name', and enter the link you tested with above, followed by
your credentials as generated in the first step.
If successful, you'll land on a screen asking you to create a user. Now, you're
likely a bit confused at this point, because you're not Xzibit, and you've
already created a user, but this is different. In the context of Radicale,
'users' are authentication groups to the server; in the context of DAVDroid,
'users' are individual instances of collections of contacts, calendars and to-do
lists in the same authentication group. Why these two terms were given such
similar names will always be beyond me. Additionally, DAVDroid will not inform
you what 'users' (from the DAVDroid point of view) exist on the 'user' (from the
Radicale point of view) that you've just logged into. It suggests using your
email address for the username, and I strongly suggest doing the same, as it'll
allow you to keep things nice and consistent if you have multiple Android
devices.
After this, you'll be on a screen with a CardDAV section and a CalDAV section.
The former is where you can create and store contact lists, while the latter is
where you can create and store calendars and to-do lists (or combinations of
both if you choose). For myself, I created one of each, but for the purposes of
this write-up, it's sufficient to just create a to-do list with an appropriate
name. Ensure that you synchronize it to the server before you continue. You may
also want to tell DAVDroid to only sync manually, to avoid draining your battery
and needlessly using mobile data when not appropriate.
5. Set up OpenTasks, again
You're probably wondering at the ordering, but unfortunately, it's a known bug
that installing DAVDroid first will break OpenTasks. This fact is hard to find
(in fact, I'm not managing to find out where I found out), but unfortunately,
them's the breaks. Add it to the list of things which make Android weird, I
guess.
Here, you just have to re-open OpenTasks, and delete the local task list which
you no longer need. Task lists are OpenTask's terminology for to-do lists; to
remove the local list, go to 'Displayed lists' in the corner menu and then
follow on from there. After you're done, you should only have the synchronized
task list you created earlier, which is empty.
6. Set up vdirsyncer
You can install vdirsyncer using either pip or your package manager as you
prefer (assuming your distro bundles it). After this, we need to set up its
configuration file, which normally resides in ~/.config/vdirsyncer/config.
The file should look something like this:
[general]
status_path = "~/.vdirsyncer/status/
[pair]
conflict_resolution = "b wins"
a = "todo_local"
b = "todo_dav"
collections = ["from a", "from b"]
metadata = ["color", "displayname"]
[storage todo_local]
type = filesystem
path = "~/documents/todo"
fileext = ".ics"
[storage todo_dav]
type = "caldav"
url = "https://your-domain.coffee"
username = "Auser"
password = "this is a secret"
This bears some explanation too. The way vdirsyncer works is by using 'pairs' of
local and remote documents. In this case, the local document is this machine's
replica of your to-do list (which is going to be stored in
~/documents/todo), and the remote document is the replica living on your
server machine, managed by Radicale. You can have as many of these as you like,
but in our case, this will do. We do this after setting everything up for
DAVDroid because it's the easiest way to do this.
You then need to run vdirsyncer discover pair, followed by vdirsyncer
sync. This will transfer a directory with a name that looks like a hash from
your Radicale instance, which will go into ~/documents/todo. Note what this
is - I will use deadbeef as the name for typing simplicity. From this point
onward, every time you make any changes to your todo list that you want
synchronized, just run vdirsyncer sync and they'll be matched. You can
automate this if you want, but I didn't bother.
7. Set up todoman
Like vdirsyncer, todoman is a Python program, so install it as appropriate. We
need to set it up so that it sends your to-dos to the folder we've set
vdirsyncer to synchronize. For that to work, todoman's configuration file should
look like:
[main]
path = ~/documents/todo/*
date_format = %Y-%m-%d
time_format = %H:%M
default_list = deadbeef
default_due = 1
The other settings are mostly self-explanatory if you're familiar with Python's
format strings. default_due is the default number of hours from the point of
a new to-do item being created that it will be due - adjust as appropriate to
your needs.
With that, run todo from your command line, which shouldn't produce any
output (as our to-do list is currently empty). To create a new item, run todo
new from the command line, and use the TUI provided. Create a test item, and
check that you can see it with todo.
8. Profit!
You are now ready to check synchronization. Try running vdirsyncer sync, and
then when it completes, try refreshing in DAVDroid, then checking OpenTasks. If
your task appears there successfully - well done, you are well on the way to
being more productive. If not, work backwards and check that you've completed
all the steps along the way. You can delete the sample task if you want - you
can do this using todo delete x, where x is the number that appears next
to the task when you run todo.
How to find out more
This has been a pretty 'quick-and-dirty' rundown of how to set all this up.
Obviously, your needs might be very different, and even if they're not, you
might at least want to know what your options are. Here is the relevant
documentation:
For DAVDroid and OpenTasks, there's not much out there, but they're fairly
self-explanatory as long as you install them in the right order.
Deficiencies
Despite this scheme being rather brilliant, there are a few missing pieces:
- todoman is a CLI client. If GUIs are more your thing, you can check out your
options here.
- No current way to set up recurring tasks. todoman devs are working on it,
but I am unsure if OpenTasks devs are.
These are not deal-breakers for me, but you may disagree. As always, think hard,
have fun, and happy to-do listing.