Phabricator and Supervisor

When Space Station 14 was first getting its sea legs, we had the need for a project management system that could easily handle hundreds of developers, and had ways for the team to easily review changes without stumbling all over each other.  We ended up using Phabricator from Phacility, and while its effectiveness is debatable (Arcanist is a pain in the ass to install on Windows, for example), it has many useful features.  Unfortunately, being a new addition to the market, it has some issues that need to be worked around.

The Problem

One of Phabricator’s handy features is the IRC bot.  SS13 and SS14 developers both make heavy use of IRC, and being able to get live updates on project status is crucial when most of your developers are in their own little worlds and need to be highlighted for attention. On top of this, Phabbot can be used to query the repository for the status of issues, diffs (which are basically PRs, but actually more like patches), and even search for class names and other symbols.

Phabricator makes use of background processes (daemons) in order to run tasks regularly and query external git repositories.  The IRC bot is its own daemon, but must be started manually after each update to the code or configuration, since it’s not part of the standard bin/phd start group.  In addition, I’ve been testing nightly builds of HHVM, which replaces PHP, and tends to segfault at inopportune times, particularly when running in the background.  So, relying on phd to manage the daemons was becoming a huge hassle.

Naturally, being a programmer, I took a 5-minute annoyance and turned it into a 24-hour coding and reverse-engineering project.

Enter Supervisor

On my server, I also run Notifico, a replacement for the ill-fated CIA.vc IRC notification service.  It also has to run background tasks and workers, but opted to use an existing system for keeping its tasks running: The wonderous, ass-saving Supervisor daemon.  I’ve since moved our HHVM fastcgi process to it, since it can automatically restart HHVM when it has a problem.  Naturally, I decided to use the same system to run Phabricator’s daemons.

However, there’s a catch:  Supervisor requires that the process stay in the foreground.  If it forks, Supervisor cannot handle it and the process is marked as being in an erroneous state. In addition, there’s a lot of separate processes that must be launched and tracked, and they change extremely often.  At the time of writing, there’s four taskmaster processes, a fetcher, and a filtering agent, in addition to the phab-bot. phd forks and launches subprocesses, which isn’t acceptable for use with Supervisor.  After searching around StackOverflow and other sites, I managed to find this article, but was saddened when the provided configuration was horribly out-of-date:  Phabricator has since refactored the hell out of their daemon API, so the provided configuration is completely useless.

Getting Dirty

Note: This section is just cataloging the crap I went through.  Skip ahead if you don’t care.

The first step was to automate generation of the phabricator.conf I feed to Supervisor.  The configuration with only 6 processes ended up being enormous, and all of these subprocesses needed to be added to a phabricator group so it wasn’t a nightmare to restart everything after an update.  I ended up writing a simple Python script to do this, configured by a simple YAML file.  This ended up being the easy part.

The next stage was figuring out how the hell to get Supervisor to launch a process without hacking it and breaking everything.  Unfortunately, like most opensource projects, Phabricator and libphutil have little to no up-to-date internal documentation, but thankfully, the code is well-organized and easy to browse.  I also ran phd a few times after chucking var_dump() into a few places, particularly to grab the JSON I mention below.

One of the new annoying things that the new daemon API does is feed configuration to the daemon via STDIN.  I suspect this is because some downstream project needed to feed in a buttload of configuration data.  Why they don’t just use a configuration file is beyond me, especially when they do actually write the configuration to a temporary file and then pipe that in via a small passthrough bash script.  But who am I to question Facebook?  In the end, this configuration data is fairly simple JSON that’s piped to scripts/daemons/launch_daemon.php:

{"daemons": [{"class": "PhabricatorBot", "argv": [".../config.json"]}], "piddir": "/var/run/phd/pid"}

So, clearly three major things are in this configuration:  The PHP class of the daemon to run, any arguments we want to pass to it, and which directory to store the PID file in.

Now, here’s the problem:  Supervisor does not like STDIN.  If your process wants something from you, it will lock up the entire supervisord process, requiring you to kill it.  There are no configuration options around this, and no plans to add such a thing, as far as I can tell.

So, like a good programmer, I hacked my way around it:  I added a simple shell script wrapper that takes the configuration file we wrote with the python script as an argument, and then pipes it to launch_daemon.php.  As far as supervisord is concerned, STDIN doesn’t exist.

After testing, everything worked fine.

The Goodies

Now, you’re not here to listen to me yammer on about the work I did.  You just want the good stuff, so here you go.

Simply clone this somewhere and follow the handy README.

WARNING: Keep in mind that Phabricator daemons running under Supervisor are completely unsupported and experimental, so do not run to #phabricator crying if something implodes.