Fixing HHVM “Read Timeout is Reached” Error

I’m working on a very large internal project that involves indexing several 4chan boards, along with content on several other *chans which currently do not have any external archive means. I’ve been using HHVM to keep performance high, as this system handles millions of rows in several tables, and uses a complex database schema for the new tagging system. It’s not quite ready yet, but I am slowly making progress.

Unfortunately, after moving to Bellevue, WA back in August, I was running into a weird problem with the background worker process that does continuous database maintenance and thumbnail generation: After a few minutes, it would just exit.

After scratching my head over this problem for a while, I decided to add some more verbosity to the program, and eventually found out that a particular SELECT LEFT JOIN query was throwing a “Read timeout is reached” error. What puzzled me was that this occurred after only 5 seconds, rather than the standard 30.

I poured over the MariaDB, TokuDB, and HHVM documentation, and I could not find any information on what conditions triggered this error. After speaking with some of the HHVM developers in their IRC channel, I was told this error was coming from the webscale SQL adapter code.  Unfortunately, we lost the trail after getting that far.  What was triggering this error, and what the configuration variables to change were was still a mystery.  I was growing concerned that it was a hardcoded value.

After plenty of experimentation and digging into GitHub issue reports, I finally got the worker running again after adding ini_set('MySQL.ReadTimeout', 62000); to the top of the PHP file, which changes the timeout to 62 seconds. (Yes, my worker is coded in PHP because the PHP API is better developed.)  This result is also achievable by adding -vMySQL.ReadTimeout=62000 to the hhvm command line.

I hope this helps someone else who is tearing what’s left of their hair out.