The following is the chronicles of a man trying to make a small Quest testing farm for use in automated tests of VR software. I’ll be trying to amalgamate knowledge from Meta forums, dying wikis, and other ethereal fountains of knowledge that tend to come and go. Hopefully someone finds this useful. If you do, please leave a comment.
For those that aren’t familiar with me, I am currently Development Lead at Alpha Blend Interactive UG, the company behind ChilloutVR. One of my areas of responsibility is helping with Quality Assurance (QA), which is something our company takes seriously. We want our product to be shipped with as much testing as we can throw at it. However, our new test script is already long as Hell, and I want to automate some tasks so QA don’t have to work as hard, and to reduce some user error by eliminating the user from the equation. While many tasks will likely still require a human to test due to complexity, it’s clear that we need to implement some automated usability tests.
However, there’s a slight problem: While there are many mobile phone testing platforms for Android phones (and billions for PC/Mac/Linux), the choices for testing on Quest, Pico and other standalone VR platforms are slim to none.
After much deliberation and research, I’ve decided to set up an experimental hardware-based automated Software Testing Farm, and, more importantly, document the steps I took here so those following in my footsteps can see what needs to be done.
What is a Software Testing Farm?
A Software Testing Farm (STF) is a method of using a collection of mobile devices to test software. Such tests are usually done to ensure a given app or website’s UI/UX works properly.
While it is possible to test apps within VMs, VMs sometimes have little quirks that don’t appear on actual devices. In addition, using a real device gives a better insight into processor load, power draw, graphics draw time, interactions with other phone software, and other considerations that an Android VM just can’t provide.
It’s surprisingly cheap to set up a traditional testing farm, as well, when one considers that a cellular subscription is not needed, heavily damaged hardware can be used and bought off of eBay for pennies on the dollar, and that the testing hardware is basically just USB 3 hubs and a computer capable of organizing the testing.
The problem with an STF is that there is next to zero documentation or resources on building them, and vendors are very suspicious of them due to rampant abuse of them as click farms.
Oh, and nobody’s ever used them for VR devices, as far as I can tell, so this’ll be a ride.
Requirements
This is a system we’re building, so we need to approach this like any systems design problem and start with the basic underlying questions: What do we need from this thing? What will it do?
For my particular purposes, I need this to be able to run tests on Quest 2, 3, and Pro (OG Quest has been unsupported by Meta for a while now), Pico 3, 4 and maybe other headsets that pop up. For cost effectiveness, we need this to run with minimal actual hardware. This means no controllers (there was a problem with this decision, we’ll discuss that later), and leeway for damaged components like displays.
After analysis, I determined that I needed software capable of running automated tests on Android-based operating systems. Since I will likely be the first person working on an STF with VR devices, I needed the testing software to be fully open-source so I could adapt it for my needs as applicable.
The only software available that met all of these requirements was DeviceFarmer STF (formerly OpenSTF), which was designed for Android phones and tablets, but also has plugins for iOS. It’s also available under the Apache 2.0 license, which is not my favorite, but should be fine for what I’m doing with it.
For testing, I will attempt to use Gamium, which is a Unity testing framework. The documentation is not very good, but the basics seem straightforward.
Setting Up
First, we need a host capable of running DeviceFarmer and its prerequisites. Fortunately, I have a pretty beefy NAS sitting in my office: A 4U Supermicro with 16 cores and 128GB of RAM running Debian. This is probably way, way overkill, but it’s available, so I will use it.
I decided to go with Docker for this installation to reduce the risk of STF or its dependencies exploding my system. To make things easier for me, I installed Portainer, and then loaded in the DeviceFarmer docker-compose.yaml as a stack. If you are following along, check over the docker-compose.yaml first, to make sure the port assignments and whatnot fit your specific scenario. You may need to change port assignments, for instance.
I then bought a $25 unlocked Orbic Q10 from eBay, checked the IMEI to make sure it wasn’t stolen, disabled cellular and made sure no SIM was installed (CSPs will come down on you like a ton of bricks if you overload their systems), and followed the DeviceFarmer tutorial to add it to the system to make sure everything worked.
Quest 2 – Everything is Terrible
To actually begin this grand experiment, I bought a $40 Quest 2 from eBay. I knew going in that it had some damage, as the seller had stated that it was “cracked” but otherwise worked fine and booted up. It did not include controllers, straps, chargers, or anything else other than the headset.
I received the Quest 2 and expected that the lenses were damaged, but upon opening the package, I was delighted to discover that not only were the lenses completely undamaged and unblemished, but the displays were in perfect condition, with no visual artifacts, not even a missing pixel. The only damage were two small cracks on the shell. It also happily took a charge.
Unfortunately, I quickly discovered that my decision not to buy controllers was a mistake: The Quest 2 has an onboarding screen (AKA setup screen) that cannot be bypassed without pairing Quest 2 or Quest Pro controllers.
However, after some research, I discovered an exploit buried deep in the Meta Quest forums:
- Start the Quest 2.
- Open the Meta Quest app on your phone.
- If not already done, setup your Quest 2 via the phone.
- Once done, the Quest 2 will still ask to be paired with a controller.
- Go to the Quest Store on your phone.
- Find and install the Quest Guide. You may have to sit and wait for it to recognize it needs to be installed.
- Launch it from your phone.
- In your Quest 2, you will get kicked out of onboarding and asked to complete another set of tutorials in a menu.
- Using the dot in the center of your screen as a cursor, and the Volume Up button as your click, select and complete the Headset Fitment tutorial.
- Once done, exit the Guide.
- Make sure your Quest is unplugged from your PC so you don’t get put into Quest Link, which requires controllers.
- Go to settings.
- Go to Hand and Body Tracking.
- Enable and complete the tutorial.
- Congrats, you now have a working headset!
You will now need to find a way to enable Debug Mode. Usually, this involves creating and verifying a Quest Developer account, opening the Meta Quest app on your phone, selecting your headset in Devices, and then going to Settings > Advanced Settings, and enabling Debug Mode. Unfortunately, it seems my device doesn’t seem to activate it. I will try again later.
