Technology: Configuring 2500 Tablets in 2 Weeks

In late 2018, I became involved with the bulk configuration process for One Acre Fund's tablets. This was... less than stellar to start. The first year of Mobile Enrollment happened before I joined the organization (late 2016), with a scale of 250 tablets. This was nearly a disaster - over-promised features and late bugs combined with a hard deadline on the field side (the rainy season doesn't wait for code bugs) to create some high-stress days for the developers. In 2017 the code was more stable, but the setup process was still rocky - one of our ITO staff members went through the effort of configuring a different password for every device, but this was too complex for field staff so the field trial team just wrote the device passwords directly on the tablet cases. The core problem of tablet setup was partly due to the relative inexperience in the organization with tablets, partly due to the application code itself. 

Setup in 2017 required:

• Giving the device a password
• Setting up a background image
• Pre-loading an API Key onto the device by navigating to a URL (this was really hard to do manually)
• Setting up an "App Lock" security software - Field Officers found this laughably easy to bypass
• Copying required files onto the device (specific versions of various software)

So, in 2018 I had the bright idea of automating the process. This made use of a lot of my previous android experience - as Android developers will attest, there's a very sophisticated "Android Debug Bridge" maintained by google. I basically wrote a custom C# frontend that could use parallel task threading to run the same Debug commands on a large number of devices all physically connected through USB hubs to a laptop  (nicknamed Hydra because I'm a nerd). Not everything can be automated (such as enabling "Debug" mode), but especially pre-loading an API key by navigating to a long URL is much, much easier in code than when done manually. However, in 2018 this process still encountered a number of issues:

• Spreadsheet-tracking of tablets failed pretty miserably. Field staff tablet tracking went fairly well, but trying to remember which developer had which tablet was a challenge.
• Remembering which tablets were configured and which ones weren't was a huge pain point. Staff members repeatedly factory reset and then reconfigured tablets for reasons I still don't understand, in one case performing seven consecutive configure-reset-configure cycles on the same tablet. This seems to have been due to the office layout - staff members were required to clean up all tablets at the end of the day so other staff could use the office, meaning that they mixed the finished and unfinished tablets in the same box.
• Deployment of new applications wasn't possible except by altering the Enrollment progressive web app - what went into the field stayed in the field until the device was returned to HQ.

Despite these issues, the 2018 trial went well - so well in fact that the organization increased the trial scope to the entire country. In 2019, we were looking at scaling from 250 tablets to 2500 - in order to handle that, we made a few improvements to the process:

• Switching to Google's Admin tools for Mobile Device Management. Tablets were treated as "Company Owned Devices" with access to a Private Play Store, letting us release future applications easily.
• Paper tracking - finished devices had stickers attached to them and explicit manual sign-off sheets at each stage of the process (initial setup, QC, etc.). Tablets were moved forward in boxes, all at the same stage of setup. This paper-and-cardboard-box method worked much better than previous high-tech methods.
• Large numbers of casual staff with a "blitz" approach (many staff members working for a short time in a dedicated space). 
• Group Incentives to hit targets (raffling off one of the decommissioned tablets every day that a "setup target" was hit).

For the 2019 setup, this approach scaled pretty well. The collective group of 30 casual staff was able to setup 240 tablets per day, meeting the target of 2 weeks for complete tablet setup and giving us time to film the process in this video:

Note: this video is unofficial and not released by OAF's communications team, it was created for a Tech Division meetup.

However, by late 2019 we had to yet again change direction. The model of tablet used for field setup had changed; all of the customization work done to setup the first type of tablet would have to be redone to apply to the second tablet model. The ITO department had been able to hire a dedicated tablet lead and code changes within the application had made it possible to configure tablets without manually entering a 20-digit API key; google authentication was now possible because every field staff member had an email address. In something of a zen moment, we decided that the best automation was no automation - ITO was more comfortable with a manual process and the effort to maintain Hydra wasn't worth it. Some things are harder this way (like copying a media folder to each tablet), but the simplicity of the new process makes it possible to do this setup without highly-trained technical staff.

Ultimately there are a lot of lessons to learn here, chief among them:

1. Physical Tracking (aka Sticker Tracking) beats any high-tech system. It's painful to admit this as a developer but a simple sticker labeled "done" was much more effective than the complex "Tablet tracking spreadsheets uploaded to google drive" method I used in 2018.
2. The best code is sometimes no code - another harsh lesson, but I'm glad we didn't maintain the Hydra tool past the end of 2019.
3. Incentives - I was initially concerned in 2019 that Casual staff had a strong incentive to work slowly (slow work = more days worked = more money, regardless of the project delays). I found the group raffle incentive a strong motivator, in general incentives that align the interests of the project with the interests of staff members are a good idea.
4. Dedicated space - a separate physical space for just tablet configuration proved immensely helpful, I don't think this speed would have been possible in a shared office.