Showing posts with label Introduction. Show all posts
Showing posts with label Introduction. Show all posts

Monday, August 17, 2020

Introduction, Part 3: The Technology of One Acre Fund

The Technology of One Acre Fund reflects the philosophy and business practices of the organization, which shouldn't be much of a surprise. Unfortunately, business practices are relatively easy to change and adapt to a new context; Software is much harder to alter and tends to be built in accretion layers. Personally, I believe that the offline system powering One Acre Fund's field operations (Roster) is a key component of the organization's scaling - to really understand it, a story makes more sense than a technical manual.

The Story of Roster

Roster wasn't planned so much as it grew organically. The core business it supports varies widely by geographic area (every OAF country program runs as a startup with near-total control over business practices), but in general consists of:

  1. An intense, short (3-4 weeks) enrollment period to sign up for a season and order products (intent to buy agreements, not legally-enforceable contracts)
  2. A required prepayment amount/percentage by a certain deadline (usually paid in multiple payments over time as clients have cash, not weekly/monthly terms).
  3. A bulk distribution at planting time, utilizing paper management tools, with clients allowed to change their orders on the day of delivery
  4. A bulk distribution at top dress time (second batch of fertilizer used when plants are almost fully grown)
  5. A period to enroll in Top-Up/AddOn orders (additional products)
  6. Multiple additional bulk distributions for Top-Ups/AddOns or Incentives
  7. Flexible Repayment throughout the season (not relying on weekly/monthly payment terms) until the Final Repayment deadline
  8. Analysis of Repayment progress with soft incentives (T-shirts, farm tools, etc.) to encourage a healthy repayment track

Field Tools

For most of its history Roster was basically a way to generate and data-enter those 4 paper forms; over time it's slowly grown to do more things but those are still the core functionality.

  1. Enrollment form (farmers sign up for a "contract" that's not actually a contract, more like an "intent to buy")
  2. Input Distribution Sheet (which client gets which item off a truck)
  3. Truck Management Sheet (which items go onto which truck)
  4. Client Repayment Progress Report (how much of their loan a client has paid back)

Offline Source-of-Truth


The main difference between Roster and almost any other modern software project is that Roster is explicitly designed to work offline; not "partially online (until you logout and it stops working)", it's meant to operate offline for weeks without an internet connection but still sync data when needed. Other tools and apps offer Offline options, but keep the source-of-truth status as the cloud system; Roster's source-of-truth is the Bookkeeper laptop.

Early One Acre Fund (2006-2010)

The earliest prototypes of One Acre Fund software were created by non-technical staff members using spreadsheets and Microsoft Access to provide inputs to dozens of farmers in relatively isolated communities. During the early days of the company, cell networks in East Africa were practically non-existent; if you visit any one of the current company residential compounds, you will almost certainly find a "Programming in Access" textbook mixed in with the various agricultural and leadership reference material. For several years, successive generations of technical people would join the organization, throw out the existing Access work and start over; the business model was changing so quickly that it made more sense to throw out the previous system and start over than to add features. Unfortunately, this approach is difficult to scale; throwing out all previous work is a controversial topic in the tech world.

Early Scaleability (2010 - 2013)

Starting in 2011, developers decided to continue working on the same Access software rather than throw out the existing work. A small team of developers (with some turnover, but never a team larger than 5 people) started iterating heavily on the Access databases built by non-technical field staff. For a period of about 5 years, the development team had the reputation of being able to quickly and easily add new "hack features" to the software to meet the needs of various trials/business processes. This development team made use of a number of tricks and strategies to "supercharge" Access far beyond the intended uses of the system - I'll provide more technical detail in a separate post, but key among these tricks:

  • Backing MSSQL databases with merge replication - merge replication is a powerful-but-complex offline tool that makes it possible to run separate databases on offline laptops, with only occasional syncing to a cloud-hosted server. Every laptop maintains a separate, isolated copy of just its own data; the cloud system keeps the complete/merged data set from all combined systems. The jargon can be a bit confusing, so it helps to remember:

Replication = Data Copying from place to place

Merge = Separate Subsets that later get combined together

  • Obfuscation and security within Access - encrypted database connection strings and security modifications that prevented unauthorized users from viewing the Access code.
  • Offline data centers - using local networks, multiple laptops could be connected as "assistant" machines to one "Bookkeeper" machine. This makes it possible for One Acre Fund to run medium-scale (10s of laptops) data entry centers in areas that don't have internet connections, then physically move the Bookkeeper machine to an area with a connection to sync it every few days.
  • Heavy reliance on stored procedures and user-defined functions - while SQL code is not as sophisticated as node.js or C#, it provides more functionality than Access VBA script. 
  • Sophisticated SQL methods - complex dynamic SQL was used rather than VBA, including:
    • Dynamic SQL statements stored inside a SQL table, distributed by replication and run on application startup to create local keys/indexes
    • Heavy investments in Indexed Views, Query Tuning and Offline Laptop Performance (a problem as many of the laptops were out-of-date and have poor hardware specifications)
    • SQL queries that dynamically write other SQL queries (Indexed Views) and SQL with embedded email notifications - I plan to write a post on these "SQL beyond what you think SQL can do" methods


However, this decision to keep the existing Access code led to a few drawbacks:
  1. Schema design - the database schema was meant for easy printing of field material, not for performance, storage or limitation of redundant information. A whole set of practices (database normalization) exists to help with designing the best possible schemas for performance and storage; these techniques were not used for the Core Business tables of One Acre Fund.
  2. Audit Trail Weaknesses - running code on a local laptop makes it possible for users to alter the time and date settings, bypassing date checks in the code. Strong authentication of users proved quite difficult to enforce at the field level, leading to shared accounts and authentication done by control of the physical laptop.
  3. Tight coupling of infrastructure and business logic - the original Roster system required physical laptops in each geographic location (district), with the idea that each district would run independently and own its own data. The idea of separate districts is used as the "sharding key" - nearly every database table and software function requires a district identifier. This is causing problems now as countries look at "districtless" logic like brick and mortar stores.

At the time, Cell Networks in East Africa were still quite unreliable; early software updates sometimes required sending a person with a flash drive on a multi-hour bus ride across rural Kenya. The Core Program and business model of One Acre Fund were being rapidly iterated in multiple countries, leading to situations where Off-the-Shelf software would have been too hard to adapt in time to meet the business needs.


Early Cloud Systems (2012-2014)


Expanding cell networks across East Africa made it possible to have centrally hosted servers holding the "merged" main database (instead of laptops in East Africa, use a fully-functional website hosted on a cloud provider called Hetzner). An operations website is built using C#/ASP.NET to handle the aggregated data from all the separate data entry laptops. The operations website is much easier to write code for, deploy and manage. However, it doesn't run offline - so core OAF functionalities are still run on the Bookkeeper laptops.

The early operations website was envisioned as a data warehouse to allow analysis by global/high-level country staff, while day-to-day operations would be handled by individual district machines. Some of the first reports were essentially just database dump operations (like the SeasonClients export, now the most commonly used export with 50,000 downloads in 2019).

Later the ability to add bundle configurations, create inputs and perform other high level configurations were added. The original idea was still to only do central configurations and reporting for districts, but all client-level data entry and data manipulation would be under the control of the district.

A few additional tools and business processes were added to this website between 2015-2017:

  • Transactional MSSQL Replication - this technology makes it possible to maintain realtime read-only copies of the Roster database on multiple cloud servers. The jargon for this feature really means:
Replication = Data Copying from place to place

Transactions = Database Events that all happen together 

  • Bulk Uploaders - Uploaders to the operations website make it possible to do bulk data changes directly to a cloud server, rather than separately on a dozen laptops. This represents a major shift away from the District as the owner and source of all data. More and more business functions start being performed centrally and uploaders allow taking advantage of mobile money rather than collecting money in the field. Uploads are optimized for efficiency (pretty common for 100,000+ line uploads to happen multiple times in a day) .Different countries have different network capability, so highly complex logic around what could happen online and offline was built into the ever-expanding Roster system.
  • Operations website additional tools (2015-present): More and more tools are created in the operations website, such as Government location entry, client search functions, specific distribution planning, etc. Tools that have overlap between the operations website and Access frontend are written as SQL stored procedures to avoid duplicate work. 
  • Solar Home Systems (2015-2017): Solar Home System paygo codes (using Angaza and Biolite, two SHS providers) are attached to the operations website. This makes it possible for clients to order solar home systems and pay the balance over time, with a monthly code entered into the device to keep it functional. If no code is entered, the SHS stops working. Included in this logic was the idea of product warranties (even warranties on parts of products) to handle malfunctioning solar lights.
  • Insurance (2015-2018): Insurance payouts were added to the database as part of the OAF model - especially funeral insurance and crop insurance. 
  • Brick-and-Mortar stores + B2B sales (2015-present): Brick-and-Mortar sales are attempted in multiple countries in order to reach new market segments. Some trials have more success than others, with Kenya's Duka project (OAF-managed brick-and-mortar shops) and Rwanda's P-Shop project (OAF partnerships with existing agrodealers) showing promise.

Separation of Finance from Field (2015-Present)


The philosophical balance that I described previously manifests in our core systems - eventually, the need for standard off-the-shelf Western business software rather than homegrown, customized accounting became obvious. While this is a gross oversimplification, the Western-esque parts of One Acre Fund (Finance, Logistics, Global Purchasing) make use of SAP for off-the-shelf capability of running a global-scale business. The uniquely East African parts of the company that require extreme customization continue to be done using the custom Roster software, with aggregated data from both sets of software combined in a data warehouse. This leads to some bizarrely different levels across different parts of the organization - finance software (SAP and custom-built frontends hosted in Azure) are used almost exclusively by staff based in capital cities with strong internet connections while Access Bookkeeper Laptops are still necessary for use in field locations.

Expanding Cell Networks (2015-2020)


Expanding Cell Networks in East Africa make a number of innovations possible:

  • Mobile Money (2015-present): Mobile money is trialed + added in Kenya - this makes it possible for farmers to repay their balance through SMS/USSD menus. Direct safaricom integration as well as two aggregators (Lipisha and Beyonic) start to be used. This provides a much clearer audit trail for incoming payments and much faster payment processing; it is currently expanded to almost all OAF countries.
  • Mobile Enrollment (2016-present): Bookkeeper laptops are designed to facilitate paper contract data entry; at the level of hundreds of thousands of clients, this starts to cause problems (too many paper contracts to enter, even with offline data center tricks). Tablet enrollment is trialed at a scale of ~250 tablets between 2016 and 2018; in 2019 Kenya goes full scale (3000 tablets, one for each FO). Other countries are currently adopting this software; it was originally written as a progressive web app but is now being adapted into a native application.
  • USSD/SMS Menus (2018-present): Telerivet (a third-party SMS/USSD tool) is used to allow clients to check their balance directly + run trials. Later this functionality is expanded into Mobile Money (a cost-savings measure uses Telerivet + Beyonic in combination). During the 2020 COVID pandemic, rapid reaction through USSD/SMS has allowed the organization to enroll more than 700,000 farmers through a USSD menu. 
  • Field Support (2018-present): Essentially a digitized Client Repayment Progress Report. Field Support was a developer-driven project to rethink the OAF backend using a new offline-capable backend technology (CouchBase). Implemented in a different platform than other Roster tools (Linux/Docker/CBase/node.js) as a caching layer between Roster and tablet users. 
  • Axe Access (2019-present): This is a much-needed push to replace all access tools with operations website tools. In several locations (especially Kenya) offline functionality isn't required; in just 10 years previously-offline locations now have high-speed internet connections and cell networks, so hard-to-maintain offline tools aren't required. Offline capability makes it difficult to release changes and debug issues, so online tools are simpler to iterate and can make use of cutting-edge tools unavailable offline. 
  • Roster Evolution (2019-present): This is a transformative vision to replace/refactor Roster system into something new and better. While the logistics of transforming a $100 Million piece of software can be daunting, the next generation of mobile-optimized tools provide exciting potential to more effectively serve a larger number of clients.

Where We Are Now:


As I write this (August 2020), One Acre Fund is undergoing a transition period. We still have Access databases in active use, but are phasing this out in favor of tablets and USSD/SMS-based tools. We're actively transforming our server hosting processes and transitioning from manually-configured Windows servers to Containerized Linux Microservices. While the Coronavirus pandemic has been an international tragedy, it has also forced the organization to adapt far faster than we thought possible, spurring new innovations in One Acre Fund's technology. 

My Part in this Story:


I started at One Acre Fund in January of 2017; at the time we were doing regular Access deployments, expanding Mobile Money, doing early trials of Mobile Enrollment and starting to look at USSD/SMS. My role was formally "Development Operations Engineer", now "Database Administrator" - informally I see my role as "make everyone else's job easier". I basically do support engineering - I don't provide value directly to the business, I provide value to the other software developers so that they can work faster and the team can accomplish more. This is a difficult role for a number of reasons - I need to understand what everyone else is doing in enough detail to fit all the pieces together, fix them when they break, provide production support to deploy them into the real world and help identify/fix scaling issues that occur when you have millions of clients. Throughout my time at One Acre Fund, I've taken on a number of support-type projects:
  1. Reinitialization Cleanup (2018): Reinitialization (the process of refreshing the full database on all offline Bookkeeper laptops) is a cumbersome process involving cross-country coordination; every BK laptop must have an internet connection at the same time. Cleanup of reinitialization code to run automated scripts rather than require manual action used in 2018; this was successful. This was a massive exercise in coordination in 2020 involving 20 cloud servers, 139 offline laptops, 10 countries and two production support personnel (myself and our sysadmin).
  2. Offline Deployments - deploying code updates to an offline system is not an easy task.
  3. Tablet Configuration - I basically ran a factory in Kenya for two weeks to configure 3000 mobile devices
  4. Production-Grade Security (SSL certificates, database obfuscation, VPNs and forensic analysis)
  5. AlwaysOn Clustering - providing immediate redundancy if our servers go offline
  6. Test Servers/Ephemeral Servers - giving other developers a complete, isolated test environment
  7. Bookkeeper Offline Tools - maintenance of the custom frontends that One Acre Fund has built to allow Merge Replicated-laptops to run throughout East Africa

Finally, through a combination of staff turnover and developer specialization, I'm now left as the only  developer with production credentials at One Acre Fund. While other developers understand subsets of the software in more detail than I do (especially specifics of the business logic), I'm the only person at One Acre Fund who is able to understand and make changes to the entire live structure (Access, MSSQL, C#, Angular, node.js, docker, couchbase and soon kubernetes + react). That's a situation I'm actively trying to change by training other developers and writing documents like this one. Along the way, I've acquired strong opinions about the things that One Acre Fund's technology has done well and things that we could definitely stand to improve; this blog is meant to capture both my specific technical experiences and the philosophy that I've acquired during my time here.


Introduction, Part 2: The Philosophy of One Acre Fund

 Note: these comments are just my opinions and are not official statements of One Acre Fund.


So... what does One Acre Fund actually do?


This is a deceptively difficult question to answer – I would know, I've been asking it for the last year. One Acre Fund is one of the world's largest providers of agricultural microloans with over a million clients and $100 million in annual cash-flow. If you ask leadership what OAF does, you'll get the donor-friendly answer:

“OAF puts farmers first!”

When you dig a bit deeper, you'll get the answer:

“OAF provides agricultural inputs on credit with flexible repayment”

When you dig a bit deeper than that, you'll get wildly different answers depending on the person you're talking to (“we distribute with trucks” or “we use group-based repayments” or “we have committed field teams with high-quality farmer training” or “we use human-centric design innovations”). All of these things are individually true, but they still miss the bigger picture. Every organization has committed staff members; most NGOs use some variant of human-centric design and you'll be hard-pressed to find a company that doesn't claim status as an innovator. So what is the bigger picture?

As part of transitioning into my current role, I had to spend a lot of time learning OAF's current business model. For most of my first 2 years with OAF, I actively avoided it – the terminology is confusing and the business is not easy to understand, so I focused on technical infrastructure. When my former boss was preparing to leave, focusing on just the architecture was no longer an option... so I spent several months studying the core business model of One Acre Fund. Here's what I found out are (in my personal opinion) the reasons why OAF grew while so many past attempts failed:

Starting in the Field

This is one of my favorite parts about OAF – it started in the field. Not “our researchers are in the field, but they'll be back by lunch” - OAF staff members have lived and worked in rural communities since the organization's founding. When I started at the organization in January 2017, the IT Development team worked on an outside patio and routinely had the neighbor's chickens walking through our software planning meetings. Some of the things that seemed like productivity killers at first (deafening rain on a tin roof blocking a call, intermittent cell connections and electricity, unpredictable delays in transportation) are actually beneficial because they make it obvious when you're designing a solution that won't work ("maybe requiring a 10 gigabyte download isn't the best idea here"). While it's true that we would have been able to work faster in an American or European office building, the solutions we designed wouldn't have worked nearly as well in rural East Africa. Working in an office building makes it much easier to answer the question

"What Can We Do?" 

but only working in the field makes it possible to answer the question

"What Should We Do?"

Respect Your Customer

During my onboarding week, I casually mentioned “the farmers we're helping”... and got a firm reprimand from a Kenyan colleague who was in my onboarding cohort. “We're not helping them – they're our clients!”. This is a really big mental shift from what I was accustomed to in the West, but a powerful one. When one person is the "Giver of Aid" and the other the "Receiver of Aid", all sorts of dynamics (even subconsciously ones) create an implicit power differential.With clients – even clients who sometimes don't pay back the loan – the power dynamic is reversed and equalized. We are not the bosses of farmers in East Africa - they are our bosses, if we're designing solutions that aren't accepted by farmers then we need to change our strategy. This mentality is another part of the company's DNA that made the company a success.

Work Offline

This is where I come in – One Acre Fund has by far the most sophisticated offline systems I've ever encountered. They're quite difficult to maintain (I'll write a separate post on that), but the systems grew up in East Africa alongside the company – our core data entry system could keep working even if East Africa had a total two-week-long internet shutdown. Now that the cell network architecture in East Africa is rapidly expanding, this offline capability may not be as necessary as it was in the past – but the strength of our offline systems was one of the major differentiators between OAF and all of its failed competitors.

Balance of Philosophies

In my personal opinion, the single largest factor behind One Acre Fund's growth has been its ability to balance between two incompatible worldviews and philosophies. One of the biggest misconceptions Westerners hold about Africa is thinking of it as a monolith; the real Africa is wildly variable, with different languages/crops/customs/taboos in every geographic region (Kenya alone has 70 distinct ethnic groups). African cities can be nearly indistinguishable from Western ones, while rural areas can be so variable that they defy easy classification. The view from the field tends to be highly localized and does not necessarily translate to an area even 50 kilometers away. While this is a broad generalization and individual people vary a lot, my experience has been that people living in East African rural areas have a very different worldview than people from Western countries or African cities - rural residents tend to be much more concerned with concrete, real, tangible objects than with intangibles and abstractions. In order to create real impact for our clients, we have to understand their concrete needs and adapt the business to match - but such a business will only apply to a region smaller than the average county in America.

This is where the Western philosophy comes in - abstractions, interchangeable parts and economies of scale. Western practices make it possible to run a business that works in many different areas, to bulk order vast quantities of farming inputs and distribute them at low cost. Note that "Western" business practices don't necessarily mean Western people - any business course in Nairobi or Kampala will teach essentially this same school of thought.

One Acre Fund thrives by embracing the two incompatible ideologies - bulk ordering millions of kilos of fertilizer and then selling it with business practices that vary by geographic area. Taken too far, either philosophy would destroy the company; a fully Western business would become too rigid to meet the needs of customers (scale without impact), while a fully Field-based business would be unable to work outside of a tiny geographic area (impact without scale). Only by balancing between the two worldviews can the company provide valuable services for over a million farmers in 10 countries.


Specific Business Practices

This combination of offline systems, field-led systems design and farmers-first mentality led to a set of business practices that is... nonstandard. To be blunt, they make very little business sense from a Western perspective. However, in local context these practices actually fit in really well with East African communities.

Clients Can Cancel on Us, We Can't Cancel on Them

All throughout our systems, the term “contract” is repeatedly referenced. This causes a lot of confusion for our software developers, because we don't actually use contracts. Clients sign up for orders that are vague “intent to buy” agreements. We send out a truck with the farm inputs available; if a client doesn't show up that day, then they don't pick up their inputs and we don't charge them. However, if the client shows up and we don't have their items available, we'll send a follow-up truck to get them their stuff. This puts us in a very one-sided relationship with our clients, but for good reasons. Life in the villages is much less predictable than life in the West – transportation is unreliable, agriculture depends on factors outside anyone's control and peoples' complex web of social bonds can cause all sorts of unforeseen delays. Charging clients for items they haven't received is not a good way to ingratiate yourself into a community; neither is failing to deliver on a promise.

Years are Meaningless, Seasons are Life-or-Death

Almost all banking and logistics software originates from the Global North; all the Banking systems in the Global North were originally built around Northern agriculture. In the Global North, winter matters; in the Global South, rains matter. Unfortunately for the banking software, rainy seasons do not follow the same patterns as snowy seasons. The reliance on yearly budgets, yearly payment terms and yearly interest is so hard-wired into Western thought that we can't imagine anything else. However, our clients don't think this way. Our clients grow food during the rainy season, sell food after it's harvested and may have some money to pay back their loans at unpredictable times depending on factors beyond anyone's control. One of the business adaptations of OAF was to allow farmers to pay back their loan for a season, connecting loan payment to agricultural cycles rather than forcing farmers to adapt to Northern banking practices.

Declining-Balance Interest is Deceitful

Standard practice in the microloan field is declining-balance interest (interest is included in the loan, the faster you pay back the less you owe). However, many of OAF's clients have only a basic education; complex formulas and hidden charges are not a good way to build client trust. OAF charges fixed fees for all goods and services, no hidden charges. We do still want to encourage fast repayment though, so we offer a wide variety of incentive items (T-Shirts, Machetes, Boots, fixed-fee rate reductions, etc.) to those clients who pay us back quickly. This combination of fixed fees and free stuff has been very popular and goes a long way towards building client trust.

Standard Economic Models Don't Apply

I have my own theories about a more accurate economic model in rural East Africa than “Every man for himself”. I call my theory “Minimal Liquidity” - I don't think anyone at OAF intended to find this effect, but it's pretty well baked into the OAF model now because it seems to work in the field. The theory is that:

1. Many East African communities have essentially socialist economies (at least within tribes)

2. Economic pressure is intense, so people are constantly helping their neighbors at the expense of being able to hold on to income-generating capital

3. The end result is that the community as a whole survives, but individual members can't make the necessary investments to break the cycle of poverty.

While it's not written into OAF's policies, this “Minimal Liquidity” theory informs a lot of key decisions. The biggest decision is when to ship agricultural items – by the theory, if you ship an item too early then it will likely be sold rather than being used as an agricultural input. If your neighbor's kid is sick and you have a bag of fertilizer sitting in your living room that can't be used for 3 months... what kind of neighbor would you be to keep the fertilizer instead of helping?

Client Simplicity is everything – even if that creates operational complexity

OAF does offer agricultural loans, but not in the normal sense. While this varies country by country, one of the factors that helped OAF scale so large in Kenya was “bundling” - what I call “Easy Agriculture Kits”. Green Revolution techniques for getting high yields out of Maize require two types of fertilizer, applied several months apart. The wildly successful “Core Product Line” of OAF in Kenya is an “Easy Agriculture Kit” comprised of:

  1. The first type of fertilizer (planting fertilizer), delivered just before the rainy season
  2. The second type of fertilizer (top dressing), delivered about two months later
  3. A fertilizer scoop to measure both types of fertilizer
  4. Maize Seeds
  5. Training on how to use the fertilizer properly delivered in person in the local language (Kenya alone has 68 local languages and most people are at least tri-lingual)
  6. Scale the kit to the acre of land the client has, rather than the exact weights/amounts (clients order the “2 Acre Easy Maize Kit”, we handle the exact amounts needed for 2 acres)


Easy enough, right? However, some complications with that kit come up – what if you need 10 kilos of fertilizer for 1 acre of maize and our supplier can only give 6 kilo bags? In the states you would just find a different supplier, but in East Africa the supply chains are much harder to stitch together. Similarly, what if you want to convince clients to take a slightly larger loan by giving price breaks (making it slightly cheaper per unit to buy 3 acres than 2.5 acres)? In an area where most clients have two SIM cards and use one phone number for texting but another for calls to save money, these price breaks can influence a lot of decision making. Nonlinear pricing, nonlinear sizes and generally complex backend processing (what if you only need 8 Kilos per Acre for fertilizer type A, but 10 Kilos for type B?) create frustrating complexity for our software developers... but make total sense at the field level.


Putting it all Together

The technology of One Acre Fund is my main area of expertise - the next post talks about what this mixture of offline capability, farmers-first innovation and operational complexity looks like at the systems level.


Introduction, Part 1: A Day in the Life

I wake up to the wind rustling through the banana trees in my yard. Each morning looks alike here - the seasons in Rwanda seem like minor variations on a theme (sometimes rainy, sometimes not). I brush aside the mosquitoes that have settled inside my net, quaff a liter of coffee while admiring the view of  the Congo across Lake Kivu, then return to my home office to plug in for the day. Occasionally I'm interrupted by a fresh avocado falling, an avocado-hungry mongoose scurrying between the slats of my white picket fence (incongruously constructed by a previous resident) or a fist-sized spider scurrying out from beneath a hidden surface in my home office. The days pass in a blur of slack communication, production deployments, occasional outages and other work virtually indistinguishable from what I'd be doing in an American office building. After work I leave the familiarity of the white picket fence to walk through the farming villages that surround us in every direction, practicing my (terrible) Kinyarwanda and making mental notes of what I'm seeing (One Acre Fund-branded T-Shirts, good or bad agriculture practices, the types of cellphones people are using, which houses have electricity and running water).


My name is Louis Racette; I'm 28 years old and for the last 9 months I've been the sole person capable of making changes to a database that coordinates $100 million dollars worth of microloans across 10 countries in East Africa. This blog is part technical (how this system actually works) and part philosophy (what we're doing here).

Cowboys vs. Soldiers

I classify developers into two main camps: Cowboys (aka Hackers) and Soldiers (aka Tweakers) - cowboys are great for startup environments wh...