Building Data Collection Systems for Field Research in Low-Connectivity Environments

Most data collection platforms are built with an implicit assumption: that the device is online. For survey teams working in Nairobi or other major East African cities, this is a reasonable assumption. For teams working in Turkana, Mandera, or the more remote parts of Kenya's arid and semi-arid counties, it's a fiction that causes real project failures — corrupted submissions, lost interviews, and field officers wasting days re-collecting data that should have been saved the first time.

At Welmah, we've built and refined an offline-first data collection architecture over several large-scale field projects, including work across Kenya's northern counties for USAID-funded programmes. This article shares the core design principles behind that system.

The Core Problem: Connectivity Is Not Binary

The naive mental model of "online" versus "offline" doesn't match field reality. In practice, connectivity in remote Kenya exists on a spectrum: no signal at all, intermittent 2G that drops mid-transmission, brief 3G windows near trading centres, and full connectivity only at the county headquarters where the team returns each evening. A robust system has to handle all of these gracefully — not just the two extremes.

Our Architecture

The system we use is built around four core layers that together ensure no data is ever lost, regardless of connectivity conditions:

1. Local-First Storage

Every survey response is written to an encrypted local database (SQLite) on the device the moment it's collected — before any attempt at transmission. The app's primary data source for everything, including reviewing past submissions, is this local store. Network connectivity is never a dependency for the enumerator's core workflow.

2. Append-Only Sync Queue

Rather than attempting to "update" records on a remote server (which creates conflict risk), completed submissions are appended to a sync queue as immutable records. This makes the synchronisation logic dramatically simpler — there's no merge conflict resolution to worry about, since nothing is ever edited remotely, only appended.

3. Differential, Compressed Transmission

When a connectivity window does open — even briefly — every byte matters. We compress payloads aggressively and prioritise the sync queue: oldest submissions first, smaller payloads (text-only forms) before larger ones (forms with photos or GPS traces). This maximises the number of records that successfully transmit during short connectivity windows, even if the connection drops again before everything syncs.

4. Idempotent Server-Side Ingestion

Because devices may retry a submission multiple times if they're unsure whether a previous attempt succeeded, the backend must treat duplicate submissions safely. We use deterministic submission IDs (generated client-side, before any network contact) so the server can silently discard duplicates rather than creating double records.

Handling Media: Photos, GPS and Audio

Photos and GPS traces present a particular challenge — they're large, and field research often requires them (proof of visit, geotagged infrastructure assessments, photographed consent forms). Our approach: store media locally at full resolution always, but transmit a compressed preview first so the record exists on the server even before the full file syncs. Full-resolution media syncs opportunistically whenever a strong connection is available, often when the team returns to a base location with WiFi.

"We stopped losing field days to data loss the moment we moved to this architecture. Enumerators don't even think about connectivity anymore — they just collect data, and it shows up." — Field Coordinator, Marsabit County programme

Lessons From the Field

Beyond the technical architecture, a few practical lessons have shaped how we deploy these systems:

• Train for the sync indicator, not the technology. Enumerators don't need to understand sync queues — they need a simple, visible indicator showing "X records pending upload" so they know to find connectivity before end of day.
• Build in a manual export fallback. For truly remote locations with no connectivity for days, we include a local export-to-USB or Bluetooth-transfer option as a last resort.
• Test on the actual hardware. Budget Android tablets used in field programmes often have weaker radios and less reliable background processes than flagship devices. Always test the real deployment hardware, not a development phone.
• Monitor sync health centrally. A dashboard showing which devices haven't synced in 48+ hours lets supervisors proactively follow up before data loss becomes irreversible.

Choosing the Right Tool

For many projects, established platforms like ODK and KoboToolbox already implement strong offline-first principles and are the right choice — there's no need to build custom software for standard survey data collection. We reach for custom development when projects need: complex conditional logic beyond what XLSForm supports, integration with other systems (DHIS2, custom dashboards), specialised media handling, or workflows that simply don't fit a standard survey paradigm — like asset tracking, multi-stage case management, or real-time alerting.

If you're scoping a field data collection system — whether that means configuring KoboToolbox correctly or building something custom — we'd be glad to help you think through the right approach for your context.