I am Michael Benedictand
while working with SMS-based applications I’ve noticed an air of
mystery
around the issue of reliability. I hear colleagues say ambiguous things like
“carriers consider SMS to be lower priority than voice”, or “SMS delivery
isn’t guaranteed”.
My personal experience has been that messages are almost always delivered
quickly and correctly, but I’ve heard stories of hours-long delays, corrupt
data, and occasionally messages that never arrive. Since I am working on two
projects that depend on reliable SMS service — one involves field-based data
collection and another employs SMS as a transport layer between computers —
I’m curious about how factors such as network, location, and time of day
impact message transmission.
I found myself in Mwanza,
Tanzanialast
week with two GPRS
modemsand
a local partner who was unenthusiastic about the work I was actually
there to do, so I tried a little experiment. I bought SIM cards and airtime
for three of the major TZ networks, put two at a time in the modems, and
wrote a simple python
scriptusing
Adam Mckaig’s excellent
pygsmlibrary.
The script sent messages in all permutations at an arbitrarily chosen
interval of about 20 minutes. This means that four messages were sent at a
time: each SIM to itself and to the other SIM, repeated about every 20
minutes for periods of up to 27 hours. Outgoing messages noted the origin
and destination networks, send time, and a sequential message number to make
it easier to spot lost messages. Incoming SMS were stamped with the time
received, and the delay in seconds between sending and delivery. Messages
were sent two seconds apart to reduce errors from the modems trying to send
and receive at the same time, and results were recorded in a CSV file.
*a) Airtel and Zantel*
Zantelis
one of the smaller networks in TZ, but I was having technical problems
with my Tigo SIM so started with Zantel and
Airtel.
Airtel was known as Zain until recently when Zain was acquired by Baharti
Airtel,
based in India. The script ran from about 1AM on January 14 to just before
5AM on January 15. Here are the results:
[image: Airtel->Zantel
Results]
*Delivery times and dropped messages between Airtel and Zantel Tanzania.
Click to see scale. Note that the Airtel modem stopped sending and receiving
at about 7AM on Jan 15, and the script hung until I woke up shortly after 8
and reset the modem. I do not know what caused the problem.*
331 messages were sent and 10 (0.3%) were dropped. Delivery times fell into
three bands. Airtel-Airtel (i.e. the Airtel SIM sending to itself) was
easily the fastest, with messages typically delivered in 6 or 7 seconds.
Messages in either direction between Airtel and Zantel took roughly twice as
long, typically between 13 and 16 seconds. One outlier, an Airtel to Zantel
message sent between 2 and 3 AM on Jan 15, took 24 seconds to arrive, and
the only dropped Airtel-Airtel message occurred around the same time.
Delivery times for Zantel-Zantel messages were split unevenly: one cluster
near 6 seconds, another around 15 seconds, and a third around 32 seconds. 9
messages sent from Zantel-Airtel were not received.
Some aspects of this plot seem reasonable. Airtel is a larger company with
wider network coverage, so it’s not too far fetched to imagine that messages
sent from an Airtel SIM to itself would be the fastest to arrive. By that
reasoning I’d expect messages sent between the two networks to take somewhat
longer, and they do. The Zantel-Zantel delivery times are less intuitive. I
don’t know what would account for the three distinct time ranges for
delivery. It is also unexpected to see that 9 Zantel-Airtel messages were
never delivered but no Airtel-Zantel messages experienced this problem.
*b) Airtel and Tigo*
*SMS delivery times between Airtel and Tigo SIMs on January 14, 2011. Click
image to see scale. *Messages marked unreadable appear to have reached the
modems but could not be read, and resulted in an (unfortunately unlogged)
exception in pygsm.*
192 messages were sent and 3 (1.6%) were not readable by the modem. Four
distinct delivery time ranges emerged. I eliminated one outlier from the
plot to make it easier to view: on January 15 at 11:47, a Tigo-Airtel
message took 129 seconds to arrive. I believe that the 3 unreadable
Tigo-Tigo messages reached the modem, but the script raised an exception and
the messages were not read. Unfortunately I was not logging pygsm’s
debugging output so I am marking the missing messages as “unreadable”
instead of dropped. I will have to log output in future runs and see if I
can reproduce the problem. I didn’t expect Tigo-Airtel messages to arrive
slower than Airtel-Tigo. The difference is small but it would be interesting
to know why it’s there.
c) Two Tigo SIMs.
The purpose of having the modems send messages to themselves in the previous
two runs was to test delivery times within a network. It occurred to me that
sending from a SIM to itself may not be equivalent to sending between two
different SIMs on the same network, so I bought a second Tigo SIM:
*SMS delivery times between two Tigo SIMs. Click to scale.*
164 messages were sent and 1 (0.6%) was not received. The data shown
excludes one outlier: the first Tigo A-Tigo B message took 457 seconds to
arrive. So: is SIM to SIM within a network equivalent to sending from a SIM
to itself? *Sometimes.* I don’t know why it varies, but note that the number
for Tigo A is +255716379091 while Tigo B is +255717435798.
In particular, after the 255 country code the numbers have different
prefixes, 716 for A and 717 for B. Typically Tanzanian SIMs on the same
network have the same first 3 digits, and the difference here may indicate
that Tigo’s servers treat the two numbers differently. The Airtel/Tigo data
above used the SIM that is labeled as “Tigo B” here.
*Thoughts and next steps*
Unexpected patterns emerged, including asymmetric delivery times between
networks, that I would like to understand better. The few dropped messages
do not seem to follow a clear pattern, which is a concern for both human and
computer-focused SMS applications where missing messages can cause confusion
and frustration.
It seems possible that having the modems send all 4 messages with only a 2
second delay between sending could be impacting the results. It would be
interesting to revise the script to randomly offset the message send times
by a few minutes and see if that increases reliability.
This is not a scientific experiment since there are variables such as time
of day, day of the week, network traffic, modem location, weather, and
likely many others that are not being controlled for. Still I think the data
gives a representative snapshot of network performance. Overall SMS delivery
was quite good. With the exception of a few of outliers most messages were
delivered in under a minute, and Airtel consistently delivered on-network
SMS in less than 10 seconds.
Occasional dropped messages show that safeguards need to be in place if SMS
is used for critical applications. Some preliminary timing data I took in
Uganda suggests that the networks there are not nearly as reliable. I’m
looking forward to looking more carefully at Ugandan networks when I’m back.
If you like this sort of thing and also have too many modems and not enough
social life, feel free to try out the
scriptyourself.
I’d be interested to see what the results look like in other
countries.
A special thanks to Henry
Corrigan-Gibbsand
Matt
Bergfor
interesting discussions about the timing script and SMS reliability.
*This was posted first as SMSpecially Reliable? Part
1and
is repulblished here with permission
*
.
Get ICTworks 3x a week – enter your email address:
You are not watching this post, click to start
watching
comments|
read
more
[image: Like SMS Testing: Text Message Delivery Time and Reliability in
Tanzania on Facebook]
[image: share on Google
Buzz]
[image: share on
Twitter]
——————————
ICT_Works3,800 followers
4,864 tweets
following 566 peoplefollow
– SMS Testing: Text Message Delivery Time and Reliability in Tanzania
http://t.co/0XUE0Ut #ICT4D4:46AM
– SMS Testing: Text Message Delivery Time and Reliability in Tanzania
http://t.co/ev0rQyo #ICT4D #connectivity #airtel #michaelbenedict4:32AM
– Lots of talk about government limiting social media/internet. @dprogram
tells you what to do if it actually happens http://t.co/wuiW1ZD9:28PM
– Development economics expert @owenbarder talks measuring – and
targeting – effective aid http://t.co/igdAJ8F9:16PM
– Interesting article by @lindsaypoirier on the curriculum vs. free-form
#edutech debate in #ICT4D http://t.co/ujdlkfo9:05PM
Thanks for your interest in ICTworks – an ICT practitioner community.
ICTworks is an initiative of Inveneo
Unsubscribe
pamoma.com@gmail.com from this list.
Our mailing address is:
ICTworks – an initiative of Inveneo
972 Mission Street, 5th Floor
San Francisco, CA 94103
Add us to your address
book
Copyright (C) 2009 ICTworks – an initiative of Inveneo All rights reserved.
Forwardthis
email to a friend
Update your profile
—
Sent from my mobile device