Articles, Blog

DevBytes: Efficient Data Transfers – Batching, Bundling, and SyncAdapters

December 15, 2019

RETO MEIER: Hi, my name
is Reto Meier. I’m the tech lead for
Google’s Android Developer Relations team. Every time your app initiates
a data connection, no matter how much data it transfers,
you cause a typical 3G wireless radio to continue
drawing power for upwards of 20 seconds. This efficient data transfer’s
DevByte will look at how you can use tools like SyncAdapters
to batch and bundle your data transfers,
effectively phase shifting them to produce a highly
efficient, defragmented network traffic profile
like this one. The underlying philosophy is
to reduce the number of transfer sessions by designing
for a small number of large downloads. We can do that by batching up
all of our delay tolerant transfers and bundling them
together with time sensitive ones, potentially
even preempting planned future transfers. For example, if your app uses
analytics, rather than uploading data as it’s collected
you can bundle the results together queuing them
for later upload when your app performs a time-sensitive
download. Similarly, any time sensitive
transfers should also preempt updates or prefetching
scheduled to occur in the near future. In each case you’re attempting
to mitigate the cost of the transfer either with a delay
tolerant upload, or a planned future download by having them
piggyback on the time sensitive ones. So now let’s look at some
techniques that you can use to implement these best
practices. You can start by creating a
batch queue to offset those delay tolerant uploads. This code snippet shows a simple
queue to which you can add either upload or download
transfers that should occur in the future. Then, whenever you execute a
time sensitive on-demand download, or the next periodic
update or prefetch is initiated, you can also execute
the transfer stored in your queue. This simple approach can be
effective but, implemented naively, it introduces the risk
of transfers being lost if your app is closed before
the queue is cleared. That’s fine if your queue
transfers are only relevant to the current session, but for
things like analytics or user created data, you want
to make sure that you never lose anything. The best solution is using
a content provider. This skeleton implementation
shows the basic query, insert, delete, and update
methods that you would need to implement. A content provider can be backed
by any data store, but most commonly an SQLite
database. Now, rather than implementing a
batch queue, you can insert data into your content
provider. And then when your next transfer
begins, query it for pending transfers and remove
successfully transmitted data from that queue. In this example I’m querying
four queue check-ins, sending them to a server and removing
the successfully completed ones from the queue. I still need a technique to
bundle these delay tolerant transfers for the time sensitive
ones, or possibly also reschedule those transfers
periodically to ensure that they do
eventually happen. Most simply, I can revisit the
earlier approach of creating a series of methods to execute
each transfer and then ensure that each calls the other. A much neater alternative
is to use a SyncAdapter. That’s going to execute each of
our transfers within its on perform sync method. It’s important to note that
each of the methods called should contain its own logic for
determining whether or not it should actually perform an
update, potentially skipping transfer sessions if the radio
hasn’t already been activated by a higher priority method. Using a SyncAdapter to manage
your transfers has a number of advantages, including making it
easy to centralize all of your app’s data transfers in one
place so they all run at the same time. Your data transfers can also
be scheduled with transfers from other apps, all working
towards the goal of reducing the number of times the system
has to switch on the radio. SyncAdapter also implements data
transfer best practices for you, including checking
for network connectivity, retrying downloads when
connectivity returns, and automated batching and time
shifting on requests that occur within the same
time window. Creating a SyncAdapter
is actually pretty straightforward. Start by extending the abstract
threaded SyncAdapter class and implementing your own
data transfer code within the onPerformSync handler
as you can see here. When executing the SyncAdapter,
we’ll spawn a thread to invoke sync
operations, so you don’t need to worry about moving your code
into a background thread. Similarly, multiple executions
will be rejected so you can write your transfer code knowing
that it won’t be run concurrently with itself. In order to run, the SyncAdapter
must be bound to a service, so you also need
to create a service that instantiates your SyncAdapter
and binds to it. SyncAdapters were originally
designed to support synchronization of a
content provider associated with an account. For example, Gmail. As a result, you also need to
associate each SyncAdapter with an account type and a
content provider authority. If you don’t use accounts for
your application, you can create your own mock account
by extending the abstract account authenticator. In this boilerplate, you’ll
notice that I’m overriding the add account method. Here you would typically ask
the user to enter their authentication details
for your service. But instead, you can just create
a new stub account and explicitly add it to the
Account Manager. Take particular note of the
account type you specify. Like the SyncAdapter, the
account authenticator also needs a service from which to
run, so create a new service that binds to your
implementation. Then create a metadata
file for your account authenticator. Note in particular the account
type, whic must match the one which you provided earlier
in your implementation. The SyncAdapter also requires
a content provider. As we described earlier, a
content provider is an effective and efficient way to
store the data that you’re planning to transfer. But if you prefer not to use one
you can create a stubbed out implementation like this. Next, create the metadata file
for the SyncAdapter that will be used to associate a
content provider and account type with it. Note that here I am marking the
SyncAdapter as not user visible, so users won’t be able
to disable syncing from their global settings. Finally, add the new services
and content providers to the application manifest,
associating the SyncAdapter metadata with the SyncAdapter
service, the account authenticator, metadata
with the account authenticator service. With your SyncAdapter complete
you can now start thinking about how to trigger your
synchronizations. And you can configure your
SyncAdapter to run when server data changes, when the data on
the device changes, when the framework sends a network Tcl,
or at predetermined intervals, or even on demand. To be as efficient as possible,
it’s best practice to rely primarily on server
initiated synchronizations, and where possible, avoid
starting syncs as the direct result of a user action. Basically, try and eliminate
the Refresh button. The hardest part of the process
is determining which transfers are delay tolerant and
which will require a sync to be initiated. Now, this flow chart gives an
indication of the most common initiators for possible syncs. And I’ll go through each flow
now demonstrating how to trigger a SyncAdapter
in each case. The best approach is to rely on
server pings notifying your app that there is new
data to download. Using Google Cloud Messaging–
which I’ll explore in more detail in another DevByte– you can notify each installed
instance of your app that it needs to sync with the server. Simply listen for an incoming
GCM message notifying you of new data becoming available,
and call requests syncs specifying the appropriate
account and authority. This eliminates the need for any
client side polling while maintaining constantly
up-to-date client data. Conversely, you may want to
synchronize changes on the client side to the Cloud. Now, this could be anything
from analytics data to new emails, so you want to be
careful when deciding which changes are sufficient to
initiate an actual transfer. If you’re using content
providers, you can configure your SyncAdapter to trigger
whenever the content provider changes by setting the
supports uploading attribute to true. Which happens to
be the default. Now, when this is set, an
upload only sync will be requested whenever your content
provider executes a notify change call with the sync
to network parameter set to true as shown here. Now within the onPerformSync
handler you can detect these upload only syncs by examining
the sync extras upload extra. This can be useful if you
want to rate limit client initiated transfers. You can also configure your
SyncAdapter to sync whenever the platform opens a connection
to keep the TCP/IP connection used for things like
Google Cloud Messaging or Live by setting it to
sync automatically. Now this approach ensures that
your app is regularly updated without having to schedule your
own client side polling. Now that said, the frequency of
network Tcls can lead to a large number of transfers, so
this should only be enabled when your app is in
the foreground. And even then, you may want
to artificially lower the frequency of syncs within your
onPerformSync method. Now, if you want to perform
regular syncs at a known frequency, you can use the
addPeriodicSync method. While you can set any frequency,
this approach is more typically used to ensure
at least one sync happens every 12 to 24 hours when the
app isn’t already running. Typically with client and server
changes initiating most of those updates, particularly
when the app is in the foreground. Now finally, if you need to
initiate a time sensitive transfer, either when the app
launches, when the user browses something that you’ve
not prefetched, or just to prefetch additional data, you
can call requestSync. In cases where an update must
occur immediately, in order to avoid impacting the user
experience, you can force a manual expedited sync by
setting the manual and expedited flags within
the settings bundle. These settings will ignore any
back offs and sync settings that might delay the sync
in attempt to execute it immediately instead. Now you can find all the details
to creating and using SyncAdapters in this Android
training class. So, having learned how to
perform our updates efficiently, other DevBytes will
look at ways to adjust our update frequency, reducing
data payload sizes, and using Google Cloud Messaging
in more detail.

You Might Also Like


  • Reply Android Developers September 11, 2013 at 3:18 pm

    Today's episode of #DevBytes  Efficient Data Transfers: Batching, Bundling, and SyncAdapters, is by @Reto Meier 

    Every time you initiate a connection you cause a typical 3G wireless radio to continue drawing power, for upwards of 20 seconds. Episode 4 of the Efficient Data Transfers series of DevBytes looks at how you can use tools like SyncAdapters to batch and bundle data transfers – effectively phase-shifting them – to produce a highly efficient, defragmented network traffic profile.

    The underlying philosophy is to reduce the number of transfer sessions by designing for a small number of large downloads — batching up all our delay tolerant transfers and bundling them together with time sensitive ones.

    Further Reading and Sample Code:

    Episode 4 of 7

    DevBytes: Efficient Data Transfers – Batching, Bundling, and SyncAdapters


  • Reply Udi Cohen September 12, 2013 at 12:05 pm

    I wrote a blog post about writing your own SyncAdapter, explaining all the steps, and also wrote a full working sample to demonstrate the use of SyncAdapters. I recommend every developer to get familiarized with this feature. Check my post on udinic(.)com

  • Reply Delton Childs September 14, 2013 at 3:49 pm

    Don't do that to me Reto! Ummmm…. that was a lot of stuff dude! You gotta slow this stuff down and break it apart! You can't be doing this like the Micro-Machines guy! But that's cool! This is a cool series!

  • Reply Daniel Frey March 17, 2014 at 2:25 pm

    @Reto Meier Can you go more into implementing the queue you mention in this episode?

  • Reply Android Developers December 19, 2014 at 11:25 pm

    One more battery saving #DevByte from the archives to send you into the weekend. In this video, hear from @Reto Meier on how you can use tools like SyncAdapters to batch and bundle data transfers – effectively phase-shifting them – to produce a highly efficient, defragmented network traffic profile. #perfmatters   

  • Reply Spaci Tron August 4, 2015 at 3:04 pm

    Sorry to contradict you but there is nothing "straightforward" with creating a sync adapter. In fact it's a mess of boilerplate code that can easily drive a developer crazy.

  • Reply Markus A. November 18, 2015 at 7:15 pm

    A quick caveat: At 6:52, the highlighted code section (requestSync) passes "null" as the last argument. I've seen at least one device (Android v4.1.1) where this leads to a fatal exception that kills the app caused by "java.lang.IllegalArgumentException: error unparceling Bundle" at "android.content.ContentResolver.validateSyncExtrasBundle(". Passing "new Bundle()" instead of "null" fixes this.

  • Reply Markus A. November 18, 2015 at 8:07 pm

    When I first heard about SyncAdapters, I was really excited about using them to minimize the user-experience-impact of gathering analytics data. Not only can updates be sent to the server lazily in big batches even after the app is already closed; they can even be sent with minimal extra battery-use by allowing them to be piggy-backed onto other network traffic (7:52)… Awesome! … But unfortunately a couple of important implications are skipped over in this talk, that, to me, make SyncAdapters worse than useless for this purpose:

    – As mentioned, SyncAdapters were initially designed to work with user accounts. While a mock account (see 4:33) resolves this issue, unfortunately the addAccountExplicitly-call requires the AUTHENTICATE_ACCOUNTS permission on devices below API level 23, which is shown to the user as "Create accounts and set passwords". Especially if the app does not itself provide any form of user accounts, requesting a permission to "set passwords" might seem suspicious to potential users. I, personally, would never install an app like this…

    – Also, while the SyncAdapter can be hidden from the System Settings via userVisible=false (see 5:33), the mock account and the stub AccountAuthenticator cannot. So, there will be an entry for the app in the list of accounts, as well as in the "Add account"-dialog. Again, sketchy stuff for an app that doesn't otherwise use accounts. An even bigger issue is that, if the authenticator's addAccount-method isn't implemented right, selecting the app from the "Add account"-dialog might crash the system.

    – For network request piggy-backing, one needs to call ContentResolver.setSyncAutomatically (see 7:52). This requires the WRITE_SYNC_SETTINGS permission… Again something that users might find suspicious for an app that shouldn't have any data to sync…

    – Last, but maybe not least, while setting android:userVisible=false on the SyncAdapter (see 5:33) will prevent the user from specifically disabling syncs for just this app, syncs will still be turned off by the global "auto-sync" setting. On my phone, I don't currently use any other apps that require syncing, so I just always had auto-sync turned off without even realizing it. It took me an hour to figure out that this was the reason why my SyncAdapter wasn't being called when I was testing my code. Likely, I'm not the only user that always has auto-sync turned off (intentionally or not). And for those users, any analytics data will never make it to the server…

    Aside: At first it seemed like I might be able to work around a lot of these issues by providing the stub AccountAuthenticator but never actually adding an account, and then using ContentResolver.notifyChange with syncToNetwork=true instead of setSyncAutomatically, which is supposed to call the SyncAdapter without an account, indicating that all accounts should be synced. But, unfortunately, if there are no accounts that match the SyncAdapter's account type, it simply won't be called at all…

  • Reply Zhuinden May 12, 2018 at 3:50 am

    3:44 HA HA HA HA HA HA no

  • Leave a Reply