Cloud Storage Support
Work was done by the GSoC student Tkachov during the GSoC 2016 to implement cloud storage support in ScummVM.
Overview
There are two main features of the Cloud: sync saves between the device and cloud storage and download game data from cloud storage. Syncing is automatic, but could be cancelled or forbidden for individual games. Game data downloading is manual. To use these features, one has to "connect" a Storage first.
Connecting a Storage
A new Cloud tab is added to Options dialog, and that's where user could choose the active Storage. It's "<none>" by default, and the following Cloud providers are supported to the moment: Dropbox, OneDrive, Google Drive and Box (API analysis/comparison).
When selecting a non-connected Cloud provider, only "Connect" button is available. It opens a Storage Connection Dialog, where user sees the instructions on how to connect a Storage.
Basically, ScummVM uses REST APIs of the specified Cloud providers. In order to do that, a special application has to be registered in the developers section of these providers. Each application has a KEY and a SECRET strings. Developers can also specify some information about their applications (name, logo, description) and add permitted redirect_uris.
Application should navigate its users to a special link. For example, Box link looks like this:
https://account.box.com/api/oauth2/authorize?response_type=code&client_id=KEY&redirect_uri=https://www.scummvm.org/c/code
So, it's obvious that KEY is not a secret, and even called "id" in some providers. We also specify a redirect_uri there, which is required to be either https:// link, or a http://localhost[:port] one. Response type "code" means that we want to get a code, not the actual access_token.
When users open such link, they have to auth to the provider first. When they do, they'd see some information about the application and would be given a choice to allow or deny that application access to user's storage. There different "scopes", which could help application to specify what they want to get access to (for example, application could be limited to read/write files only in a special directory). Providers usually show which scopes application wants to use, so we shouldn't ask for everything if we need only a few permissions (list of scopes used).
If users allow the application, provider redirects them to the redirect_uri with code passed as GET parameter. This code should be somehow passed to ScummVM, so it could request the access_token needed to use the API.
ScummVM supports two different ways to get that code. The first is using https://www.scummvm.org/c/code page, where the long code is automatically transformed into a few short groups with hashsum added. The Storage Connection Dialog then contains 8 fields, where these short groups should be typed in. It automatically checks the hashsums, so if there is a typo, ScummVM would notify user about it. If no typos are detected, "Connect" button would be enabled. When user presses it, hashsum characters are removed and the passed code used to receive access_token.
The other way is starting a local webserver, which listens on a specific port (12345 to the moment) and awaits HTTP GET request with code passed. When "Allow" button is pressed on provider's site, user is redirected to "http://localhost:12345/?code=CODE", where ScummVM already waits for that code. Webserver is automatically started when Storage Connection Dialog is opened and stopped when it's closed. In this case user has to open a browser on the same device ScummVM is working on.
When application gets the code, it should exchange it for an access_token. Dropbox has an "offline" scope, which makes that access_token last forever. Other providers usually give the token for an hour and provide an additional refresh_token which lasts longer and could be used to get new access_token when the previous is no longer valid.
To get the access_token, application should do a HTTP POST to some special provider's "endpoint". For example, the following OneDrive URL is used:
https://login.live.com/oauth20_token.srf
And that's the content we pass there:
code=CODE&grant_type=authorization_code&client_id=KEY&client_secret=SECRET&redirect_uri=http%3A%2F%2Flocalhost%3A12345%2F
Quite similar content is passed when we use it to refresh token:
refresh_token=REFRESH_TOKEN&grant_type=refresh_token&client_id=KEY&client_secret=SECRET&redirect_uri=http%3A%2F%2Flocalhost%3A12345%2F
Both KEY and SECRET are passed over HTTPS. Yet it means those are kept within application somewhere. To simplify developing, those could be loaded from scummvm.ini, but in ScummVM releases the other approach would be used.
Using Storage API
All Cloud-related methods require making a HTTP request to the provider's REST API (and even more than one request sometimes). As that's not a synchronous operation, all such methods return a Request *, which could be used to control the request. To get the result, one must pass callbacks (one for success and one for failure) to the method, and Request would call one on these when it's complete. More information on Requests system is available on a separate page.
Requests could be retried, and that's useful when access_token suddenly becomes invalid. Storages which has non-durable tokens provide a special TokenRefresher classes, which are used to wrap actual Requests. If Request gets an error which is identified as "token is no more valid" error, TokenRefresher Request automatically refreshes the token and then retries the original Request it wraps.
Some Requests are using the other Requests within. For example, FolderDownloadRequest lists directory with ListDirectoryRequest and then downloads files one by one with DownloadRequest.
While any Requests are running, Storage is considered "working" and a special pulsating cloud icon is shown in the corner of ScummVM window.
Downloading game data
When Storage is connected, two new buttons appear in the Cloud tab: "Refresh" and "Downloads". The first one initiates user information refresh (such as email or used/available bytes) and the second open a special Download Dialog.
Users should select a directory within their remote storage, then choose where that directory should be downloaded into, and wait until it's downloaded. The progress is shown by progress bar and a percentage label. User can also run the download in the background, and use ScummVM freely. An OSD message would notify user when download finishes.
If user doesn't leave the Download Dialog, ScummVM would try to automatically detect the game within the downloaded directory. User can't specify that directory in "Add Game" while it's being downloaded into.
Syncing saves
If Storage is connected, saves sync would be started on ScummVM launch, on each manual and autosave, when a storage is connected and when Save/Load Dialog is opened. If one saves sync is already running, the other would be scheduled after the first one is finished to check whether there are new files to sync.
Loading from files which are being synced is not available. A popup with progress bar is shown in the Save/Load Dialog while ScummVM downloads files from the remote storage. In games with "complex" saves names loading is completely unavailable while files are being downloaded. While ScummVM uploads files, it's safe to use these files already.
User can cancel the sync with a button in that popup.
Saves sync actually syncs all the files within the "ScummVM/Saves" remote directory and "Saved games" local directory (the savespath, that is). Directories are considered flat, so subdirectories are not synced (if there are any). A special "timestamps" local file is used to determine which files should be synced. It contains the remote "modified at" timestamps for each file. If remote file has a newer timestamp or is not listed, it's being downloaded. If local file has a newer timestamp or is not listed (but exists), it's being uploaded. We download files first, upload second. Timestamps are updated after each downloaded and uploaded file, so if sync is interrupted, next sync will continue from the right state.
When file fails to upload, storage providers ignore that there was some uploading session and file is not created or modified. ScummVM stops the sync, and tries to upload that file again on next sync.
When file fails to download, ScummVM deletes the invalid file and stops the sync. As its timestamp is not updated, ScummVM would try to download the file again on next sync.