Scoping Scoped Storage Extended

11
10
P
N
O TBD
L
JB
HC
M
KK
ICS
F
G
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
Scoped Storage
Scoping
@cafonsomota

View Slide

Android Dev Lead @WITSoftware
Founder @GDGCoimbra and co-founder @Kotlin_Knights
✍ Author @rwenderlich and @opinioesonline
Podcaster wannabe
Loves travel, photography and running
speakerdeck.com/cmota/scoping-scoped-storage-extended
MATERIALS
@cafonsomota
github.com/cmota/ScopedStorage

View Slide

scoped storage
#android

View Slide

10:00
android
this is typically the file
explorer of a brand new
phone

View Slide

this is my file explorer,
that I periodically clean
android
10:00
10:00

View Slide

10:00
10:00
#influencer
android

View Slide

10:00
10:00
what?
android

View Slide

10:00
10:00
another cache?
android

View Slide

10:00
10:00
no longer exists
android

View Slide

10:00
10:00
no idea
android

View Slide

10:00
10:00
sounds fishy
(especially with lower case)
android

View Slide

10:00
10:00
challenge
sounds fishy
(especially with lower case)
share your storage
with the hashtag
#nomorescatteredfiles

View Slide

android

View Slide

why
Scattered files throughout your disk
Files from uninstalled apps are kept in disk using up space
An app can access any file on external storage after the permission is granted
There’s no clear distinction where files should be stored
Some times you use full path for files other Uri’s
You end up doing “hacks” in some file operations

View Slide

you are here
10 11
scoped storage
(can be opted out)
scoped storage
(mandatory)
6
runtime permissions
dark ages 5
install permissions
(free for all)
4.4
SAF released
(Storage Access Framework)
timeline: privacy
7
ﬁle provider
…

View Slide

timeline: 2020
you are here
Dec
target min API level 29+
for app updates
Sep
Aug
Jul
for new apps
Oct Nov
developers can start submitting
their apps for review for
background location access
Jun
May
…

View Slide

timeline: 2021?
you are going to be here
Dec
for app updates
Sep
Aug?
Jul
for new apps
Oct Nov?
Jun
May
…

View Slide

concepts
No longer rely on file paths to access files, but instead Uri’s
Files from uninstalled apps are removed (unless if saved through MediaStore/ SAF)
MediaStore API is going to be your best friend
Easy access to Images, Videos and Audio
For other files use SAF
Files are now secure and private (as they should be)

View Slide

directories
Type Access method Permissions Access Uninstalled
app-specific files
(internal storage)
getFilesDir()
/data/user/…
Not needed
No other app
can access
Data removed
app-specific files
(external storage)
getExternalFilesDir()
/storage/emulated/…
Not needed
No other app
can access*
Data removed
Media MediaStore API
READ_EXTERNAL_STORAGE
WRITE_EXTERNAL_STORAGE (Yes Nothing
Other files SAF None
Through the
system file
picker
Nothing

View Slide

directories
Type Access method Permissions Access Uninstalled
app-specific files
(internal storage)
getFilesDir()
/data/user/…
Not needed
No other app
can access
Data removed
app-specific files
(external storage)
getExternalFilesDir()
/storage/emulated/…
Not needed Yes Data removed
Media MediaStore API
READ_EXTERNAL_STORAGE
WRITE_EXTERNAL_STORAGE (Yes Nothing
Other files SAF None
Through the
system file
picker
Nothing

Select the appropriate folder for your app requirements
Asking for permissions is always suspicious for the user

View Slide

building a gallery
#android

View Slide

Get all media files
Filter by images, videos, starred and trashed items
Create a new file
Update one or more existing files
Delete one or more files
features

View Slide

ANDROID GALLERY APP
adapted from IKEA instructions manual

View Slide

getMedia(Environment.getExternalStorageDirectory().listFiles())

private fun getMedia(files: Array?): List {

if (files #== null) {

return emptyList()

}

val media = mutableListOf()

for (file in files) {

if (file.isDirectory) {

media += getMedia(file.listFiles())

} else {

if (isAnImage(file.extension)) {

media += Media(file.path, file.name, file.length(), file.lastModified())

}

}

}

return media

}

dark ages

View Slide




return emptyList()

}





} else {



}

}

}

return media

}

dark ages
getting all files from disk

View Slide




return emptyList()

}





} else {



}

}

}

return media

}

dark ages

View Slide




return emptyList()

}





} else {



}

}

}

return media

}

deprecated
dark ages

View Slide

dark ages



return emptyList()

}





} else {



}

}

}

return media

}

View Slide




return emptyList()

}





} else {



}

}

}

return media

}

Well… not the best approach out there
Bad performance
Doesn’t take into account that new files can be added/ modified during iteration
Uses deprecated methods
Individually check for files media type
dark ages

View Slide

suspend fun queryImages(): List {

withContext(Dispatchers.IO) {

val projection = arrayOf(

MediaStore.Images.Media._ID,

MediaStore.Images.Media.DATA,

MediaStore.Images.Media.DISPLAY_NAME,

MediaStore.Images.Media.SIZE,

MediaStore.Images.Media.DATE_MODIFIED)

getApplication().contentResolver.query(

MediaStore.Images.Media.EXTERNAL_CONTENT_URI,

projection,

null,

null,

"${MediaStore.Images.Media.DATE_MODIFIED} DESC"

)#?.use { cursor #->

val id = cursor.getLong(cursor.getColumnIndexOrThrow(MediaStore.Images.Media._ID))

val path = cursor.getString(cursor.getColumnIndex(MediaStore.Images.Media.DATA))

…

}

MediaStore API

View Slide











projection,

null,

null,





…

}

MediaStore API
you can select which
columns you want to
access

View Slide











projection,

null,

null,





…

}

MediaStore API

View Slide











projection,

null,

null,





…

}

MediaStore API
You can filter the
results by adding an
instruction, for
instance: “${MediaStore.Images.Media.IS_FAVORITE} = 1”

View Slide











projection,

null,

null,





…

}

MediaStore API

View Slide

deprecated










projection,

null,

null,





…

}

deprecated
MediaStore API

View Slide

MediaStore API










projection,

null,

null,





…

}

View Slide

MediaStore API










projection,

null,

null,





…

}

Files should no longer be accessed via direct paths
Uri’s increase data security

View Slide

building a gallery
ready for scoped
storage
#scopedstorage

View Slide

scoped storage
not possible to disable
requires runtime permissions
target: android 10 target: android 11 target: a
all other android versions
android 6.0+
scoped storage
enabled by default
accessing files

View Slide

android 6.0+
scoped storage
enabled by default
scoped storage
accessing files

View Slide

scoped storage
AndroidManifest.xml

android 6.0+
scoped storage
enabled by default
val permissions = arrayOf(

Manifest.permission.READ_EXTERNAL_STORAGE,

Manifest.permission.WRITE_EXTERNAL_STORAGE)

ActivityCompat.requestPermissions(this, permissions, PERMISSION_STORAGE)
Activity.kt
android 6.0+

View Slide

…

android:requestLegacyExternalStorage="true">

scoped storage
AndroidManifest.xml
android 6.0+
scoped storage
enabled by default
to disable
android 10

View Slide

scoped storage
android 6.0+
scoped storage
enabled by default
build.gradle
android {

compileSdkVersion 30

defaultConfig {

targetSdkVersion 30

…

android 11

View Slide

scoped storage
android 6.0+
scoped storage
enabled by default
although it can be preserved
build.gradle
android {

compileSdkVersion 30

defaultConfig {

targetSdkVersion 30

…

android:maxSdkVersion="28"#/>

…

android:preserveLegacyExternalStorage="true">
AndroidManifest.xml
android 11

View Slide

permissions
Allow Playground to access
Allow Playground to access

View Slide

Using MediaStore API
Filtering for media types
Selecting which attributes we’re looking for
listing files

View Slide

deprecated










projection,

null,

null,





…

}

deprecated
no scoped storage

View Slide











projection,

null,

null,





…

}

no scoped storage

View Slide

scoped storage
listing files










projection,

null,

null,





…

}

View Slide










projection,

null,

null,




val uri = ContentUris.withAppendedId(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, id)

…

}

scoped storage
listing files

View Slide










projection,

null,

null,





…

}

scoped storage
listing files scoped
approved
storage

View Slide










projection,

null,

null,





…

}

scoped storage
approved
storage

What about listing videos?

View Slide




MediaStore.Video.Media._ID,

MediaStore.Video.Media.DISPLAY_NAME,

MediaStore.Video.Media.SIZE,

MediaStore.Video.Media.DATE_MODIFIED)


MediaStore.Video.Media.EXTERNAL_CONTENT_URI,

projection,

null,

null,

"${MediaStore.Video.Media.DATE_MODIFIED} DESC"


val id = cursor.getLong(cursor.getColumnIndexOrThrow(MediaStore.Video.Media._ID))


…

}

scoped storage
approved
storage

View Slide




MediaStore.Video.Media._ID,

MediaStore.Video.Media.DISPLAY_NAME,

MediaStore.Video.Media.SIZE,

MediaStore.Video.Media.DATE_MODIFIED)


MediaStore.Video.Media.EXTERNAL_CONTENT_URI,

projection,

null,

null,

"${MediaStore.Video.Media.DATE_MODIFIED} DESC"


val id = cursor.getLong(cursor.getColumnIndexOrThrow(MediaStore.Video.Media._ID))

val uri = ContentUris.withAppendedId(MediaStore.Video.Media.EXTERNAL_CONTENT_URI, id)

…

}

scoped storage
approved
storage

View Slide

scoped storage
media location
Allows to retrieve image coordinates
The information is stored in the image metadata
Sensitive information
It can be the user home location, for instance

View Slide

val exifInterface = ExifInterface(image.path)

val latLong = exifInterface.latLong

scoped storage
no scoped storage

View Slide

scoped storage
no scoped storage
val exifInterface = ExifInterface(image.path)

val latLong = exifInterface.latLong

Well, remember? No paths.

View Slide

val uri = MediaStore.setRequireOriginal(media[0].uri)

contentResolver.openInputStream(uri).use { stream #->

ExifInterface(stream#!!).run {

val coordinates = latLong#!!.toList()

Log.d("TAG", "Coordinates = (${coordinates[0]}, ${coordinates[0]})")

}

}

scoped storage
media location

View Slide

E/AndroidRuntime: FATAL EXCEPTION: main

Process: com.cmota.playground.storage, PID: 21205

java.lang.UnsupportedOperationException: Caller must hold ACCESS_MEDIA_LOCATION permission to access original

logcat





}

}

scoped storage
media location

View Slide

scoped storage
media location





}

}




logcat

Media location is really sensitive information
Malware apps might get access to the user’s home location, work, etc.
Additional permissions are required for this operation

View Slide

required on
android 10+





}

}

scoped storage
media location

View Slide

required on
android 10+






}

}

Coordinates = (51.9159379, 17.6344794)
logcat
scoped storage
media location

View Slide

View Slide

scoped storage
media location






}

}

Coordinates = (51.9159379, 17.6344794)
logcat

What if we run this code on an Android 11 device?

View Slide






}

}

scoped storage
media location

View Slide






}

}




logcat
scoped storage
media location

View Slide

scoped storage
media location





}

}




logcat

For third-party files you need to request user permission
Security above everything
We’re accessing sensitive information

View Slide

if (ActivityCompat.checkSelfPermission(baseContext,

Manifest.permission.ACCESS_MEDIA_LOCATION) #== PackageManager.PERMISSION_GRANTED) {






}

}

} else {

ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.ACCESS_MEDIA_LOCATION),
PERMISSION_MEDIA_ACCESS)

}

scoped storage
media location

View Slide

if (ActivityCompat.checkSelfPermission(baseContext,

Manifest.permission.ACCESS_MEDIA_LOCATION) #== PackageManager.PERMISSION_GRANTED) {






}

}

} else {

ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.ACCESS_MEDIA_LOCATION),
PERMISSION_MEDIA_ACCESS)

}

scoped
approved
storage
scoped storage
media location

View Slide

scoped storage
creating a new file
Created through the MediaStore API
Can be stored either in PICTURES or DCIM folders

View Slide

scoped storage
no scoped storage
fun duplicate(media: Media) {

viewModelScope.launch {

val file = File(media.path)

val target = File("${file.parent}/${file.nameWithoutExtension}-copy.${file.extension}")

file.copyTo(target, true)

sendScanFileBroadcast(this, target)

}

}

View Slide

scoped storage
no scoped storage






}

}

fun sendScanFileBroadcast(context: Context, file: File) {

val intent = Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE)

intent.data = Uri.fromFile(file)

context.sendBroadcast(intent)

}

View Slide

no scoped storage






}

}





}

No update on MediaStore tables
You could create a file almost anywhere on disk

View Slide

scoped storage
creating new media
suspend fun duplicateMedia(resolver: ContentResolver, media: Media, bitmap: Bitmap) {


val collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY)

val newMedia = ContentValues().apply {

put(MediaStore.Images.Media.DISPLAY_NAME, “${media.name}-cp.${media.extension}")

put(MediaStore.MediaColumns.MIME_TYPE, "image/png")

put(MediaStore.MediaColumns.DATE_MODIFIED, System.currentTimeMillis())

put(MediaStore.Images.Media.IS_PENDING, 1)

}

val newMediaUri = resolver.insert(collection, newMedia)

resolver.openOutputStream(newMediaUri#!!, "w").use {

bitmap.compress(Bitmap.CompressFormat.JPEG, 100, it)

}

newMedia.clear()

newMedia.put(MediaStore.Images.Media.IS_PENDING, 0)

resolver.update(newMediaUri, newMedia, null, null)

}

}

View Slide









}




}

newMedia.clear()

newMedia.put(MediaStore.Images.Media.IS_PENDING, 0)

resolver.update(newMediaUri, newMedia, null, null)

}

}

scoped storage
creating new media

What if we want to save this file on a different location?

View Slide




val dirDest = File(Environment.DIRECTORY_PICTURES, context.getString(R.string.app_name))





put(MediaStore.MediaColumns.RELATIVE_PATH, "$dirDest${File.separator}")


}




}

…
scoped storage
creating new media

View Slide

scoped storage
creating new media



val dirDest = File(Environment.DIRECTORY_PICTURES, context.getString(R.string.app_name))





put(MediaStore.MediaColumns.RELATIVE_PATH, "$dirDest${File.separator}")


}




}

…
scoped
approved
storage
or DIRECTORY_DCIM

View Slide

scoped storage
defining a location
Location selected through system file picker
The user can select where to save the file
And change it’s name

View Slide

scoped storage
specific location
fun openLocationPicker(media: Media) {

val intent = Intent(Intent.ACTION_CREATE_DOCUMENT).apply {

addCategory(Intent.CATEGORY_OPENABLE)

putExtra(Intent.EXTRA_TITLE, media.name)

type = "image#/*"

}

startActivityForResult(intent, PERMISSION_SAVE_ON_LOCATION)

}

View Slide

scoped storage
specific location
fun openLocationPicker(media: Media) {

val intent = Intent(Intent.ACTION_CREATE_DOCUMENT).apply {

addCategory(Intent.CATEGORY_OPENABLE)

putExtra(Intent.EXTRA_TITLE, media.name)

type = "image#/*"

}

startActivityForResult(intent, PERMISSION_SAVE_ON_LOCATION)

}

override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {

if (resultCode #== Activity.RESULT_OK) {

when(requestCode) {

PERMISSION_SAVE_ON_LOCATION #-> duplicateMedia(data#?.data#!!)

…

scoped
approved
storage

View Slide

scoped storage
updating media
The file was created by our app
Request permission to modify third-party files

View Slide

scoped storage
updating media
fun convertToBW(media: Media) {


saveImageBW(getApplication().contentResolver, media, paint)

} catch (securityException: SecurityException) {

if (hasSdkHigherThan(Build.VERSION_CODES.Q)) {

val recoverableSecurityException =

securityException as? RecoverableSecurityException #?: throw securityException

_permissionNeededForUpdate.postValue(

recoverableSecurityException.userAction.actionIntent.intentSender)

} else

throw securityException

}

}

}

View Slide

scoped storage
updating media
fun convertToBW(media: Media) {


saveImageBW(getApplication().contentResolver, media, paint)


if (hasSdkHigherThan(Build.VERSION_CODES.Q)) {



_permissionNeededForUpdate.postValue(

recoverableSecurityException.userAction.actionIntent.intentSender)

} else


}

}

}

scoped
approved
storage
viewModel.permissionNeededForUpdate.observe(this, Observer { sender #->

startIntentSenderForResult(sender, PERMISSION_UPDATE, null, 0, 0, 0, null)

})

View Slide

scoped storage
deleting media
Erase media from disk
Data removed through the MediaStore API
No need to additionally send notification broadcasts

View Slide

suspend fun delete(media: Media) {


File(media.path).delete()

sendScanFileBroadcast(context, target)

}

}

scoped storage
no scoped storage

View Slide





}

scoped storage
no scoped storage




}

}

View Slide





}





}

}

scoped storage
no scoped storage

Manually need to notify other apps that you’ve deleted a file
Might end up with references on MediaStore table from files that no longer exist

View Slide

scoped storage
no scoped storage


File(media.uri.path).delete()


}

}

View Slide

logcat


kotlin.io.NoSuchFileException: /external/images/media/74: The source file doesn't exist.

scoped storage
no scoped storage




}

}

View Slide





}

}

scoped storage
no scoped storage
logcat


kotlin.io.NoSuchFileException: /external/images/media/74: The source file doesn't exist.

Uri is not a direct access to a file
Even if you retrieve the path from Uri it won’t be a exact path on disk
You can only delete files your app creates
Other app files require additional user permission

View Slide

suspend fun delete(resolver: ContentResolver, item: Media): IntentSender? {


try {

val uri = item.uri

resolver.delete(uri, "${MediaStore.Images.Media._ID} = ?", arrayOf(item.id.toString()))


if (hasSdkHigherThan(Build.VERSION_CODES.P)) {



return recoverableSecurityException.userAction.actionIntent.intentSender

} else


}

}

return null

}

scoped storage
scoped storage
deleting media

View Slide



try {

val uri = item.uri







} else


}

}

return null

}

scoped storage
scoped storage
deleting media
viewModel.permissionNeededForDelete.observe(this, Observer { sender #->

startIntentSenderForResult(sender, PERMISSION_DELETE, null, 0, 0, 0, null)

})

scoped
approved
storage

View Slide



try {

val uri = item.uri







} else


}

}

return null

}

scoped storage
scoped storage
deleting media


})

What if we have multiple files to delete?

View Slide

scoped storage
scoped storage
deleting media

View Slide

new in Android 11
scoped storage
#scopedstorage

View Slide

scoped storage
android 11
Changes on ACTION_OPEN_DOCUMENT and ACTION_OPEN_DOCUMENT_TREE

No longer access to root folders of internal storage/sd card
Android/data and Android/obb are also blocked
Downloads/ blocked for ACTION_OPEN_DOCUMENT_TREE
File paths can be used on native libraries
Bulk operations
Starred and trash categories

View Slide

scoped storage
bulk operations
Better UX
One screen shows the media that’s going to be accessed

View Slide

scoped storage
bulk operations
private fun deleteMediaBulk(resolver: ContentResolver, items: List) {

val uris = items.map { it.uri }

_permissionNeededForDelete.postValue(

MediaStore.createDeleteRequest(resolver, uris).intentSender)

}

View Slide

scoped storage
bulk operations
private fun deleteMediaBulk(resolver: ContentResolver, items: List) {


_permissionNeededForDelete.postValue(

MediaStore.createDeleteRequest(resolver, uris).intentSender)

}



})

available on
android 11+
only

View Slide

Allows to define specific media as favorite
Which can be used to give them higher importance
Showing them first, for instance
scoped storage
starred

View Slide

fun addToFavorites(items: List, state: Boolean): PendingIntent {

val resolver = getApplication().contentResolver


return MediaStore.createFavoriteRequest(resolver, uris, state)

}

scoped storage
starred

View Slide

val intent = viewModel.addToFavorites(media, true)

startIntentSenderForResult(intent.intentSender, PERMISSION_FAVORITES, null, 0, 0, 0)

scoped storage
starred
fun addToFavorites(items: List, state: Boolean): PendingIntent {



return MediaStore.createFavoriteRequest(resolver, uris, state)

}

available on
android 11+
only

View Slide

It’s different from delete
Files can be restored during a certain period of time
It will be automatically deleted after 30 days
Similar behaviour of your computer recycler bin
scoped storage
trash

View Slide

fun addToTrash(items: List, state: Boolean): PendingIntent {



return MediaStore.createTrashRequest(resolver, uris, state)

}

scoped storage
trash

View Slide

val intent = viewModel.addToTrash(media, true)

startIntentSenderForResult(intent.intentSender, PERMISSION_TRASH, null, 0, 0, 0)

scoped storage
trash
fun addToTrash(items: List, state: Boolean): PendingIntent {



return MediaStore.createTrashRequest(resolver, uris, state)

}

available on
android 11+
only

View Slide

app exceptions
scoped storage
#scopedstorage

View Slide

scoped storage
exceptions
File managers and backup apps
Need to access the entire file system
The user needs to manually grant this access
Google Play needs to also approve this access
Playground

View Slide

scoped storage
exceptions

fun openSettingsAllFilesAccess(activity: AppCompatActivity) {

val intent = Intent(Settings.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION)

activity.startActivity(intent)

}

scoped
approved
storage

View Slide

scoped storage
limitations
App’s internal and external files are still unavailable
No detailed information about disk usage in-app
No longer possible to clear all apps cache

View Slide

fun openNativeFileExplorer(activity: AppCompatActivity) {

val intent = Intent(Settings.ACTION_MANAGE_STORAGE)


}

scoped storage
disk usage scoped
approved
storage

View Slide

fun clearAppsCacheFiles(activity: AppCompatActivity) {

val intent = Intent(Settings.ACTION_CLEAR_APP_CACHE)


}

scoped storage
clear cache scoped
approved
storage

View Slide

migrating files
scoped storage
#scopedstorage

View Slide

scoped storage
migration
How to migrate data?
Remember: users cannot lose their apps data!
Depends on your app and it’s use cases
You can role out an app update targeting android 10
With requestLegacyStorage enable
Migrate all files to the new directories before targeting android 11
You can use preserveLegacyStorage for app updates
Rethink your app UX

View Slide

final remarks
scoped storage
#scopedstorage

View Slide

scoped storage
TL;DR
Scoped storage is mandatory when targeting Android 11
It can be disabled on Android 10 via requestLegacyStorage
Redesign your app to use the appropriate folders
Don’t forget to migrating files
You can still use preserveLegacyStorage if it’s an app update
Use MediaStore API for accessing images, videos and audio
Files are accessible via Uri’s and not file paths
SAF (Storage Access Framework) should be used for other shareable content
If your app didn’t created that file you’ll need to ask for user permission

View Slide

scoped storage
useful links
Storage updates in Android 11
developer.android.com/preview/privacy/storage
Preparing for Scoped Storage
raywenderlich.com/10217168-preparing-for-scoped-storage
Storage access with Android 11
youtube.com/watch?v=RjyYCUW-9tY
Android 11 Storage FAQ
medium.com/androiddevelopers/android-11-storage-faq-78cefea52b7c
Bringing modern storage to Viber’s users
android-developers.googleblog.com/2020/07/bringing-modern-storage-to-vibers-users.html

View Slide

10
P
N
O TBD
L
JB
HC
M
KK
ICS
F
G
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
Scoped Storage
11
@cafonsomota
github.com/cmota/ScopedStorage

View Slide

Scoping Scoped Storage Extended

Scoping Scoped Storage Extended

cmota

More Decks by cmota

Other Decks in Programming

Featured

Transcript