"not enough space" while updating maps

lightful commented

2022-12-20 15:19:05 +00:00

(Migrated from github.com)

Description:

The device has nearly 50 GB of free space and the application rejects to update the maps

Steps to reproduce:

Start a maps update
Change the Wifi hotspot on the server from 2.4 to 5GHz (that will cause a connection reset on clients)
Some map files are created with 0 bytes of size (as the Google files app reports)
The app no longer offers to update the maps when starting
Manually selecting "Download ALL" reports: "not enough space - Please delete any unnecessary data"

Expected behaviour:

The maps update is interrupted (this was expected)
The "Download ALL" button should resume and download the remaining map files

Provisional fix:

The user can't delete e.g. the 202216 folder (not even from adb) in a non rooted device
The only recovery is to delete the entire app storage (as when the application was initially installed)

Description: - The device has nearly 50 GB of free space and the application rejects to update the maps Steps to reproduce: - Start a maps update - Change the Wifi hotspot on the server from 2.4 to 5GHz (that will cause a connection reset on clients) - Some map files are created with 0 bytes of size (as the Google files app reports) - The app no longer offers to update the maps when starting - Manually selecting "Download ALL" reports: "not enough space - Please delete any unnecessary data" Expected behaviour: - The maps update is interrupted (this was expected) - The "Download ALL" button should resume and download the remaining map files Provisional fix: - The user can't delete e.g. the 202216 folder (not even from adb) in a non rooted device - The only recovery is to delete the entire app storage (as when the application was initially installed)

👍 1

lightful commented

2022-12-20 15:21:35 +00:00

(Migrated from github.com)

Directory contents while rejecting the maps downloading:
adb_ls.txt

Directory contents while rejecting the maps downloading: [adb_ls.txt](https://github.com/organicmaps/organicmaps/files/10269689/adb_ls.txt)

biodranik commented

2022-12-21 10:09:04 +00:00

(Migrated from github.com)

Wow, that's a serious bug. Can you please enable logs in OM settings, try to reproduce the issue, and share logs with us via "?" -> Report a bug?

lightful commented

2022-12-21 11:25:27 +00:00

(Migrated from github.com)

I have not managed to reproduce it, even using the same way to cause the network failure (switching the hotspot frequency) after a few tries.

However, the previous "prerequisites" are not met. That is, yesterday, I started from already donwloaded maps (outdated) and proceeed to download a new set (updated). Yesterday it there were two maps folders. Since now there aren't new maps for updating now I'm starting from scratch (data cleaned, only a single maps folder).

It could also be that the behaviour is not consistent or always reproducible.

Anyway, yesterday it was not possible to recover the application download function (even forceclosing it, and once restarted, it always rejected to updated the maps due to "no space available"). You could look also on what conditions that message is triggered, because indeed it is not always linked to lack of storage.

When a new maps update becomes available I'll try to reproduce the issue again.

PS: A moment ago I was tempted to manually create empty files to see if that was the cause (its accidental presence), but due to Android permissions that wouldn't be just a suitable way (e.g. files created and owned by adb user can't be deleted, and perhaps also couldn't be written by organicmaps user).

I have not managed to reproduce it, even using the same way to cause the network failure (switching the hotspot frequency) after a few tries. However, the previous "prerequisites" are not met. That is, yesterday, I started from already donwloaded maps (outdated) and proceeed to download a new set (updated). Yesterday it there were two maps folders. Since now there aren't new maps for _updating_ now I'm starting from scratch (data cleaned, only a single maps folder). It could also be that the behaviour is not consistent or always reproducible. Anyway, yesterday it was not possible to recover the application download function (even forceclosing it, and once restarted, it always rejected to updated the maps due to "no space available"). You could look also on what conditions that message is triggered, because indeed it is not always linked to lack of storage. When a new maps update becomes available I'll try to reproduce the issue again. PS: A moment ago I was tempted to manually create empty files to see if that was the cause (its accidental presence), but due to Android permissions that wouldn't be just a suitable way (e.g. files created and owned by adb user can't be deleted, and perhaps also couldn't be written by organicmaps user).

biodranik commented

2022-12-21 12:19:36 +00:00

(Migrated from github.com)

@lightful thanks for the feedback. You can try to reproduce the issue by installing an older version from Github, FDroid (but please backup your bookmarks first, they're removed on uninstall!) or an older beta if you subscribed to it (installs separately).

Regarding permissions, is your device rooted? Did you do something manually with the file system? What is the android version? Did you modify somehow the SD-card?

@lightful thanks for the feedback. You can try to reproduce the issue by installing an older version from Github, FDroid (but please backup your bookmarks first, they're removed on uninstall!) or an older beta if you subscribed to it (installs separately). Regarding permissions, is your device rooted? Did you do something manually with the file system? What is the android version? Did you modify somehow the SD-card?

lightful commented

2022-12-23 00:43:48 +00:00

(Migrated from github.com)

I have managed to reproduce the problem again by downgrading the version.

When upgrading the maps, after the network error, the screen shows a retry icon for each map. I didn't tried here at first, instead I just went back to the previous screen (where all maps pending to download are summarized). The "Download ALL" buttom there reported the wrong error:

However, entering again in the maps detail screen, I could retry downloading one by one (I don't remember if there is also a "downloading all" in such nested screen... I think that I retried them separately). So it seems that there is a recovery way, without resetting the entire storage, which I didn't manage to work out a couple of days ago.

Here are the logs:

logs.zip

Perhaps the problem is that sometimes (at least while in the summary screen shown above) the call to GetWritableStorageStatus() may request a huge (and wrong) amount of data:

Free space check: requested = 0 ; available = 41913384960
Free space check: requested = 55140493 ; available = 41908989952
Free space check: requested = 19252082 ; available = 41908277248
Free space check: requested = 132664421 ; available = 41908281344
Free space check: requested = 45850505 ; available = 41895841792
Free space check: requested = 66071607967 ; available = 41895329792
Free space check: requested = 74788475 ; available = 41895186432
Free space check: requested = 66071607967 ; available = 41842851840
Free space check: requested = 0 ; available = 41902669824
Free space check: requested = 66071607967 ; available = 41902661632

Note that 66071607967 is just by chance... the TOTAL USED space on my device!. So in a device with less than half the space used this bug likely would have not shown up.

I have managed to reproduce the problem again by downgrading the version. When upgrading the maps, after the network error, the screen shows a retry icon for each map. I didn't tried here at first, instead I just went back to the previous screen (where all maps pending to download are summarized). The "Download ALL" buttom there reported the wrong error: ![image](https://user-images.githubusercontent.com/10393152/209247070-a5d49241-1bb9-4278-a26c-a60d02ce59d9.png) However, entering again in the maps detail screen, I could retry downloading one by one (I don't remember if there is also a "downloading all" in such nested screen... I think that I retried them separately). So it seems that there is a recovery way, without resetting the entire storage, which I didn't manage to work out a couple of days ago. Here are the logs: [logs.zip](https://github.com/organicmaps/organicmaps/files/10291133/logs.zip) Perhaps the problem is that sometimes (at least while in the summary screen shown above) the call to GetWritableStorageStatus() may request a huge (and wrong) amount of data: ``` Free space check: requested = 0 ; available = 41913384960 Free space check: requested = 55140493 ; available = 41908989952 Free space check: requested = 19252082 ; available = 41908277248 Free space check: requested = 132664421 ; available = 41908281344 Free space check: requested = 45850505 ; available = 41895841792 Free space check: requested = 66071607967 ; available = 41895329792 Free space check: requested = 74788475 ; available = 41895186432 Free space check: requested = 66071607967 ; available = 41842851840 Free space check: requested = 0 ; available = 41902669824 Free space check: requested = 66071607967 ; available = 41902661632 ``` Note that 66071607967 is just by chance... the TOTAL USED space on my device!. So in a device with less than half the space used this bug likely would have not shown up.

lightful commented

2022-12-23 01:02:23 +00:00

(Migrated from github.com)

PS: No, my device is not rooted. A couple of days ago I didn't used adb nor tried any manual filesystem operation until I couldn't resume the download.

I simply tried to delete the partial download, but that was not possible due to lack of permissions:

drwxr-s--- 2 u0_a343 ext_data_rw   8192 2022-12-22 20:07 221216
-rw-rw---- 1 u0_a343 ext_data_rw 888208 2022-12-22 19:25 icudt70l.dat
drwxrws--- 2 u0_a343 ext_data_rw   3452 2022-12-23 01:49 logs

I can delete de logs (via adb shell) but can't delete (nor write) inside the 221216 folder, nor just wipe it (despite the files folder at the previous level do is writtable for the shell user):

OnePlus9:/sdcard/Android/data/app.organicmaps $ ll
total 3
drwxrws--- 4 u0_a343 ext_data_rw 3452 2022-12-23 01:53 files

That is not good for non-root hackers ;-)

PS: No, my device is not rooted. A couple of days ago I didn't used adb nor tried any manual filesystem operation until I couldn't resume the download. I simply tried to delete the partial download, but that was not possible due to lack of permissions: ``` drwxr-s--- 2 u0_a343 ext_data_rw 8192 2022-12-22 20:07 221216 -rw-rw---- 1 u0_a343 ext_data_rw 888208 2022-12-22 19:25 icudt70l.dat drwxrws--- 2 u0_a343 ext_data_rw 3452 2022-12-23 01:49 logs ``` I can delete de logs (via adb shell) but can't delete (nor write) inside the 221216 folder, nor just wipe it (despite the _files_ folder at the previous level do is writtable for the shell user): ``` OnePlus9:/sdcard/Android/data/app.organicmaps $ ll total 3 drwxrws--- 4 u0_a343 ext_data_rw 3452 2022-12-23 01:53 files ``` That is not good for non-root hackers ;-)

biodranik commented

2022-12-23 07:28:14 +00:00

(Migrated from github.com)

Thanks for details! That's very interesting that you can access files with adb on Android 13.

@pastk Looks like the global g_totalBytesToDownload is not reset properly.

Thanks for details! That's very interesting that you can access files with adb on Android 13. @pastk Looks like the global `g_totalBytesToDownload` is not reset properly.

lightful commented

2022-12-23 12:27:12 +00:00

(Migrated from github.com)

In addition, Android 13 still allows efficient (high performance) read/write on sdcard/android/obb to any application having the permission to "install applications". In Android 11 this permission was listed right the others, now it is separated in "installation sources".

I have granted such permission to Termux and to Osmand (organicmaps don't allows to be granted such permission, but anyway there is no option to manually configure the storage folder). Some file explorers (Amaze) still request SAF permission, but this is because they don't know they don't actually need it! (Amaze can be granted too). It is incredible how slow is SAF.

Thank to this feature (still not killed by Google nonsense) I can automate the creation of gpx files using Termux scripts, placing them in the android/obb/net.osmand.plus subfolders...

By the way, I think that app developers could easily add some kind "public shared" storage support to every application, via a shared API, so that e.g. a sshfs/ftp server could expose it to the users. Many people use safe applications (in fact I think that most of the malware comes just from the Play Store) and don't need that recent "security improvements". Google security features, as poorly implemented as they are, prevent power users from having control on their devices, and are in fact creating a horde of stupid people instead of leaving the natural selection to do its job... :-)

In addition, Android 13 still allows efficient (high performance) read/write on sdcard/android/obb to any application having the permission to "install applications". In Android 11 this permission was listed right the others, now it is separated in "installation sources". I have granted such permission to Termux and to Osmand (organicmaps don't allows to be granted such permission, but anyway there is no option to manually configure the storage folder). Some file explorers (Amaze) still request SAF permission, but this is because they don't know they don't actually need it! (Amaze can be granted too). It is incredible how slow is SAF. Thank to this feature (still not killed by Google nonsense) I can automate the creation of gpx files using Termux scripts, placing them in the android/obb/net.osmand.plus subfolders... By the way, I think that app developers could easily add some kind "public shared" storage support to every application, via a shared API, so that e.g. a sshfs/ftp server could expose it to the users. Many people use safe applications (in fact I think that most of the malware comes just from the Play Store) and don't need that recent "security improvements". Google security features, as poorly implemented as they are, prevent power users from having control on their devices, and are in fact creating a horde of stupid people instead of leaving the natural selection to do its job... :-)

biodranik commented

2022-12-23 18:15:13 +00:00

(Migrated from github.com)

Thanks for the details! Are there any docs about that "fast" file access and how to properly enable it? We may try to use it for FDroid.

lightful commented

2022-12-28 20:33:29 +00:00

(Migrated from github.com)

There is no need to enable anything... folders under sdcard/Android/obb are as fast as the internal app storage, unlike every other folder under sdcard/ which is too slow in recent android versions, or using the SAF. In fact, Google storagement poor tech is likely taking battery from the users (I assume the slowness is due to high CPU usage).

The only risk is the required "unsafe" permission to be able to read/write... when will Google remove this "hole"?.

And only power users would use it (I had to manually grant permission to "install applications" to my file explorer, Termux,... etc).

Edit: apps with "install applications" permission seems to be added to the ext_obb_rw filesystem group in Android 13 (in Android 10 it was just the "everybody" group as the entire sdcard).

A Termux shell session:

(OnePlus9):~$ id
uid=10240(u0_a240) gid=10240(u0_a240) groups=10240(u0_a240),1079(ext_obb_rw),3003(inet),9997(everybody),20240(u0_a240_cache),50240(all_a240)

(OnePlus9):~$ cd sdcard/Android/obb/

(OnePlus9):~/.../Android/obb$ ls -Fals
total 11
4 drwxrws--x  3 media_rw ext_obb_rw 3452 Dec 28 21:34 ./
4 drwxrws--x  5 media_rw media_rw   3452 Apr  1  2021 ../
0 -rw-rw----  1 u0_a217  ext_obb_rw    0 Oct  8  2021 .nomedia
4 drwxrws--- 16 u0_a238  ext_obb_rw 3452 Dec 28 02:12 net.osmand.plus/

(OnePlus9):~/.../Android/obb$ time cp -rL ~/../usr/lib .
real	0m3.594s
user	0m0.057s
sys	0m2.455s

(OnePlus9):~/.../Android/obb$ time rm -rf lib
real	0m0.465s
user	0m0.009s
sys	0m0.450s

(OnePlus9):~/.../Android/obb$ cd ~/sdcard/

(OnePlus9):~/sdcard$ time cp -rL ~/../usr/lib .
real	0m51.388s
user	0m0.385s
sys	0m4.520s

(OnePlus9):~/sdcard$ time rm -rf lib
real	0m46.094s
user	0m0.157s
sys	0m1.265s

There is no need to enable anything... folders under sdcard/Android/obb are as fast as the internal app storage, unlike every other folder under sdcard/ which is too slow in recent android versions, or using the SAF. In fact, Google storagement poor tech is likely taking battery from the users (I assume the slowness is due to high CPU usage). The only risk is the required "unsafe" permission to be able to read/write... when will Google remove this "hole"?. And only power users would use it (I had to manually grant permission to "install applications" to my file explorer, Termux,... etc). Edit: apps with "install applications" permission seems to be added to the **ext_obb_rw** filesystem group in Android 13 (in Android 10 it was just the "everybody" group as the entire sdcard). A Termux shell session: ``` (OnePlus9):~$ id uid=10240(u0_a240) gid=10240(u0_a240) groups=10240(u0_a240),1079(ext_obb_rw),3003(inet),9997(everybody),20240(u0_a240_cache),50240(all_a240) (OnePlus9):~$ cd sdcard/Android/obb/ (OnePlus9):~/.../Android/obb$ ls -Fals total 11 4 drwxrws--x 3 media_rw ext_obb_rw 3452 Dec 28 21:34 ./ 4 drwxrws--x 5 media_rw media_rw 3452 Apr 1 2021 ../ 0 -rw-rw---- 1 u0_a217 ext_obb_rw 0 Oct 8 2021 .nomedia 4 drwxrws--- 16 u0_a238 ext_obb_rw 3452 Dec 28 02:12 net.osmand.plus/ (OnePlus9):~/.../Android/obb$ time cp -rL ~/../usr/lib . real 0m3.594s user 0m0.057s sys 0m2.455s (OnePlus9):~/.../Android/obb$ time rm -rf lib real 0m0.465s user 0m0.009s sys 0m0.450s (OnePlus9):~/.../Android/obb$ cd ~/sdcard/ (OnePlus9):~/sdcard$ time cp -rL ~/../usr/lib . real 0m51.388s user 0m0.385s sys 0m4.520s (OnePlus9):~/sdcard$ time rm -rf lib real 0m46.094s user 0m0.157s sys 0m1.265s ```

biodranik commented

2022-12-29 00:18:12 +00:00

(Migrated from github.com)

Thanks! We are looking for a generic solution suitable for every user.

pastk commented

2023-01-07 13:58:00 +00:00

Member

It looks like duplicate of #2436 (the second case listed there)

If there is a single interrupted map inside of a big country, then a tap on the "!" (error) country icon tries to download a whole country (or at least free space estimation is done for the whole country)

Probably Download All triggers whole country size estimations too.

It looks like duplicate of #2436 (the second case listed there) > If there is a single interrupted map inside of a big country, then a tap on the "!" (error) country icon tries to download a whole country (or at least free space estimation is done for the whole country) Probably `Download All` triggers whole country size estimations too.

Rows
Columns

"not enough space" while updating maps #4122