organicmaps/organicmaps

Fork 18

custom map regions #9094

New issue

Open

opened 2024-08-25 14:45:57 +00:00 by elandorr · 7 comments

elandorr commented

2024-08-25 14:45:57 +00:00

(Migrated from github.com)

⚠ Have you searched for similar, already existing issues?
yes

Is your feature request related to a problem? Please describe.

For those who aren't rich, storage is finite. Devices with 4-8GB storage (effectively much less as android takes up a large amount, even with the cleanest custom ROMs) are abound. They often even work better than newer ones as they tend to be sturdier and have replaceable batteries.
This makes organicmaps impossible to use, as even the currently mandatory world map takes up a large amount. (~60MB you never use. Why would anyone want a world map? You ought to know which continent you're on.)

Describe the ideal solution
A tool, preferably a desktop one like MOBAC, that lets you define regions as needed.
It'd generate a file you can copy to the android device to use whenever, and share.
Ideally with the option to select only data you really need: Places/streets, but no tourist/commercial/whatever crap for instance. OSM contributors in some regions, particularly Germany, are so diligent, they add EVERYTHING. (the upstream OSM map includes curbs and individual trash bins, for reference) A total waste of resources. It also makes the map harder to view. And these are perpetually out of date by nature, so to really give all the metadata a chance to be useful, we'd have to hammer the servers regularly which is far away from the 'organic' theme. (E.g. streaming is literally killing the planet. Why add pointless traffic? Ignoring the fact that it costs money unless you have some bourgeois unlimited plan.)

Benefits:

no internet needed on the device, which is a pain to begin with
- some people do not want any internet on their device, be it for money or privacy reasons
no wasted space
more life out of perfectly functioning android devices, matching your 'organic' theme
much less server load as you don't have to download data you'll never use
much less server load as you can quickly share what you already have locally, offline even
if it's possible to select only datapoints you need you may be able to download a giant region only consisting of roads in the space you'd otherwise only fit a small region with all the other metadata in
- this'd be great as a quick paper-map replacement

Bonus:

Now a really novel and resource saving idea would be a tool to hone in on small blocks as needed. You could generate only enough "tiles" to get x meters surrounding the route desired. 90% of a map is usually never looked at after all.

Describe alternatives you have considered

I used to recommend MOBAC with backcountry navigator, but the only generic mapsource for MOBAC has died. (4umaps.eu is gone)
It worked very well despite being tile based. You got exactly what you needed, no waste.
For example, for a reasonably sized city I liberally created a map in about 10MB. Much easier to share in pleb life, and it can be copied to any device. It's fast and works perfectly, like a big paper map, and has an additional zoom level for comfort if you chose.
Large image files are not useful as they lag like hell on android, and OSM won't let you create them anyway. (at least not in any reasonably accessible way I found) A bunch of (compressed?) images in a proper database like BCN does work well.
A real paper map. Doesn't run out of battery, doesn't lag, doesn't get on your nerves. But somehow they have become prohibitively expensive and rare. They used to be everywhere and cheap enough to wipe with. Lost? Find the nearest village, buy a bunch of maps for a few bucks, repeat as required. I can't even find one in a quick online search now, incredible.

Additional context
On an average-low device I just tried I found the search to be excruciatingly slow, even though I only downloaded a single individual region. Maybe also related to the massive amount of waste data? It even suggests store opening hours and such, which are not remotely the job of a map. Thus it ends up being a simple paper map you scroll around anyhow, like BCN.

It's weird to me that we have all these contributors creating OSM in extreme detail in their spare time, but we have no way of exporting a bunch of large-but-not-laggy-yet image files. All the sites I found are either dead or commercializing volunteer labor. As long as you don't download all the metadata, even a continent-sized roadmap should cause comparably minimal traffic. The only active and serious looking thing I found is geofabrik, but that is huge and includes everything. Also requires a small singularity to process and advanced knowledge of GIS tools, not quite practical.
MOBAC+BCN were just geeky enough to still be useful in everyday life.

⚠ Have you searched for similar, already existing issues? yes **Is your feature request related to a problem? Please describe.** - For those who aren't rich, storage is finite. Devices with 4-8GB storage (effectively **much** less as android takes up a large amount, even with the cleanest custom ROMs) are abound. They often even work better than newer ones as they tend to be sturdier and have replaceable batteries. - This makes organicmaps impossible to use, as even the currently mandatory world map takes up a large amount. (~60MB you never use. Why would anyone want a world map? You ought to know which continent you're on.) **Describe the ideal solution** A tool, preferably a desktop one like MOBAC, that lets you define regions **as needed**. It'd generate a file you can copy to the android device to use whenever, and share. Ideally with the option to select only data you really **need**: Places/streets, but no tourist/commercial/whatever crap for instance. OSM contributors in some regions, particularly Germany, are so diligent, they add EVERYTHING. (the upstream OSM map includes curbs and _individual_ trash bins, for reference) A total waste of resources. It also makes the map harder to view. And these are **perpetually** out of date by nature, so to really give all the metadata a chance to be useful, we'd have to hammer the servers regularly which is far away from the 'organic' theme. (E.g. streaming is literally killing the planet. Why add pointless traffic? Ignoring the fact that it costs money unless you have some bourgeois unlimited plan.) Benefits: - **no internet needed** on the device, which is a pain to begin with - some people do not want any internet on their device, be it for money or privacy reasons - **no wasted space** - **more life out of perfectly functioning android devices**, matching your 'organic' theme - **much less server load** as you don't have to download data you'll never use - much less server load as you can quickly share what you already have locally, offline even - if it's possible to select only **datapoints you need** you may be able to download **a giant region only consisting of roads in the space you'd otherwise only fit a small region with all** the other metadata in - this'd be great as a quick paper-map replacement Bonus: Now a really novel and resource saving idea would be a tool to hone in on small blocks as needed. You could generate only enough "tiles" to get x meters surrounding the route desired. 90% of a map is usually never looked at after all. **Describe alternatives you have considered** - I used to recommend MOBAC with backcountry navigator, but the only generic mapsource for MOBAC has died. (4umaps.eu is gone) It worked very well despite being tile based. You got exactly what you needed, no waste. For example, for a reasonably sized city I liberally created a map in about 10MB. Much easier to share in pleb life, and it can be copied to any device. It's fast and works perfectly, like a big paper map, and has an additional zoom level for comfort if you chose. - Large image files are not useful as they lag like hell on android, and OSM won't let you create them anyway. (at least not in any reasonably accessible way I found) A bunch of (compressed?) images in a proper database like BCN does work well. - A real paper map. Doesn't run out of battery, doesn't lag, doesn't get on your nerves. But somehow they have become prohibitively expensive and rare. They used to be everywhere and cheap enough to wipe with. Lost? Find the nearest village, buy a bunch of maps for a few bucks, repeat as required. I can't even find one in a quick online search now, incredible. **Additional context** On an average-low device I just tried I found the search to be **excruciatingly** slow, even though I only downloaded a single individual region. Maybe also related to the massive amount of waste data? It even suggests store opening hours and such, which are not remotely the job of a map. Thus it ends up being a simple paper map you scroll around anyhow, like BCN. It's weird to me that we have all these contributors creating OSM in extreme detail in their spare time, but we have no way of exporting a bunch of large-but-not-laggy-yet image files. All the sites I found are either dead or commercializing volunteer labor. As long as you don't download all the metadata, even a continent-sized roadmap should cause comparably minimal traffic. The only active and serious looking thing I found is geofabrik, but that is huge and includes everything. Also requires a small singularity to process and advanced knowledge of GIS tools, not quite practical. MOBAC+BCN were just geeky enough to still be useful in everyday life.

biodranik commented

2024-08-25 14:58:24 +00:00

(Migrated from github.com)

What is your device model?

Am I correctly understanding that you want developers to do a huge amount of work to improve user experience on some older/outdated devices?

What is your device model? Am I correctly understanding that you want developers to do a huge amount of work to improve user experience on some older/outdated devices?

elandorr commented

2024-08-25 18:08:10 +00:00

(Migrated from github.com)

Thank you for your reply.

I don't expect anything.
Heck I'm fine with real paper maps.

I'm pointing out obvious issues that prevent organicmaps from really being used where it matters. And obvious inefficiencies that I used to handle as described, but can't anymore as the tileserver is down.

Realistically, just about everyone will be on live-streamed google maps anyway.
Those who consider or even find organicmaps will tend to be:

F-Droid users, who are mostly
interested in offline software via the privacy angle
interested in offline software due to speed/traffic/financial limitations in their region
likely users of devices on the lower than average end, who care more about ROM selection and longevity (With so many devices released every day you're limited to slightly older ones if you care about any of this. Except the pixel of course, which has graphene, but is prohibitively expensive for most of the world.)
people who want a map for use during sports/trips - for which you'd greatly prefer a cheaper device due to risk of breaking it

There are too many android devices to point out a single one. Billions of them land on the trash heap because of the malicious and egotistic mentality of replacing working things.

The "outdated devices" as you call them make up the VAST majority of android in the world.
Gazillions of devices from 4-32G, maybe 64 if lucky. (again, effective storage sans android/app reserved being much less)
Particularly outside the little spots of privilege.
There are even companies collecting trashed devices to resell them to the poor in western countries. (weird "3rd hand for squeezing profit even of the poor" model) Not to mention all the other crisis-laden regions today where all the "old" devices are dumped.
Even millions of people in "rich" western countries can't afford enough food - why would they spend money on yet another smartphone when they already have one that works?
They're most likely to just use google maps.

Storage demands continually grow for no reason because of such mentalities. In 10 years, there may be another zero at the end, but the same situation.

Longer device lifetime benefits everyone including earth itself!
Should be relevant to someone using 'organic' branding.

Storage is quite limited even on average current devices. Android takes up a very large chunk, apps generally don't care about wasting 100s of MB anymore either. Add some music/pictures and you're full before realizing.
At that point most people with enough traffic in their plan will use an online app. Then you don't need organicmaps at all and can just use whatever flashy high end thing google has preinstalled.

As everyone who's seen a datacenter knows, streaming/downloading isn't free just because you don't see it. The dependence on the internet is a bigger factor in killing the planet than most things an ordinary human does in a day. People don't think twice about wasting data 24/7, "it's just digital". It still has to run and process with most of it never being used.

And even if you had let's say 64G where it becomes less immediately relevant - would you want to spend it on a map you barely use a few % of, or on music/pictures/whateverisimportanttoyou?

And as I said: Why waste resources? At all? This planet is polluted to high heaven and toxins circulate everyone's bloodstream because of such mentality. If it's not needed, why do it. Why would it be acceptable to generate useless traffic or waste storage on something never needed?

Why increase hosting costs for no reason?

Even ignoring the environmental reality, you wouldn't go to the store and buy 10kg of meat when you know you can only eat 1kg and the rest will never be touched.
If you're rich enough, yes you can still do that, but is it reasonable?
Wouldn't you feel like a spoiled brat for doing so?

It shouldn't be impossibly complicated for someone knowledgeable in the OSM scene, either:

MOBAC already exists and is open source, the plain workflow is already covered many times over by multiple tools. ("allow user to make a rectangular selection on a map and iterate over it")
Tools for processing OSM data exist, though far too complicated for outsiders and typical users. (in case of the big geofabrik ones you need a very strong PC and must be deeply in tech, very quick skill escalation, not fun for one-off/occasional use)
For QGIS something exists that allows downloading specific datatypes from OSM, the core problem is already solved!

You ALREADY get everything when visiting https://www.openstreetmap.org/.
This currently has a 50k node limit for the web export - in the worst possible format, uncompressed not even minified XML. Without all the unneeded types, this should cover enough to iterate even over large regions without stressing the servers.
The nodes seem to be conveniently self-contained (but I haven't studied OSM's format at all, just a quick glance), and should be highly compressible with e.g. zstd so traffic could easily be reduced by a lot.
(zstd is ridiculously efficient at low levels while still being faster than most everything! If you've never tried it, give it a shot, it's great and has essentially zero overhead at low levels. High levels scale badly though.)

The data exist - the data can be downloaded by type (as per QGIS tool) - it should be no big deal to iterate a rectangle instead of downloading the proverbial kitchen sink.
It's not like you do anything differently when you browse using live online mode in any of the dozens of OSM apps.

An example of useless data that also exists dis-proportionally more often than e.g. streets (because a single street might have 5 trash bins etc.):

  <tag k="amenity" v="waste_basket"/>

And tons of impressive-yet-useless details like:

  <tag k="conveying" v="yes"/>
  <tag k="highway" v="steps"/>
  <tag k="incline" v="up"/>
  <tag k="level" v="-1;-2"/>
  <tag k="lit" v="24/7"/>

Opening hours that can only be out-of-date unless constantly re-downloaded (afaik it doesn't diff), things like clothing shops and others you likely won't ever need (especially not for sports/transportation) etc. pp.

Just some food for thought.
I'm aware this is not a commercial project obviously. Coming from a linux background the modern mentality screams 'WHY?!'. People used to pride themselves in creating efficient software, finding clever ways to get more out of less.
It struck me as obvious that one wouldn't throw away a perfectly fine device that might just exist another 10 years. And do the exact same job then. A map is a map.
The lighter, faster, more responsive, the better.

It's not like anything would even change if one were to constantly buy new devices.
The device you buy today will be crap low end in a few years.
By then people will have gotten more lazy, storage requirements will have multiplied, and you repeat for no reason.
Ad infinitum.....

It struck me as particularly odd while having a paper map next to it.
I could take pictures of every page, run an image optimizer over it, and have a pretty small map.
It would use barely any resources and be super responsive on everything as long as the individual picture isn't too big.
All it'd need is a way to easily browse them - which is just what 'backcountry navigator' does/did.

Have a nice one!

Thank you for your reply. **I don't expect anything.** Heck I'm fine with real paper maps. I'm pointing out obvious issues that prevent organicmaps from really being used where it matters. And obvious inefficiencies that I used to handle as described, but can't anymore as the tileserver is down. Realistically, just about everyone will be on live-streamed google maps anyway. Those who consider or even find organicmaps will tend to be: - F-Droid users, who are mostly - interested in offline software via the privacy angle - interested in offline software due to speed/traffic/financial limitations in their region - likely users of devices on the lower than average end, who care more about ROM selection and longevity (With so many devices released every day you're limited to slightly older ones if you care about any of this. Except the pixel of course, which has graphene, but is prohibitively expensive for most of the world.) - people who want a map for use during sports/trips - for which you'd greatly prefer a cheaper device due to risk of breaking it There are too many android devices to point out a single one. Billions of them land on the trash heap because of the malicious and egotistic mentality of _replacing working things_. The "outdated devices" as you call them make up the VAST majority of android in the world. Gazillions of devices from 4-32G, maybe 64 if lucky. (again, effective storage sans android/app reserved being much less) Particularly outside the little spots of privilege. There are even companies collecting trashed devices to resell them to the poor in western countries. (weird "3rd hand for squeezing profit even of the poor" model) Not to mention all the other crisis-laden regions today where all the "old" devices are dumped. Even millions of people in "rich" western countries can't afford enough food - why would they spend money on yet another smartphone when they already have one that works? They're most likely to just use google maps. Storage demands continually grow for _no reason_ because of such mentalities. In 10 years, there may be another zero at the end, but the same situation. **Longer device lifetime benefits everyone including earth itself!** _Should be relevant to someone using 'organic' branding._ Storage is quite limited even on average current devices. Android takes up a very large chunk, apps generally don't care about wasting 100s of MB anymore either. Add some music/pictures and you're full before realizing. At that point most people with enough traffic in their plan will use an online app. Then you don't need organicmaps at all and can just use whatever flashy high end thing google has preinstalled. As everyone who's seen a datacenter knows, [streaming/downloading isn't free just because you don't see it](https://www.aljazeera.com/economy/2020/2/28/emissions-possible-streaming-music-swells-carbon-footprints). The dependence on the internet is a bigger factor in killing the planet than most things an ordinary human does in a day. People don't think twice about wasting data 24/7, "it's just digital". It still has to run and process with most of it never being used. And even if you had let's say 64G where it becomes less immediately relevant - would you want to spend it on a map you barely use a few % of, or on music/pictures/whateverisimportanttoyou? And as I said: **Why waste resources?** At all? This planet is polluted to high heaven and toxins circulate everyone's bloodstream because of such mentality. If it's not needed, why do it. Why would it be acceptable to generate useless traffic or waste storage on something **never** needed? Why increase hosting costs for no reason? Even ignoring the environmental reality, you wouldn't go to the store and buy 10kg of meat when you know you can only eat 1kg and the rest will never be touched. If you're rich enough, yes you can still do that, but is it reasonable? Wouldn't you feel like a spoiled brat for doing so? It shouldn't be impossibly complicated for someone knowledgeable in the OSM scene, either: MOBAC already exists and is open source, the plain workflow is already covered many times over by multiple tools. ("allow user to make a rectangular selection on a map and iterate over it") Tools for processing OSM data exist, though far too complicated for outsiders and typical users. (in case of the big geofabrik ones you need a very strong PC and must be deeply in tech, very quick skill escalation, not fun for one-off/occasional use) For QGIS something exists that allows downloading specific datatypes from OSM, the core problem is already solved! You ALREADY get everything when visiting https://www.openstreetmap.org/. This currently has a 50k node limit for the web export - in the **worst** possible format, uncompressed not even minified XML. Without all the unneeded types, this should cover enough to iterate even over large regions without stressing the servers. The nodes seem to be conveniently self-contained (but I haven't studied OSM's format at all, just a quick glance), and should be highly compressible with e.g. zstd so traffic could easily be reduced by a lot. (zstd is ridiculously efficient at low levels while still being faster than most everything! If you've never tried it, give it a shot, it's great and has essentially zero overhead at low levels. High levels scale badly though.) The data exist - the data can be downloaded by type (as per QGIS tool) - it should be no big deal to iterate a rectangle instead of downloading the proverbial kitchen sink. It's not like you do anything differently when you browse using live online mode in any of the dozens of OSM apps. An example of useless data that also exists dis-proportionally more often than e.g. streets (because a single street might have 5 trash bins etc.): ``` <tag k="amenity" v="waste_basket"/> ``` And tons of impressive-yet-useless details like: ``` <tag k="conveying" v="yes"/> <tag k="highway" v="steps"/> <tag k="incline" v="up"/> <tag k="level" v="-1;-2"/> <tag k="lit" v="24/7"/> ``` Opening hours that can only be out-of-date unless constantly re-downloaded (afaik it doesn't diff), things like clothing shops and others you likely won't ever need (especially not for sports/transportation) etc. pp. Just some food for thought. I'm aware this is not a commercial project obviously. Coming from a linux background the modern mentality screams 'WHY?!'. People used to pride themselves in creating efficient software, finding clever ways to get more out of less. It struck me as obvious that one wouldn't throw away a perfectly fine device that might just exist another 10 years. And do the exact same job then. A map is a map. The lighter, faster, more responsive, the better. It's not like anything would even change if one were to constantly buy new devices. The device you buy today will be crap low end in a few years. By then people will have gotten more lazy, storage requirements will have multiplied, and you repeat for no reason. Ad infinitum..... It struck me as particularly odd while having a paper map next to it. I could take pictures of every page, run an image optimizer over it, and have a pretty small map. It would use barely any resources and be super responsive on everything as long as the individual picture isn't too big. All it'd need is a way to easily browse them - which is just what 'backcountry navigator' does/did. Have a nice one!

biodranik commented

2024-08-25 19:48:41 +00:00

(Migrated from github.com)

What exact problem are you trying to solve? Installing an app and downloading one map of your local region/country may take around 150-300 Mb. What do you want to optimize and why?

_What exact problem are you trying to solve?_ Installing an app and downloading one map of your local region/country may take around 150-300 Mb. What do you want to optimize and why?

elandorr commented

2024-08-30 16:21:07 +00:00

(Migrated from github.com)

I think I explained this extensively enough already.

Why waste at all? Multiply by user count/reinstalls/updates/etc. It's an absurd mentality.
"low" bandwidth situations
"low" storage situations
you want to use the storage for data you really use, not useless datapoints
android already uses a large chunk for itself, when updating applications/etc. this temporarily increases even more, forcing you to delete and re-transfer things
extend device lifetime
when you really need a map and don't just use it like a toy you want something sturdy/expendable, i.e. an "older" device
etcpp.

Random size test to humor you:
a medium city via MOBAC - ~10 MB incl. multiple zoom levels which is already comfort only/could be saved, very fast
a medium city via organic - ~100 MB, search too slow to use, initial load also somewhat laggy

a longer trip through the country via MOBAC - maybe ~100 MB if you're liberal with regions
same trip via organic - you need 3 GB+ even if you hand-pick the regions that cross your path, realistically you won't do that and use "download all", search will be utterly unusable if the small one is already too slow

This is especially hilarious considering MOBAC is by definition inefficient due to being tile-based. Organic should be much smaller than that when you don't include all the never-used data.

I think I explained this extensively enough already. - Why waste at all? Multiply by user count/reinstalls/updates/etc. It's an absurd mentality. - "low" bandwidth situations - "low" storage situations - you want to use the storage for data you really use, not useless datapoints - android already uses a large chunk for itself, when updating applications/etc. this temporarily increases even more, forcing you to delete and re-transfer things - extend device lifetime - when you really need a map and don't just use it like a toy you want something sturdy/expendable, i.e. an "older" device - etcpp. Random size test to humor you: a medium city via MOBAC - ~10 MB incl. multiple zoom levels which is already comfort only/could be saved, very fast a medium city via organic - ~100 MB, search too slow to use, initial load also somewhat laggy a longer trip through the country via MOBAC - maybe ~100 MB if you're liberal with regions same trip via organic - you need 3 GB+ even if you hand-pick the regions that cross your path, realistically you won't do that and use "download all", search will be utterly unusable if the small one is already too slow This is especially hilarious considering MOBAC is by definition inefficient due to being tile-based. Organic should be much smaller than that when you don't include all the never-used data.

biodranik commented

2024-08-30 18:03:19 +00:00

(Migrated from github.com)

There is no "never-used data" in Organic Maps.

pastk commented

2024-10-11 21:10:38 +00:00

Member

FYI its possible to download "roads only" maps in OsmAnd (though they're not too small anyway because OsmAnd's map file format is not too space-efficient generally).

What you described (a very minimalistic map) could be very nice for some use cases, but quite niche still. It'd be more suited for a separate fork (or at least a separate set of minimalist map files maintained by someone).

FYI its possible to download "roads only" maps in OsmAnd (though they're not too small anyway because OsmAnd's map file format is not too space-efficient generally). What you described (a very minimalistic map) could be very nice for some use cases, but quite niche still. It'd be more suited for a separate fork (or at least a separate set of minimalist map files maintained by someone).

meenbeese commented

2024-10-12 19:13:06 +00:00

(Migrated from github.com)

I think this issue is wildly out of scope for OM, and not in line with where the project is going towards. Thus, I suggest closing this issue.