Use plain text format for compiled styles #10341

New issue

Open

opened 2025-02-25 05:50:54 +00:00 by pastk · 0 comments

pastk commented

2025-02-25 05:50:54 +00:00

Owner

At the moment styles are compiled into data/*.bin files which are google protobuf serializations.

Issues with that approach:

all style PRs have to be applied sequentially as merge doesn't work for binaries, hence one has to always regenerate on top of the last merged style commit, which leads to many unnecessary regenerations and force pushes and ultimately to organicmaps/organicmaps#6167
we have to generate (keep in the repo) plain text versions of compiled *.bin files anyway, as they're necessary for debugging and QA/reviews
protobuf is a growing big dependency hence we don't want to update it organicmaps/organicmaps#6310
the OM repo grows bigger and bigger with every binary regeneration commit merged (and it'll grow faster when we introduce more styles, e.g. cycling)

Seemingly the only benefit of using a binary format is that it loads faster.

However, is that really a big deal?

it has to load once only (at startup)
- it'd be great to measure what percent of the startup time does it take
- could be optimized to load only one style first (which is being displayed; as atm all 6 styles are loaded at once)
OM loads many plain text data files at the startup e.g.
- categories.txt
- localization files (usually xml)
- classificator.txt, types.txt, etc.
- tracks and bookmarks (xml)
  - we even ditched the binary kmb format in favor of plain kml as there was little benefit, right?

So the idea is to load e.g. data/drules_proto_default_light.txt instead of data/drules_proto_default_light.bin. The format is pretty simple and could be easily changed to something more standard like yaml or xml. It could be also made much more compact as it is now (there is a lot of data duplication).

At the moment styles are compiled into `data/*.bin` files which are google protobuf serializations. Issues with that approach: - all style PRs have to be applied sequentially as merge doesn't work for binaries, hence one has to always regenerate on top of the last merged style commit, which leads to many unnecessary regenerations and force pushes and ultimately to https://git.omaps.dev/organicmaps/organicmaps/issues/6167 - we have to generate (keep in the repo) plain text versions of compiled *.bin files anyway, as they're necessary for debugging and QA/reviews - protobuf is a growing big dependency hence we don't want to update it https://git.omaps.dev/organicmaps/organicmaps/pulls/6310 - the OM repo grows bigger and bigger with every binary regeneration commit merged (and it'll grow faster when we introduce more styles, e.g. cycling) Seemingly the only benefit of using a binary format is that it loads faster. However, is that really a big deal? - it has to load once only (at startup) - it'd be great to measure what percent of the startup time does it take - could be optimized to load only one style first (which is being displayed; as atm all 6 styles are loaded at once) - OM loads many plain text data files at the startup e.g. - categories.txt - localization files (usually xml) - classificator.txt, types.txt, etc. - tracks and bookmarks (xml) - we even ditched the binary kmb format in favor of plain kml as there was little benefit, right? So the idea is to load e.g. `data/drules_proto_default_light.txt` instead of `data/drules_proto_default_light.bin`. The format is pretty simple and could be easily changed to something more standard like yaml or xml. It could be also made much more compact as it is now (there is a lot of data duplication).

👍 2

This repo is archived. You cannot comment on issues.