The generator_tool script is launched using "node_storage" of type "mem" even with LXC containers below 64GB of RAM #10395

New issue

Open

opened 2025-03-03 14:56:53 +00:00 by Paspieds · 0 comments

Paspieds commented

2025-03-03 14:56:53 +00:00

(Migrated from github.com)

Describe the issue
Running:

python -m maps_generator --countries="Macedonia" --skip="Coastline" # any input would result to the same event anyway

Inside an LXC container (tested with a Debian and Alma container) will result the generator_tool script to get killed by the system because of too much memory usage.
After some investigation I found the cause. It seems that the function verifying the total available memory takes it from the host and not the container. Since my host has more than 64GB (my container has less), the node storage type is set to mem instead of map. Here is the code involved:

organicmaps/tool/python/generator/settings.py:

...
NODE_STORAGE = "mem" if total_virtual_memory() / 10 ** 9 >= 64 else "map"
...

organicmaps/tool/python/utils/system.py:

...
def total_virtual_memory():
    if sys.platform.startswith("linux"):
        return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")
    else:
	return 0

The SC_PAGE_SIZE and SC_PHYS_PAGES system calls won't take the container values but the host values, probably because they share the host kernel.

The issue

I managed to easily fix the problem enforcing the use of map node storage type. I have not tested to run the maps generator on Docker or Podman (under similar circumstances, host > 64GB RAM and container < 64GB RAM). Does it make sense to fix the total_virtual_memory function to support containers since running such tools inside containers with less than 64GB of RAM would make sense in some cases (maybe someone can attest that it is not a problem on other container engines: Docker, Podman ...)?

System information:

LXC Debian 12 or Almalinux 9

Organic Maps Version:

Commit: Bump Version
After verification, latest commit (when this issue was posted) does not fix the problem: [github] Add leading slash (root) to all dirs.

**Describe the issue** Running: ```bash python -m maps_generator --countries="Macedonia" --skip="Coastline" # any input would result to the same event anyway ``` Inside an LXC container (tested with a Debian and Alma container) will result the generator_tool script to get killed by the system because of too much memory usage. After some investigation I found the cause. It seems that the function verifying the total available memory takes it from the host and not the container. Since my host has more than 64GB (my container has less), the node storage type is set to **mem** instead of **map**. Here is the code involved: organicmaps/tool/python/generator/settings.py: ```python ... NODE_STORAGE = "mem" if total_virtual_memory() / 10 ** 9 >= 64 else "map" ... ``` organicmaps/tool/python/utils/system.py: ```python ... def total_virtual_memory(): if sys.platform.startswith("linux"): return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") else: return 0 ``` The **SC_PAGE_SIZE** and **SC_PHYS_PAGES** system calls won't take the container values but the host values, probably because they share the host kernel. The issue I managed to easily fix the problem enforcing the use of **map** node storage type. I have not tested to run the maps generator on Docker or Podman (under similar circumstances, host > 64GB RAM and container < 64GB RAM). Does it make sense to fix the **total_virtual_memory** function to support containers since running such tools inside containers with less than 64GB of RAM would make sense in some cases (maybe someone can attest that it is not a problem on other container engines: Docker, Podman ...)? **System information:** - LXC Debian 12 or Almalinux 9 **Organic Maps Version:** - Commit: [Bump Version](https://git.omaps.dev/organicmaps/organicmaps/commit/69bdc867795d7ba83e25050ea5cb078d9379d1e9) After verification, latest commit (when this issue was posted) does not fix the problem: [[github] Add leading slash (root) to all dirs](https://git.omaps.dev/organicmaps/organicmaps/commit/d21800ec25bf2a084c1be238ceb0eb822129e28f).

👍 1

This repo is archived. You cannot comment on issues.