From 2734f9fd9e5d4e7dd2eeb8fbbf7c77d157225140 Mon Sep 17 00:00:00 2001 From: Alexey Zakharenkov Date: Fri, 18 Jun 2021 01:10:59 +0300 Subject: [PATCH] Update README --- README.md | 83 +++++++++++++++++++++++++++++--------- scripts/process_subways.sh | 24 ++++++----- 2 files changed, 77 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index c1edfa3..31a96fa 100644 --- a/README.md +++ b/README.md @@ -5,44 +5,86 @@ systems in the world from OpenStreetMap. `subway_structure.py` produces a list of disjunct systems that can be used for routing and for displaying of metro maps. -## How To Validate +## How To Validate * Choose transport data source: 1. Download or update a planet file in o5m format (using `osmconvert` and `osmupdate`). Run `osmfilter` to extract a portion of data for all subways. Or 2. If you don't specify `--xml` or `--source` option to the `process_subways.py` script - it tries to fetch data over [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API). **Not suitable for whole planet or large countries.** + it tries to fetch data over [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API). + **Not suitable for the whole planet or large countries.** * Run `process_subways.py` with appropriate set of command line arguments to build metro structures and receive a validation log. * Run `validation_to_html.py` on that log to create readable HTML tables. -## Validation Script -There is a `process_subways.sh` in the `scripts` directory. The author uses it for -updating both the planet and a city he's working on. Here is an example of a script -for updating the London Underground network: +## Validating of all metro networks + +There is a `process_subways.sh` in the `scripts` directory that is suitable +for validation of all or many metro networks. It relies on a bunch of +environment variables and takes advantage of previous validation runs +for effective recurring validations. See +```bash +./scripts/process_subways.sh --help +``` +for details. Here is an example of the script usage: ```bash -PLANET_PATH=$HOME/osm/planet -export OSMCTOOLS="$PLANET_PATH" -export PLANET="$PLANET_PATH/london.o5m" -export HTML_DIR=tmp_html -export BBOX=-0.681152,51.286758,0.334015,51.740636 -export CITY="London" -export DUMP=london.yaml +export PLANET=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf +export PLANET_METRO="$HOME/metro/planet-metro.o5m +export OSMCTOOLS="$HOME/osmctools" +export TMPDIR="$HOME/metro/tmp" +export HTML_DIR="$HOME/metro/tmp_html" +export DUMP="$HTML_DIR" scripts/process_subways.sh ``` -The bounding box can be found in the -[Google Spreadsheet](https://docs.google.com/spreadsheets/d/1-UHDzfBwHdeyFxgC5cE_MaNQotF3-Y0r1nW9IwpIEj8/edit?usp=sharing). +Set the PLANET_METRO variable to avoid the whole planet processing each time. +Delete the file (but not the variable) to re-generate it if a new city has been added or +a city's bbox has been extended. -This can be simplified by using the `build_city.sh` script, which fetches the bbox from the web: - scripts/build_city.sh london.o5m London +## Validating of a single city + +A single city or a country with few metro networks can be validated much faster +if you allow the `process_subway.py` to fetch data from Overpass API. Here are the steps: + +1. Python3 interpreter required (3.5+) +2. Clone the repo + ``` + git clone https://github.com/alexey-zakharenkov/subways.git subways_validator + cd subways_validator + ``` +3. Execute + ```bash + python3 ./process_subways.py --bbox -c "London" \ + -l validation.log -d London.yaml + ``` + here + - `-c` stands for "city" i.e. network name from the google spreadsheet + - `-l` - path to validation log file + - `-d` (optional) - path to dump network info in YAML format + - `-i` (optional) - path to save overpass-api JSON response + - `-j` (optional) - path to output network GeoJSON (used for rendering) + + `validation.log` would contain the list of errors and warnings. + To convert it into pretty HTML format +4. do + ```bash + mkdir html + python3 ./validation_to_html.py validation.log html + ``` + +## Related external resources + +Summary information about all metro networks that are monitored is gathered in the +[Google Spreadsheet](https://docs.google.com/spreadsheets/d/1SEW1-NiNOnA2qDwievcxYV1FOaQl1mb1fdeyqAxHu3k). + +Not so regular updates of validation results are available at +[this website](https://alexey-zakharenkov.github.io/subways/rapid/). -Daily updates of validation results are available at [this website](http://osm-subway.maps.me). ## Adding Stop Areas To OSM @@ -53,4 +95,7 @@ just upload it. ## Author and License -All scripts were written by Ilya Zverev for MAPS.ME. Published under Apache Licence 2.0. +The main scripts were originally written by Ilya Zverev for MAPS.ME +and were published under Apache Licence 2.0 at https://github.com/mapsme/subways/. + +This fork is maintained by Alexey Zakharenkov and is also published under Apache Licence 2.0. diff --git a/scripts/process_subways.sh b/scripts/process_subways.sh index 1376955..8a02442 100755 --- a/scripts/process_subways.sh +++ b/scripts/process_subways.sh @@ -6,32 +6,34 @@ if [ $# -gt 0 -a \( "${1-}" = "-h" -o "${1-}" = '--help' \) ]; then This script updates a planet or an extract, processes metro networks in it and produces a set of HTML files with validation results. -Usage: $0 +Usage: $0 [] In more detail, the script does the following: - If \$PLANET is a remote file, downloads it. - - Unless \$POLY variable is set and the file exists, generates a *.poly file with union of bboxes of all cities having metro. + - If \$BBOX variable is set, proceeds with this setting for the planet clipping. Otherwise uses \$POLY: + unless \$POLY variable is set and the file exists, generates a *.poly file with union of bboxes of all cities having metro. - Makes a *.o5m extract of the \$PLANET using the *.poly file. - Updates the extract. - Filters railway infrastructure from the extract. - - Uses filtered file for validation and generates other output files. - - Copies results onto remove server, if it is set up. + - Uses filtered file for validation and generates a bunch of output files. + - Copies results onto remote server, if it is set up. During this procedure, as many steps are skipped as possible. Namely: - Making metro extract is skipped if \$PLANET_METRO variable is set and the file exists. - - Update with osmupdate is skipped if SKIP_PLANET_UPDATE or \$SKIP_FILTERING is set. + - Update with osmupdate is skipped if \$SKIP_PLANET_UPDATE or \$SKIP_FILTERING is set. - Filtering is skipped if \$SKIP_FILTERING is set and \$FILTERED_DATA is set and the file exists. -Generated files \$POLY, \$PLANET_METRO, \$FILTERED_DATA are deleted if the correponding variable is not defined or is null, otherwise they are kept. -The PLANET file from remote URL is saved to a tempfile and is removed at the end. +Generated files \$POLY, \$PLANET_METRO, \$FILTERED_DATA are deleted if the corresponding +variable is not defined or is null, otherwise they are kept. +The \$PLANET file from remote URL is saved to a tempfile and is removed at the end. Environment variable reference: - PLANET: path to a local or remote o5m or pbf source file (the entire planet or an extract) - - PLANET_METRO: path to local o5m file with extract of cities having metro. + - PLANET_METRO: path to a local o5m file with extract of cities having metro It's used instead of \$PLANET if exists otherwise it's created first - CITY: name of a city/country to process - BBOX: bounding box of an extract; x1,y1,x2,y2. Has precedence over \$POLY - - POLY: *.poly file with [multi]polygon comprising cities with metro. + - POLY: *.poly file with [multi]polygon comprising cities with metro If neither \$BBOX nor \$POLY is set, then \$POLY is generated - SKIP_PLANET_UPDATE: skip \$PLANET file update. Any non-empty string is True - SKIP_FILTERING: skip filtering railway data. Any non-empty string is True @@ -70,8 +72,8 @@ function check_osmctools() { function check_poly() { - # Checks or generates *.poly file with city bboxes where - # there is a metro but only once during script run + # Checks or generates *.poly file covering cities where + # there is a metro; does this only once during script run. if [ -z "${POLY_CHECKED-}" ]; then if [ -n "${BBOX-}" ]; then