Update README

README.md

@@ -5,44 +5,86 @@ systems in the world from OpenStreetMap. `subway_structure.py` produces
 a list of disjunct systems that can be used for routing and for displaying
 of metro maps.
 
-## How To Validate
 
+## How To Validate
 
 * Choose transport data source:
   1. Download or update a planet file in o5m format (using `osmconvert` and `osmupdate`).
      Run `osmfilter` to extract a portion of data for all subways. Or
   2. If you don't specify `--xml` or `--source` option to the `process_subways.py` script
-     it tries to fetch data over [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API). **Not suitable for whole planet or large countries.**
+     it tries to fetch data over [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API).
+     **Not suitable for the whole planet or large countries.**
 * Run `process_subways.py` with appropriate set of command line arguments
   to build metro structures and receive a validation log.
 * Run `validation_to_html.py` on that log to create readable HTML tables.
 
-## Validation Script
 
-There is a `process_subways.sh` in the `scripts` directory. The author uses it for
-updating both the planet and a city he's working on. Here is an example of a script
-for updating the London Underground network:
+## Validating of all metro networks
+
+There is a `process_subways.sh` in the `scripts` directory that is suitable
+for validation of all or many metro networks. It relies on a bunch of
+environment variables and takes advantage of previous validation runs
+for effective recurring validations. See
+```bash
+./scripts/process_subways.sh --help
+```
+for details. Here is an example of the script usage:
 
 ```bash
-PLANET_PATH=$HOME/osm/planet
-export OSMCTOOLS="$PLANET_PATH"
-export PLANET="$PLANET_PATH/london.o5m"
-export HTML_DIR=tmp_html
-export BBOX=-0.681152,51.286758,0.334015,51.740636
-export CITY="London"
-export DUMP=london.yaml
+export PLANET=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf
+export PLANET_METRO="$HOME/metro/planet-metro.o5m
+export OSMCTOOLS="$HOME/osmctools"
+export TMPDIR="$HOME/metro/tmp"
+export HTML_DIR="$HOME/metro/tmp_html"
+export DUMP="$HTML_DIR"
 
 scripts/process_subways.sh
 ```
 
-The bounding box can be found in the
-[Google Spreadsheet](https://docs.google.com/spreadsheets/d/1-UHDzfBwHdeyFxgC5cE_MaNQotF3-Y0r1nW9IwpIEj8/edit?usp=sharing).
+Set the PLANET_METRO variable to avoid the whole planet processing each time.
+Delete the file (but not the variable) to re-generate it if a new city has been added or
+a city's bbox has been extended.
 
-This can be simplified by using the `build_city.sh` script, which fetches the bbox from the web:
 
-    scripts/build_city.sh london.o5m London
+## Validating of a single city
+
+A single city or a country with few metro networks can be validated much faster
+if you allow the `process_subway.py` to fetch data from Overpass API. Here are the steps:
+
+1. Python3 interpreter required (3.5+)
+2. Clone the repo
+    ```
+    git clone https://github.com/alexey-zakharenkov/subways.git subways_validator
+    cd subways_validator
+   ```
+3. Execute
+    ```bash
+    python3 ./process_subways.py --bbox -c "London" \
+        -l validation.log -d London.yaml
+    ```
+    here
+    - `-c` stands for "city" i.e. network name from the google spreadsheet
+    - `-l`  - path to validation log file
+    - `-d` (optional) - path to dump network info in YAML format
+    - `-i` (optional) - path to save overpass-api JSON response
+    - `-j` (optional) - path to output network GeoJSON (used for rendering)
+
+    `validation.log` would contain the list of errors and warnings.
+    To convert it into pretty HTML format
+4. do
+    ```bash
+    mkdir html
+    python3 ./validation_to_html.py validation.log html
+    ```
+
+## Related external resources
+
+Summary information about all metro networks that are monitored is gathered in the
+[Google Spreadsheet](https://docs.google.com/spreadsheets/d/1SEW1-NiNOnA2qDwievcxYV1FOaQl1mb1fdeyqAxHu3k).
+
+Not so regular updates of validation results are available at
+[this website](https://alexey-zakharenkov.github.io/subways/rapid/).
 
-Daily updates of validation results are available at [this website](http://osm-subway.maps.me).
 
 ## Adding Stop Areas To OSM
 
@@ -53,4 +95,7 @@ just upload it.
 
 ## Author and License
 
-All scripts were written by Ilya Zverev for MAPS.ME. Published under Apache Licence 2.0.
+The main scripts were originally written by Ilya Zverev for MAPS.ME
+and were published under Apache Licence 2.0 at https://github.com/mapsme/subways/.
+
+This fork is maintained by Alexey Zakharenkov and is also published under Apache Licence 2.0.

scripts/process_subways.sh

@@ -6,32 +6,34 @@ if [ $# -gt 0 -a \( "${1-}" = "-h" -o "${1-}" = '--help' \) ]; then
 This script updates a planet or an extract, processes metro networks in it
 and produces a set of HTML files with validation results.
 
-Usage: $0 <local/planet.{pbf,o5m} | http://mirror.osm.ru/planet.{pbf,o5m}>
+Usage: $0 [<local/planet.{pbf,o5m} | http://mirror.osm.ru/planet.{pbf,o5m}>]
 
 In more detail, the script does the following:
   - If \$PLANET is a remote file, downloads it.
-  - Unless \$POLY variable is set and the file exists, generates a *.poly file with union of bboxes of all cities having metro.
+  - If \$BBOX variable is set, proceeds with this setting for the planet clipping. Otherwise uses \$POLY:
+    unless \$POLY variable is set and the file exists, generates a *.poly file with union of bboxes of all cities having metro.
   - Makes a *.o5m extract of the \$PLANET using the *.poly file.
   - Updates the extract.
   - Filters railway infrastructure from the extract.
-  - Uses filtered file for validation and generates other output files.
-  - Copies results onto remove server, if it is set up.
+  - Uses filtered file for validation and generates a bunch of output files.
+  - Copies results onto remote server, if it is set up.
 
 During this procedure, as many steps are skipped as possible. Namely:
   - Making metro extract is skipped if \$PLANET_METRO variable is set and the file exists.
-  - Update with osmupdate is skipped if SKIP_PLANET_UPDATE or \$SKIP_FILTERING is set.
+  - Update with osmupdate is skipped if \$SKIP_PLANET_UPDATE or \$SKIP_FILTERING is set.
   - Filtering is skipped if \$SKIP_FILTERING is set and \$FILTERED_DATA is set and the file exists.
 
-Generated files \$POLY, \$PLANET_METRO, \$FILTERED_DATA are deleted if the correponding variable is not defined or is null, otherwise they are kept.
-The PLANET file from remote URL is saved to a tempfile and is removed at the end.
+Generated files \$POLY, \$PLANET_METRO, \$FILTERED_DATA are deleted if the corresponding
+variable is not defined or is null, otherwise they are kept.
+The \$PLANET file from remote URL is saved to a tempfile and is removed at the end.
 
 Environment variable reference:
   - PLANET: path to a local or remote o5m or pbf source file (the entire planet or an extract)
-  - PLANET_METRO: path to local o5m file with extract of cities having metro.
+  - PLANET_METRO: path to a local o5m file with extract of cities having metro
     It's used instead of \$PLANET if exists otherwise it's created first
   - CITY: name of a city/country to process
   - BBOX: bounding box of an extract; x1,y1,x2,y2. Has precedence over \$POLY
-  - POLY: *.poly file with [multi]polygon comprising cities with metro.
+  - POLY: *.poly file with [multi]polygon comprising cities with metro
     If neither \$BBOX nor \$POLY is set, then \$POLY is generated
   - SKIP_PLANET_UPDATE: skip \$PLANET file update. Any non-empty string is True
   - SKIP_FILTERING: skip filtering railway data. Any non-empty string is True
@@ -70,8 +72,8 @@ function check_osmctools() {
 
 
 function check_poly() {
-  # Checks or generates *.poly file with city bboxes where
-  # there is a metro but only once during script run
+  # Checks or generates *.poly file covering cities where
+  # there is a metro; does this only once during script run.
 
   if [ -z "${POLY_CHECKED-}" ]; then
     if [ -n "${BBOX-}" ]; then