WIP: Gtfs Converter Skips GTFS feeds #2776

Draft

fardeenfs wants to merge 15 commits from fardeenfs/public-transit into master

fardeenfs commented

2022-06-19 06:49:46 +00:00

(Migrated from github.com)

The current implementation of the GTFS converter tool skips feeds with more than 5 corrupted shapes. This ends up skipping a very large number of feeds. I think it would be better if all feeds with at least 1 valid shape be considered and skip the corrupted shapes instead. I removed a few lines of the existing code to ensure this.

biodranik (Migrated from github.com) reviewed 2022-06-19 06:49:46 +00:00

biodranik commented

2022-06-19 07:07:55 +00:00

(Migrated from github.com)

What are exactly "corrupted shapes"?
Why are they considered "corrupted"?
What can be done on our side to properly process everything?

1. What are exactly "corrupted shapes"? 2. Why are they considered "corrupted"? 3. What can be done on our side to properly process everything?

fardeenfs commented

2022-06-19 21:38:53 +00:00

(Migrated from github.com)

Shapes.txt is an optional file (https://developers.google.com/transit/gtfs/reference#shapestxt) and a few locations have feeds that do skip them. In cases where an incorrect/corrupted and no shape is detected, wouldn't a better solution be creating a shape using the stops.txt data for those regions? I realize such routes wouldn't match the actual route the vehicle takes, but won't it still be able to generate accurate routes to the final destination as the stops remain the same?

Skipping feeds with more than 5 bad shapes ends up skipping a lot of the major Latin American cities, from what I have seen. I still haven't figured out the exact reason why there are so many shapes classified as bad, I will have to look into it further to come up with a solution.

Shapes are corrupted when the stops of a trip/line attached to a shape can't be properly projected. There are multiple reasons as to why a shape can be corrupt (incorrect plotting of a stop which makes it quite farther than the acceptable limit from a point on the shape, an incorrect point on the shape itself). Shapes.txt is an optional file (https://developers.google.com/transit/gtfs/reference#shapestxt) and a few locations have feeds that do skip them. In cases where an incorrect/corrupted and no shape is detected, wouldn't a better solution be creating a shape using the stops.txt data for those regions? I realize such routes wouldn't match the actual route the vehicle takes, but won't it still be able to generate accurate routes to the final destination as the stops remain the same? Skipping feeds with more than 5 bad shapes ends up skipping a lot of the major Latin American cities, from what I have seen. I still haven't figured out the exact reason why there are so many shapes classified as bad, I will have to look into it further to come up with a solution.

biodranik commented

2022-06-20 09:24:40 +00:00

(Migrated from github.com)

I still haven't figured out the exact reason why there are so many shapes classified as bad, I will have to look into it further to come up with a solution.

Please investigate deeper, this is important. Am I correctly understanding that skipped "corrupted" values are actually good ones, the problem is that they do not map on the existing OpenStreetMap stops?

> I still haven't figured out the exact reason why there are so many shapes classified as bad, I will have to look into it further to come up with a solution. Please investigate deeper, this is important. Am I correctly understanding that skipped "corrupted" values are actually good ones, the problem is that they do not map on the existing OpenStreetMap stops?

biodranik commented

2022-06-20 09:25:55 +00:00

(Migrated from github.com)

I also recommend to always use rebase instead of the merge for your work. Now you can try to squash all changes into the one commit to clean it up.

kalon33 commented

2022-07-31 08:03:38 +00:00

(Migrated from github.com)

Shapes are corrupted when the stops of a trip/line attached to a shape can't be properly projected. There are multiple reasons as to why a shape can be corrupt (incorrect plotting of a stop which makes it quite farther than the acceptable limit from a point on the shape, an incorrect point on the shape itself).

Shapes.txt is an optional file (https://developers.google.com/transit/gtfs/reference#shapestxt) and a few locations have feeds that do skip them. In cases where an incorrect/corrupted and no shape is detected, wouldn't a better solution be creating a shape using the stops.txt data for those regions? I realize such routes wouldn't match the actual route the vehicle takes, but won't it still be able to generate accurate routes to the final destination as the stops remain the same?

Skipping feeds with more than 5 bad shapes ends up skipping a lot of the major Latin American cities, from what I have seen. I still haven't figured out the exact reason why there are so many shapes classified as bad, I will have to look into it further to come up with a solution.

Maybe a solution would be to use pfaedle on feeds with no shapes.txt or after removing the corrupted shapes. This tool has been designed to build missing GTFS shapes using OSM data and it works pretty well according to my own experience. This would greatly enhance the number of cities covered.

> Shapes are corrupted when the stops of a trip/line attached to a shape can't be properly projected. There are multiple reasons as to why a shape can be corrupt (incorrect plotting of a stop which makes it quite farther than the acceptable limit from a point on the shape, an incorrect point on the shape itself). > > Shapes.txt is an optional file (https://developers.google.com/transit/gtfs/reference#shapestxt) and a few locations have feeds that do skip them. In cases where an incorrect/corrupted and no shape is detected, wouldn't a better solution be creating a shape using the stops.txt data for those regions? I realize such routes wouldn't match the actual route the vehicle takes, but won't it still be able to generate accurate routes to the final destination as the stops remain the same? > > Skipping feeds with more than 5 bad shapes ends up skipping a lot of the major Latin American cities, from what I have seen. I still haven't figured out the exact reason why there are so many shapes classified as bad, I will have to look into it further to come up with a solution. Maybe a solution would be to use [pfaedle](https://github.com/ad-freiburg/pfaedle) on feeds with no shapes.txt or after removing the corrupted shapes. This tool has been designed to build missing GTFS shapes using OSM data and it works pretty well according to my own experience. This would greatly enhance the number of cities covered.

biodranik commented

2022-10-19 06:34:15 +00:00

(Migrated from github.com)

Before applying a solution, let's try to understand what is at the core of the problem. Why exactly are feeds skipped? Then it will become clear what is the best approach to fix it.

islam2hamy commented

2022-10-19 09:36:08 +00:00

(Migrated from github.com)

Please test on this :
https://data.transportforcairo.com/documents/41

Please test on this : https://data.transportforcairo.com/documents/41

This repo is archived. You cannot comment on pull requests.

No reviewers

biodranik