[generator][search][routing] Long time Generator refactoring. #6537
Labels
No labels
Accessibility
Accessibility
Address
Address
Android
Android
Android Auto
Android Auto
Android Automotive (AAOS)
Android Automotive (AAOS)
API
API
AppGallery
AppGallery
AppStore
AppStore
Battery and Performance
Battery and Performance
Blocker
Blocker
Bookmarks and Tracks
Bookmarks and Tracks
Borders
Borders
Bug
Bug
Build
Build
CarPlay
CarPlay
Classificator
Classificator
Community
Community
Core
Core
CrashReports
CrashReports
Cycling
Cycling
Desktop
Desktop
DevEx
DevEx
DevOps
DevOps
dev_sandbox
dev_sandbox
Directions
Directions
Documentation
Documentation
Downloader
Downloader
Drape
Drape
Driving
Driving
Duplicate
Duplicate
Editor
Editor
Elevation
Elevation
Enhancement
Enhancement
Epic
Epic
External Map Datasets
External Map Datasets
F-Droid
F-Droid
Fonts
Fonts
Frequently User Reported
Frequently User Reported
Fund
Fund
Generator
Generator
Good first issue
Good first issue
Google Play
Google Play
GPS
GPS
GSoC
GSoC
iCloud
iCloud
Icons
Icons
iOS
iOS
Legal
Legal
Linux Desktop
Linux Desktop
Linux packaging
Linux packaging
Linux Phone
Linux Phone
Mac OS
Mac OS
Map Data
Map Data
Metro
Metro
Navigation
Navigation
Need Feedback
Need Feedback
Night Mode
Night Mode
NLnet 2024-06-281
NLnet 2024-06-281
No Feature Parity
No Feature Parity
Opening Hours
Opening Hours
Outdoors
Outdoors
POI Info
POI Info
Privacy
Privacy
Public Transport
Public Transport
Raw Idea
Raw Idea
Refactoring
Refactoring
Regional
Regional
Regression
Regression
Releases
Releases
RoboTest
RoboTest
Route Planning
Route Planning
Routing
Routing
Ruler
Ruler
Search
Search
Security
Security
Styles
Styles
Tests
Tests
Track Recording
Track Recording
Translations
Translations
TTS
TTS
UI
UI
UX
UX
Walk Navigation
Walk Navigation
Watches
Watches
Web
Web
Wikipedia
Wikipedia
Windows
Windows
Won't fix
Won't fix
World Map
World Map
No milestone
No project
No assignees
3 participants
Due date
No due date set.
Dependencies
No dependencies set.
Reference: organicmaps/organicmaps-tmp#6537
Loading…
Add table
Reference in a new issue
No description provided.
Delete branch "vng-gen-1"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
175919 lines are from OSM test data for the Generator.
The pure code lines number is ~ 8000
You can add any edits on top in a separate commit.
@ -74,0 +107,4 @@
std::vector<PtrT> vec;
vec.reserve(count);
std::greater?
@ -74,0 +113,4 @@
vec.push_back(queue.top());
queue.pop();
}
What if count == 0?
Is this log needed?
Remove?
std?
@ -24,0 +71,4 @@
/// @note Mutable function!
template <class FnT> void ForEachLocality(FnT && fn)
{
for (auto & loc : m_data)
Should it be initialized?
TODO?
Is it a hack?
Don't it make the search worse?
@ -769,1 +881,3 @@
{
CHECK(calcOrg, ());
auto const org = calcOrg(p);
if (org && s_countriesChecker.IsTransformToState(*org))
const?
Can clear be done in ApplyRules?
@ -22,0 +29,4 @@
template <class Source> void Load(Source & src, RoadAccess::Type & ac)
{
uint8_t const res = ReadPrimitiveFromSource<uint8_t>(src);
CHECK_LESS(res, uint8_t(RoadAccess::Type::Count), ());
Should this list be manually updated after adding other metadata fields?..
@ -74,0 +107,4 @@
std::vector<PtrT> vec;
vec.reserve(count);
No, there is a l->second.
The main purpose of this function is LOG: PrintTop
Good idea to get back in future.
@ -24,0 +71,4 @@
/// @note Mutable function!
template <class FnT> void ForEachLocality(FnT && fn)
{
for (auto & loc : m_data)
Default initialization is ok here.
Actually, no. But, will add todo if we will decide to process OSM's short_name.
We make 4 replacements only now, no problem here:
ApplyRules is called in other places so decided not to change. Maybe I can make it a little better.
@ -22,0 +29,4 @@
template <class Source> void Load(Source & src, RoadAccess::Type & ac)
{
uint8_t const res = ReadPrimitiveFromSource<uint8_t>(src);
CHECK_LESS(res, uint8_t(RoadAccess::Type::Count), ());
Function's overloading is used here with common notation: void Load(Source &, T &);
Well, not obvious, but I didn't have a better solution.
@ -873,0 +1040,4 @@
{"motor_vehicle", "private", [&flags] { flags[Flags::MotorVehicle] = -1; }},
{"motor_vehicle", "!", [&flags] { flags[Flags::MotorVehicle] = -1; }},
{"motor_vehicle", "yes", [&flags] { flags[Flags::MotorVehicle] = 1; }},
{"motor_vehicle", "designated", [&flags] { flags[Flags::MotorVehicle] = 1; }},
Leads to double logging, e.g.
@ -409,29 +427,35 @@ void FeatureParams::SetRwSubwayType(char const * cityName)
}
}
Did you overwrite it by accident? It was changed in
d01929a3ee
I'm worried some other more important changes could be reverted elsewhere..
@ -409,29 +427,35 @@ void FeatureParams::SetRwSubwayType(char const * cityName)
}
}
Should be deleted at all, seems like a mistake in a merge conflict.
@ -759,0 +851,4 @@
auto const countryId = m_infoGetter->GetRegionCountryId(pt);
if (!countryId.empty())
{
auto const country = storage::GetTopmostParentFor(*m_countryTree, countryId);
nit whitespace
@ -759,0 +873,4 @@
// 2. Check valid capital. We support only place-city-capital-2 (not town, village, etc).
if (isCapital && !value.empty())
value = "city";
"stronger" here and below
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
what cases did you want to fix with that change?
this could work for some cases, but could be controversial/detrimental in others...
e.g. imagine
amenity-parking covered=yes
(which implies it has some kind of roof, it doesn't imply its an underground parking), then this change will make itlayer=-1
and it'll disappear if it was mapped over e.g. a park or industrial landuse etc.is it possible it'd create a duplicate tag if it existed already?
@ -162,3 +153,1 @@
// TODO : re-checks geom limit rect size via IsDrawableForIndexGeometryOnly() which was checked already in CalculateMidPoints.
if (fb.IsDrawableInRange(scales::PatchMinDrawableScale(i > 0 ? m_header.GetScale(i - 1) + 1 : 0),
scales::PatchMaxDrawableScale(level)))
bool const isLine = fb.IsLine();
why did remove the message?
just looking at the code its not obvious it checks for geometry types (need to look into isLine()/isArea() impl)
won't it be better to make coasts' trg0 fallback to trg1 always?
or explicitly keep only 3 geom levels for coasts (expand trg0 to
[1-5]
instead of[1-3]
)maybe just leave a todo comment for now if you agree
wrong indent left
@ -62,20 +76,38 @@ bool RelationTagsWay::IsAcceptBoundary(RelationElement const & e) const
{
// Do not accumulate boundary types (boundary=administrative) for inner polygons.
Would be good to add some examples of "skipped useful linear tags", otherwise its not clear what is it about.
nit: "according to the the drule's geometry type"
@ -130,2 +130,4 @@
}
/// @todo Should we also take into account "none", "false" here?
remove this newline, otherwise its not very clear to which block the comment belongs
maybe it'd be better to remove all useless
;name;int_name;
in one go, as single changes like this can confuse people?why smoothness is ignored for trunks & motorways?
there could be even unpaved ones
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
Probably this one organicmaps/organicmaps#2564
Can't say for sure if it is really needed.
Maybe, but no problem here.
@ -162,3 +153,1 @@
// TODO : re-checks geom limit rect size via IsDrawableForIndexGeometryOnly() which was checked already in CalculateMidPoints.
if (fb.IsDrawableInRange(scales::PatchMinDrawableScale(i > 0 ? m_header.GetScale(i - 1) + 1 : 0),
scales::PatchMaxDrawableScale(level)))
bool const isLine = fb.IsLine();
Seems like merging conflicts. Well, check body and context is more than enough here, no? "масло масленное" otherwise.
@ -162,3 +153,1 @@
// TODO : re-checks geom limit rect size via IsDrawableForIndexGeometryOnly() which was checked already in CalculateMidPoints.
if (fb.IsDrawableInRange(scales::PatchMinDrawableScale(i > 0 ? m_header.GetScale(i - 1) + 1 : 0),
scales::PatchMaxDrawableScale(level)))
bool const isLine = fb.IsLine();
Well, trg0 -> 1 maybe better, but maybe a bit slow. With increments, current simplification zooms are:
{5, 6, 8, 10}.
@ -62,20 +76,38 @@ bool RelationTagsWay::IsAcceptBoundary(RelationElement const & e) const
{
// Do not accumulate boundary types (boundary=administrative) for inner polygons.
Just after the comment, we skip tags inheritance for the multipolygon relation (). Added barrier as an exception, but suppose that there are more tags like this, when ways should inherit from parent relation.
@ -130,2 +130,4 @@
}
/// @todo Should we also take into account "none", "false" here?
Looks good here IMO, since the next 2 separated blocks below process "no".
"surface" tag is not ignored, but smoothness is controversial in the examples I saw. There are a lot of them in Ukraine, where somebody tagged all main roads like smoothness=intermediate and the app prefers nearby primary/secondary roads without tags, which are definitely not better than trunk/motorway.
The purpose is to keep readable diff in this not obvious block.
the diff is bogus also, that's how I did notice this mistake :)

and the result is obviously very bad readability and comprehension-wise:
This sounds a bit like mismapping and OM helping to legalise it :)
I think its fine if OM's behavior is consistent with other routers. Is it?
And in any case add a comment describing a reason for that exception please.
@ -228,1 +221,4 @@
LOG(LDEBUG, ("Area: too small or degenerate 1st (outer) polygon of", polys.size(),
", points count", points.size(), "at scale", i, DebugPrint(fb)));
continue;
}
with the latest commit the most of this method's lines are changed anyway, so maybe its better to re-indent the block properly;
or
@ -162,3 +153,1 @@
// TODO : re-checks geom limit rect size via IsDrawableForIndexGeometryOnly() which was checked already in CalculateMidPoints.
if (fb.IsDrawableInRange(scales::PatchMinDrawableScale(i > 0 ? m_header.GetScale(i - 1) + 1 : 0),
scales::PatchMaxDrawableScale(level)))
bool const isLine = fb.IsLine();
take out of the loop too?
so it makes trg0 coasts in country mwms unnecessary as they duplicate trg3 in WorldCoasts :) might save a few kilobytes someday in the future :)
The last commit is a nice change, makes the code easier to understand!
@ -162,3 +153,1 @@
// TODO : re-checks geom limit rect size via IsDrawableForIndexGeometryOnly() which was checked already in CalculateMidPoints.
if (fb.IsDrawableInRange(scales::PatchMinDrawableScale(i > 0 ? m_header.GetScale(i - 1) + 1 : 0),
scales::PatchMaxDrawableScale(level)))
bool const isLine = fb.IsLine();
also am I right that
fb.IsCoastCell();
is true not only for coastline features but for any other feature residing in the same "cell"?@ -205,0 +175,4 @@
if (level <= scales::GetUpperWorldScale())
++level;
if (i == 0)
++level;
is it possible there are several outer polygons?
or they're split into individual features?
Just to make it more obvious
I agree. We can't store several versions of inner triangles so no point to try simplify and process outlines for lower zooms.
Makes sense! But its a good idea indeed to first try log such cases and see (might be cases when bad simplification made the outer degenerate while inners came out fine?).
and similarly below when processing inner polygons
OSM routers work fine here. Mapping also satisfies OSM rules, I think. The big problem here (and we have TODOs and issues) is that we try to fit all OSM surface and smoothness tags into 4 OM types:
@ -162,3 +153,1 @@
// TODO : re-checks geom limit rect size via IsDrawableForIndexGeometryOnly() which was checked already in CalculateMidPoints.
if (fb.IsDrawableInRange(scales::PatchMinDrawableScale(i > 0 ? m_header.GetScale(i - 1) + 1 : 0),
scales::PatchMaxDrawableScale(level)))
bool const isLine = fb.IsLine();
No, only for coastlines. I think we can (try to) remove this check since we increased the coast's precision.
@ -205,0 +175,4 @@
if (level <= scales::GetUpperWorldScale())
++level;
if (i == 0)
++level;
Yes, that is the point! Outer polygon is 1st, others are inners and I can't remember why it was done like this.
We now split multiple outers into several FeatureBuilders. And it is also controversial logic :) because we have several park features (for example) with names where it is one logical feature.
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
I think its better to remove it for now.
This tag is very tricky and depending on the main feature type it should be processed differently.
We don't need
building-address
added toentrance*
types at all.Housenumbers (if present) are displayed above entrance's icons just like with any other POI.
and actually ATM presence of
building-address
hides entrance's icons in vehicle style, becausebuilding-address
has a higher prio there (we wanted to prioritize housenumbers in nav mode compared to other micromapping)ref
/rank
should be cleared too? onlyhouse
should remain, right?it'd also be good to add the
was:
prefix, around 90k uses roughly@ -478,6 +480,7 @@ void FeatureBuilder::SerializeAccuratelyForIntermediate(Buffer & data) const
But you made it displayed in PP in
f7db63b959
, no?fixed typos:
Actually looking at https://wiki.openstreetmap.org/wiki/Lifecycle_prefix we should avoid displaying other prefixed POIs names like
Ok, did you test it locally? Remove adding for "entrance" and see how it is drawn. Because all our maps generation was with this change.
@ -478,6 +480,7 @@ void FeatureBuilder::SerializeAccuratelyForIntermediate(Buffer & data) const
Yes, we remove duplicates. Don't store a brand if it is equal to a name.
Will add "was:". The other list is not about POIs, IMO ..
organicmaps/organicmaps#6537/commits/b3c4104bbdec795da24773da1bedf22b472a47b4
I double-checked - it works fine!
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
Now I remember, we had an issue with a building and a highway. Most of the covered tags are for the highways under buildings.
You can check locally here for example: https://www.openstreetmap.org/#map=19/48.14226/11.55020
Before and after. Probably, many things has changed since our styles refactoring, but anyway ..
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
For this particular case adding
layer=-1
changes nothing in the main 3D buildings mode.But hides the "covered" roads completely in 2D.
Which is not nice neither for comprehension nor for navigation, especially given that some footways are rendered over the building still (they don't have
covered=yes
).IMHO in this case we should show covered roads as tunnels (but w/o a "Tunnel" title as it'll be misleading).
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
Yes, hiding highways is not good. Same as I don't like hiding highways under the bridge.
@ -588,0 +638,4 @@
// Check surface.
if (surface.empty())
{
CHECK(!smoothness.empty(), ());
Removed
@vng how to review your newer changes? It would be easier if they came in separate commits. Now I can't review everything again. Approving blindly.
ACK. LGTM
const
@ -873,0 +1040,4 @@
{"motor_vehicle", "private", [&flags] { flags[Flags::MotorVehicle] = -1; }},
{"motor_vehicle", "!", [&flags] { flags[Flags::MotorVehicle] = -1; }},
{"motor_vehicle", "yes", [&flags] { flags[Flags::MotorVehicle] = 1; }},
{"motor_vehicle", "designated", [&flags] { flags[Flags::MotorVehicle] = 1; }},
Please add total types count, like it was before (otherwise have to count manually for each line - very inconvenient).
LOG(LWARNING, ("Exceeded max types count, got total", m_types.size(), ":", TypesToString(m_types)));
@ -409,29 +427,35 @@ void FeatureParams::SetRwSubwayType(char const * cityName)
}
}
But total types count was very useful, please add to the new log line.
Typo in "juctions".
Overall its hard to understand this comment. I had to read three times :)
My understanding: in England there are many
junction=circular
which are oneway for sure (how do we know? looking at aerial images?), but they are not tagged as such, hence we forceoneway=yes
explicitly?Yes, we had several user claims. I checked some more "circulars" manually. They are obvious oneway (connected roads are mapped as oneway). I didn't find any circular that is not oneway. Even so, false-positive (put oneway while it is not) doesn't break routing. While false-negative makes really bad routes.
@ -1028,0 +1302,4 @@
// Зеленоград
m_rects.emplace_back(37.119113, 55.944925, 37.273608, 56.026874);
// Add new {lon, lat} city bboxes here.
nit: please put
typesCount =
under the if@ -1028,0 +1302,4 @@
// Зеленоград
m_rects.emplace_back(37.119113, 55.944925, 37.273608, 56.026874);
// Add new {lon, lat} city bboxes here.
But the purpose is to know the initial exceeded types count. Under the "if" it will be already truncated = 8.
@ -1028,0 +1302,4 @@
// Зеленоград
m_rects.emplace_back(37.119113, 55.944925, 37.273608, 56.026874);
// Add new {lon, lat} city bboxes here.
Right, I've missed it
How do you think to manage experimental generator changes from now on?
Maybe dedicated a
generator
branch for this?