Move OSM authentication to OAuth 2.0

pnorman commented

2023-09-27 07:36:12 +00:00

(Migrated from github.com)

Is your feature request related to a problem? Please describe.
OAuth 1.0a is deprecated, and programs which use it to authenticate requests to OpenStreetMap should move to OAuth 2.0. A date has not yet been set for turning off OAuth 1.0a and HTTP Basic.

It's best to move to OAuth 2.0 well in advance of any turn-off, because users may take some time to upgrade software and to re-authenticate.

See https://github.com/openstreetmap/operations/issues/867 for details.

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

**Is your feature request related to a problem? Please describe.** OAuth 1.0a is deprecated, and programs which use it to authenticate requests to OpenStreetMap should move to OAuth 2.0. A date has not yet been set for turning off OAuth 1.0a and HTTP Basic. It's best to move to OAuth 2.0 well in advance of any turn-off, because users may take some time to upgrade software and to re-authenticate. See https://github.com/openstreetmap/operations/issues/867 for details.  ## Upvote & Fund - We're using [Polar.sh](https://polar.sh/organicmaps) so you can upvote and help fund this issue. - We receive the funding once the issue is completed & confirmed by you. - Thank you in advance for helping prioritize & fund our backlog. <a href="https://polar.sh/organicmaps/organicmaps/issues/6144"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://polar.sh/api/github/organicmaps/organicmaps/issues/6144/pledge.svg?darkmode=1"> <img alt="Fund with Polar" src="https://polar.sh/api/github/organicmaps/organicmaps/issues/6144/pledge.svg"> </picture> </a>

pnorman commented

2023-09-27 07:36:31 +00:00

(Migrated from github.com)

mentioned in issue #867

mmdosm commented

2023-12-23 10:59:15 +00:00

(Migrated from github.com)

I think you're using OrganicMapsTestUser for your Unit tests on master.apis.dev.openstreetmap.org. Are you using Basic Authentication or OAuth 1.0a there?

I need to know if we have to take additional steps before merging the OAuth 1.0a removal in CGImap. Since we have an automated deployment in place, that would immediately break OAuth 1.0a for changeset uploads on the dev instance.
Our goal is to avoid any disruptive and unexpected changes on the dev instance.

Relevant commit is: https://github.com/zerebubuth/openstreetmap-cgimap/pull/354

NB: Production is not impacted by any of this at this point in time.

I think you're using `OrganicMapsTestUser` for your Unit tests on master.apis.dev.openstreetmap.org. Are you using Basic Authentication or OAuth 1.0a there? I need to know if we have to take additional steps before merging the OAuth 1.0a removal in CGImap. Since we have an automated deployment in place, that would immediately break OAuth 1.0a for changeset uploads on the dev instance. Our goal is to avoid any disruptive and unexpected changes on the dev instance. Relevant commit is: https://github.com/zerebubuth/openstreetmap-cgimap/pull/354 NB: Production is not impacted by any of this at this point in time.

biodranik commented

2023-12-23 11:43:32 +00:00

(Migrated from github.com)

The same OAuth1.0a as in production is used in debugging/CI on the dev server. Do you have any recommendations or examples on how to properly migrate the manual auth code?

mmdosm commented

2023-12-23 12:22:17 +00:00

(Migrated from github.com)

The same OAuth1.0a as in production is used in debugging/CI on the dev server.

Thank you for your quick reply. I will discuss this point with our sysadmins, to make sure that we're not breaking anything here.

Do you have any recommendations or examples on how to properly migrate the manual auth code?

My recommendation here would be very clear to reach out to both @tsmock (JOSM) and @westnordost (StreetComplete). They have either already completed the switch to OAuth 2.0, or are currently in the process of rolling out OAuth 2.0 as part of a beta version. I hope they're ok to mention them here, since I believe they could provide very valuable insights.

Are you looking for solutions for both Java and C++? JOSM and StreetComplete should cover Java/Kotlin. @tsmock also had some discussions with Merkaartor, a C++ app. I'm not sure if this would be helpful in your case.

You can of course also post to https://github.com/openstreetmap/operations/issues/867, which is the central issue to manage the transition. Besides all admins/core maintainers I would expect that you could reach out to a larger group of app developers this way.

Linking to a few relevant issues/PRs.

> The same OAuth1.0a as in production is used in debugging/CI on the dev server. Thank you for your quick reply. I will discuss this point with our sysadmins, to make sure that we're not breaking anything here. > Do you have any recommendations or examples on how to properly migrate the manual auth code? My recommendation here would be very clear to reach out to both `@tsmock` (JOSM) and `@westnordost` (StreetComplete). They have either already completed the switch to OAuth 2.0, or are currently in the process of rolling out OAuth 2.0 as part of a beta version. I hope they're ok to mention them here, since I believe they could provide very valuable insights. Are you looking for solutions for both Java and C++? JOSM and StreetComplete should cover Java/Kotlin. `@tsmock` also had [some discussions](https://github.com/openstreetmap/merkaartor/issues/286) with Merkaartor, a C++ app. I'm not sure if this would be helpful in your case. You can of course also post to https://github.com/openstreetmap/operations/issues/867, which is the central issue to manage the transition. Besides all admins/core maintainers I would expect that you could reach out to a larger group of app developers this way. Linking to a few relevant issues/PRs. * https://josm.openstreetmap.de/ticket/22810 * https://github.com/westnordost/osmapi/issues/33 * https://github.com/westnordost/osmapi/pull/35/files

westnordost commented

2023-12-23 12:53:53 +00:00

(Migrated from github.com)

Hey @biodranik , I summarized it here:

https://github.com/MarcusWolschon/osmeditor4android/issues/2401#issuecomment-1858819801

Feel free to ask me questions, I very recently concerned myself with it so my knowledge of it is fresh.

In a nutshell, using an OAuth2 access token is as simple as adding Authorization: Bearer <the OAuth2 access token> in the header of each HTTPS request. For getting the authorization token, the flow is similar but not exactly the same as for OAuth 1.0a.

Hey `@biodranik` , I summarized it here: https://github.com/MarcusWolschon/osmeditor4android/issues/2401#issuecomment-1858819801 Feel free to ask me questions, I very recently concerned myself with it so my knowledge of it is fresh. In a nutshell, using an OAuth2 access token is as simple as adding `Authorization: Bearer <the OAuth2 access token>` in the header of each HTTPS request. For getting the authorization token, the flow is similar but not exactly the same as for OAuth 1.0a.

biodranik commented

2023-12-24 02:33:41 +00:00

(Migrated from github.com)

Thanks, @westnordost, that is helpful. The problem is that liboauthcpp used in OM does not support OAuth2: https://github.com/sirikata/liboauthcpp/issues/24

Implementing the spec manually in C++ is not an easy task. If anyone is aware of simple and lightweight C++ implementations without networking code, please let us know.

Thanks, `@westnordost`, that is helpful. The problem is that liboauthcpp used in OM does not support OAuth2: https://github.com/sirikata/liboauthcpp/issues/24 Implementing the spec manually in C++ is not an easy task. If anyone is aware of simple and lightweight C++ implementations without networking code, please let us know.

westnordost commented

2023-12-24 09:57:44 +00:00

(Migrated from github.com)

You don't need a library for passing the access token. For getting the access token, ... maybe, but it is really not a lot. I implemented the logic in ~120 source lines of code, so that's the order of magnitude of code you would like to have outsourced into a lib. (The links to the source code are contained in my previously linked comment.)

For getting the access token, indeed to implement the whole spec would be quite a large undertaking, so an OAuth2 library that does that would not be lightweight.

Also, note that to get the access token at the end, the result from the server must be parsed as a JSON. So, a library that does all that (120 source lines of code worth of logic) for you will very likely pull in another dependency to a JSON parsing library, i.e. maybe a different one that you are using for other stuff. Not to mention the dependency to actually do HTTP requests. Oh, and, and optional feature that is nevertheless supported by OSM and recommended to be used is to include a PKCE key exchange in the process, which means a library that implements that (I think it is pretty standard these days) also needs a dependency to some sha256 hasher.

I am writing this because you mentioned that it should be lightweight. If you are looking for a library without networking code and without JSON parsing code, to be honest, there is not much code left. The exchange that happens to get the access token is basically nothing else than a bunch of HTTPS requests and reading the responses.

You don't need a library for passing the access token. For getting the access token, ... maybe, but it is really not a lot. I implemented the logic in ~120 source lines of code, so that's the order of magnitude of code you would like to have outsourced into a lib. (The links to the source code are contained in my previously linked comment.) For getting the access token, indeed to implement the whole spec would be quite a large undertaking, so an OAuth2 library that does that would **not be lightweight**. Also, note that to get the access token at the end, the result from the server **must** be parsed as a JSON. So, a library that does all that (120 source lines of code worth of logic) for you will very likely pull in another dependency to a JSON parsing library, i.e. maybe a different one that you are using for other stuff. Not to mention the dependency to actually do HTTP requests. Oh, and, and optional feature that is nevertheless supported by OSM and recommended to be used is to include a PKCE key exchange in the process, which means a library that implements that (I think it is pretty standard these days) also needs a dependency to some sha256 hasher. I am writing this because you mentioned that it should be lightweight. If you are looking for a library without networking code and without JSON parsing code, to be honest, there is not much code left. The exchange that happens to get the access token is basically nothing else than a bunch of HTTPS requests and reading the responses.

westnordost commented

2023-12-24 10:05:52 +00:00

(Migrated from github.com)

(I edited my post significantly to include more information)

rtsisyk commented

2023-12-28 07:29:29 +00:00

Owner

mentioned in issue #7000

rtsisyk commented

2023-12-28 07:30:02 +00:00

Owner

This issue is getting more important because of #7000

This issue is getting more important because of https://git.omaps.dev/organicmaps/organicmaps/issues/7000

rtsisyk commented

2023-12-28 08:02:48 +00:00

Owner

Probably it will be easier to implement auth in Java and Swift separately instead of doing this business in C++.

biodranik commented

2023-12-28 14:00:14 +00:00

(Migrated from github.com)

@rtsisyk as it was mentioned by @westnordost and @Zverik, the decision to use login and password was a deliberate one, forced by issues with using Webview or browser login.

To clarify: the existing OAuth1.0a solution was using webview from the start. And it can be adopted to using external web browser if necessary without migrating to OAuth2.

`@rtsisyk` as it was mentioned by `@westnordost` and `@Zverik`, the decision to use login and password was a deliberate one, forced by issues with using Webview or browser login. To clarify: the existing OAuth1.0a solution was using webview from the start. And it can be adopted to using external web browser if necessary without migrating to OAuth2.

rtsisyk commented

2023-12-29 07:55:05 +00:00

Owner

@rtsisyk as it was mentioned by @westnordost and @Zverik, the decision to use login and password was a deliberate one, forced by issues with using Webview or browser login.

I didn't get the problem with WebView. OAuth2 is definitely better than asking the user to enter their password.

> `@rtsisyk` as it was mentioned by `@westnordost` and `@Zverik`, the decision to use login and password was a deliberate one, forced by issues with using Webview or browser login. I didn't get the problem with WebView. OAuth2 is definitely better than asking the user to enter their password.

biodranik commented

2023-12-29 13:28:23 +00:00

(Migrated from github.com)

@rtsisyk
https://developers.googleblog.com/2016/08/modernizing-oauth-interactions-in-native-apps.html

https://developers.googleblog.com/2021/06/upcoming-security-changes-to-googles-oauth-2.0-authorization-endpoint.html

`@rtsisyk` https://developers.googleblog.com/2016/08/modernizing-oauth-interactions-in-native-apps.html https://developers.googleblog.com/2021/06/upcoming-security-changes-to-googles-oauth-2.0-authorization-endpoint.html

westnordost commented

2023-12-30 21:23:28 +00:00

(Migrated from github.com)

As far as I know it is easy to circumvent this restriction by setting the user agent of the WebView to something other than the default. (You could test this by trying to login with OAuth with Google on OSM within STreetComplete.)

But anyway, using the browser is probably the better idea anyway. Though, expect issue tickets to appear that complain about that the login doesn't work on their browser or maybe even the default browser of certain Android distributions such as LineageOS (last time I checked).

Also, the information may be outdated, but back when I looked into the topic, the recommendation (or even obligation?) for apps on the Apple App store was the exact opposite, "to not break the native experience".

But take my comment with a grain of salt, because this was quite some years ago. Browser support may have been fixed, Apple may have either changed their policies or I misremember it in the first place.

As far as I know it is easy to circumvent this restriction by setting the user agent of the WebView to something other than the default. (You could test this by trying to login with OAuth with Google on OSM within STreetComplete.) But anyway, using the browser is probably the better idea anyway. Though, expect issue tickets to appear that complain about that the login doesn't work on their browser or maybe even the default browser of certain Android distributions such as LineageOS (last time I checked). Also, the information may be outdated, but back when I looked into the topic, the recommendation (or even obligation?) for apps on the Apple App store was the exact opposite, "to not break the native experience". But take my comment with a grain of salt, because this was quite some years ago. Browser support may have been fixed, Apple may have either changed their policies or I misremember it in the first place.

biodranik commented

2023-12-30 21:26:23 +00:00

(Migrated from github.com)

As far as I know it is easy to circumvent this restriction by setting the user agent of the WebView to something other than the default. (You could test this by trying to login with OAuth with Google on OSM within STreetComplete.)

Thanks for the hint!

Apple is ok with the app login. It's a Google Play issue only.

CC @rtsisyk

> As far as I know it is easy to circumvent this restriction by setting the user agent of the WebView to something other than the default. (You could test this by trying to login with OAuth with Google on OSM within STreetComplete.) Thanks for the hint! Apple is ok with the app login. It's a Google Play issue only. CC `@rtsisyk`

mmdosm commented

2023-12-30 21:53:17 +00:00

(Migrated from github.com)

OSM OWG mentioned the following in their API usage policy: https://operations.osmfoundation.org/policies/api/

Do not submit website forms in an automated manner or on behalf of users. Website forms are intended only for users interacting directly with the website.

I’m not exactly sure how you have implemented app login, and if this would impact you. I thought it might be good to double check this point, and reach out to OWG in case of further questions.

OSM OWG mentioned the following in their API usage policy: https://operations.osmfoundation.org/policies/api/ _Do not submit website forms in an automated manner or on behalf of users. Website forms are intended only for users interacting directly with the website._ I’m not exactly sure how you have implemented app login, and if this would impact you. I thought it might be good to double check this point, and reach out to OWG in case of further questions.

RedAuburn commented

2023-12-30 21:59:01 +00:00

Member

Do not submit website forms in an automated manner or on behalf of users. Website forms are intended only for users interacting directly with the website.

the registration button takes you to the OSM site, you can't create an account in-app.
The login is using OAuth, not the website forms 👍️

> _Do not submit website forms in an automated manner or on behalf of users. Website forms are intended only for users interacting directly with the website._ the registration button takes you to the OSM site, you can't create an account in-app. The login is using OAuth, not the website forms 👍️

biodranik commented

2023-12-30 21:59:43 +00:00

(Migrated from github.com)

...And the login is not automated, users are doing it manually.

rtsisyk commented

2024-02-01 12:07:06 +00:00

Owner

We are looking for Android engineers to implement new OSM authorization in the app. This work will be financially rewarded based on fair estimation of time and efforts spent. Please post your proposal in this thread.

westnordost commented

2024-02-01 13:26:50 +00:00

(Migrated from github.com)

This is not a proposal, just a few comments after a quick look at the source code in order to help make clear the scope of it:

The biggest part of the current OAuth 1.0a authentication is implemented in osm_oauth.cpp.

The normal OAuth authorization flow (OAuth 1.0a and OAuth 2.0 are quite similar) would be to

get the request token
open the authorize URL on osm.org (passing the request token) in a webview or browser, let user login there and let him click on the authorize-button so that the verifier is sent back to the app via the callback URL
get the verifier from the callback URL parameter and use that to get the access token

OM is currently using the following curious approach to OAuth authorization instead. As far as I understand:

In its own UI, let user enter username+password, or alternatively through Google, Facebook, ...
act like a web browser that logs in to osm.org/login with these credentials and save the session cookie
get the request token as normal
act like a web browser that opens the authorize URL on osm.org (passing the request token and with the session cookie set) and extract the verifier from the link that would open when one clicked the authorize-button in the HTML
get the verifier from the callback URL parameter and use that to get the access token

I believe it is done that way to avoid the user leaving the app's UI for authorization but certainly is somewhat of a misuse of OAuth. Could just as well use HTTP Basic Auth, then. However, this is akin to how JOSM did it in the past, as far as I know.
But anyway, this is the reason why there is so much code.

Obviously, I strongly recommend just doing the normal OAuth 2.0 authorization flow, which is akin to the OAuth 1.0a but even a little simpler (actually). It is less code, it is more trustworthy and more future-proof (less chance that something breaks, when e.g. the HTML on osm.org changes). Google additionally requires (but it can be tricked) to use the browser instead of a WebView.
Of course, it is up for OM folks to decide how it should be implemented.

Note that OAuth 2.0 requires some basic JSON parsing, as the access token response is in JSON.

The Android part of the Login is in OsmLoginFragment.java and OsmOAuth.java, which is naturally almost empty because it does almost nothing else than provide the UI for entering the username+password and initiate the authorization in native code.

This also means that to implement OAuth 2.0 authorization only for Android but not for iOS makes no sense, as the main part is implemented in native code.

For further help on implementing OAuth 2.0, see https://github.com/MarcusWolschon/osmeditor4android/issues/2401#issuecomment-1858819801 , it also includes links to implementation.

*This is not a proposal, just a few comments after a quick look at the source code in order to help make clear the scope of it:* The biggest part of the current OAuth 1.0a authentication is implemented in [osm_oauth.cpp](https://github.com/organicmaps/organicmaps/blob/master/editor/osm_auth.cpp). The normal OAuth authorization flow (OAuth 1.0a and OAuth 2.0 are quite similar) would be to 1. get the request token 2. open the authorize URL on osm.org (passing the request token) in a webview or browser, let user login there and let him click on the authorize-button so that the verifier is sent back to the app via the callback URL 3. get the verifier from the callback URL parameter and use that to get the access token OM is currently using the following curious approach to OAuth authorization instead. As far as I understand: 1. In its own UI, let user enter username+password, or alternatively through Google, Facebook, ... 2. act like a web browser that logs in to osm.org/login with these credentials and save the session cookie 3. get the request token as normal 4. act like a web browser that opens the authorize URL on osm.org (passing the request token and with the session cookie set) and extract the verifier from the link that would open when one clicked the authorize-button in the HTML 5. get the verifier from the callback URL parameter and use that to get the access token I believe it is done that way to avoid the user leaving the app's UI for authorization but certainly is somewhat of a misuse of OAuth. Could just as well use HTTP Basic Auth, then. However, this is akin to how JOSM did it in the past, as far as I know. But anyway, this is the reason why there is so much code. Obviously, I strongly recommend just doing the normal OAuth 2.0 authorization flow, which is akin to the OAuth 1.0a but even a little simpler (actually). It is less code, it is more trustworthy and more future-proof (less chance that something breaks, when e.g. the HTML on osm.org changes). Google additionally requires (but it can be tricked) to use the browser instead of a WebView. Of course, it is up for OM folks to decide how it should be implemented. Note that OAuth 2.0 requires some basic JSON parsing, as the access token response is in JSON. The Android part of the Login is in [OsmLoginFragment.java](https://github.com/organicmaps/organicmaps/blob/master/android/app/src/main/java/app/organicmaps/editor/OsmLoginFragment.java) and [OsmOAuth.java](https://github.com/organicmaps/organicmaps/blob/master/android/app/src/main/java/app/organicmaps/editor/OsmOAuth.java), which is naturally almost empty because it does almost nothing else than provide the UI for entering the username+password and initiate the authorization in native code. This also means that to implement OAuth 2.0 authorization only for Android but not for iOS makes no sense, as the main part is implemented in native code. For further help on implementing OAuth 2.0, see https://github.com/MarcusWolschon/osmeditor4android/issues/2401#issuecomment-1858819801 , it also includes links to implementation.

rtsisyk commented

2024-02-06 08:46:26 +00:00

Owner

assigned to @strump

assigned to `@strump`

mmdosm commented

2024-02-18 15:25:22 +00:00

(Migrated from github.com)

mentioned in issue #7426

RedAuburn commented

2024-03-04 16:31:34 +00:00

Member

mentioned in merge request !7478

rtsisyk commented

2024-03-28 15:40:28 +00:00

Owner

I am removing this ticket from Release Blockers because it is not a blocker and it is actually going into the subsequent release (after the current one).

pastk commented

2024-04-11 15:59:38 +00:00

Member

Done in #7333 thanks to @strump !

Done in https://git.omaps.dev/organicmaps/organicmaps/pulls/7333 thanks to `@strump` !

RedAuburn closed this issue

2024-04-11 17:37:45 +00:00

Rows
Columns

Move OSM authentication to OAuth 2.0 #6144

Upvote & Fund