Automate F-Droid updates and metadata

rugk commented

2021-07-09 12:11:23 +00:00

(Migrated from github.com)

Follow-up of #33

As I said in organicmaps/organicmaps#33 (comment):

AFAIK you can choose another update mode based on git tags, e.g. instead, if that better suits.

I.e. you can change the update mode to automatically update based on tag changes instead of source code stuff.

Or let's find another way to automate these updates.
@relan said you do this manually, but without wanting to deny your commitment, I fear this does not scale, especially for the amount of fast updates you do here. That's why I continue to propose automatic updates, then the maintenance burden of that F-Droid… I don't get why you all call for people to do that, just let machines do it, they are far more reliable! 🙃

Ideas

use the tag thing based above
Also, AFAIK you can use a fastlane structure for your metadata in your GitHub repo and thus automate releasing that (with changelogs etc.) here too (plus fastlane could be used for other stores like Google Play).
More information on how to do this on this page.

/cc @IzzySoft as he knows more stuff about that, I'm just an F-Droid user

Follow-up of #33 As I said in https://git.omaps.dev/organicmaps/organicmaps/issues/33#issuecomment-876349674: > AFAIK you can [choose another update mode](https://f-droid.org/en/docs/Build_Metadata_Reference/#UpdateCheckMode) based on git tags, e.g. instead, if that better suits. I.e. you can change the update mode to automatically update based on tag changes instead of source code stuff. Or let's find another way to automate these updates. @relan said you do this manually, but without wanting to deny your commitment, I fear this does not scale, especially for the amount of fast updates you do here. That's why I continue to propose **automatic updates**, then the maintenance burden of that F-Droid… I don't get why you all call for people to do that, just let machines do it, they are far more reliable! :upside_down_face: ## Ideas * use the tag thing based above * Also, AFAIK you can use a fastlane structure for your metadata in your GitHub repo and thus automate releasing that (with changelogs etc.) here too (plus fastlane could be used for other stores like Google Play). More information on how to do this [on this page](https://fdroid.gitlab.io/fdroid-website/docs/All_About_Descriptions_Graphics_and_Screenshots/). /cc @IzzySoft as he knows more stuff about that, I'm just an F-Droid user

👍 2

IzzySoft commented

2021-07-09 19:39:26 +00:00

(Migrated from github.com)

I fully agree this is better automated – as manual updates unnecessarily bind resources on both ends, also delaying updates to show up on F-Droid as you'll always have to wait until a maintainer finds time to review and merge.

Well, the versionName is a bit crazy and will make that hard. Example:

tag: 2021-06-23
versionName: 2021.06.23-9-gc3f05a1646-FDroid
versionCode: 21062309

No way to map that. We can do with a fixed pre- or suffix (like 2021-06-23-FDroid), but not with a dynamic one (here: including the commit hash). We might be able to skip the tag name and specify a file to look at – but your build.gradle does not even mention the literal versionCode and versionName (I guess you dynamically calculate them at build time – that won't work with auto-updates at F-Droid). So we'd need at least a file containing the literal versionName & versionCode to get this running. Your current declaration (calculating the values) won't work; further, it's not reproducible as it depends on the day the actual build is done; so if we need to rebuild for some reason, versionName and versionCode would change. That's no good.

I fully agree this is better automated – as manual updates unnecessarily bind resources on both ends, also delaying updates to show up on F-Droid as you'll always have to wait until a maintainer finds time to review and merge. Well, the `versionName` is a bit crazy and will make that hard. Example: * tag: 2021-06-23 * versionName: 2021.06.23-9-gc3f05a1646-FDroid * versionCode: 21062309 No way to map that. We can do with a fixed pre- or suffix (like `2021-06-23-FDroid`), but not with a dynamic one (here: including the commit hash). We might be able to skip the tag name and specify a file to look at – but your `build.gradle` does not even mention the literal `versionCode` and `versionName` (I guess you dynamically calculate them at build time – that won't work with auto-updates at F-Droid). So we'd need at least a file containing the literal versionName & versionCode to get this running. Your [current declaration](https://github.com/organicmaps/organicmaps/blob/master/android/build.gradle#L133) (calculating the values) won't work; further, it's not reproducible as it depends on the day the actual build is done; so if we need to rebuild for some reason, versionName and versionCode would change. That's no good.

👍 1

rugk commented

2021-07-09 19:43:03 +00:00

(Migrated from github.com)

Thanks, @IzzySoft, what is with another UpdateCheckMode? I.e. can't F-Droid use Tags there and instead check the git repository for a new tag?

Thanks, @IzzySoft, what is with another [UpdateCheckMode](https://f-droid.org/en/docs/Build_Metadata_Reference/#UpdateCheckMode)? I.e. can't F-Droid use `Tags` there and instead check the git repository for a new tag?

biodranik commented

2021-07-09 20:01:47 +00:00

(Migrated from github.com)

Our builds/versions are reproducible. The date tag is generated from the last commit's UTC date, not from the date of the build.

IzzySoft commented

2021-07-09 20:03:44 +00:00

(Migrated from github.com)

Yes, take a look at UpdateCheckData. Since a recent update, that can be used in conjunction with tags and files inside the repo itself. Example block from a Flutter app:

AutoUpdateMode: Version
UpdateCheckMode: Tags
UpdateCheckData: pubspec.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+
CurrentVersion: 0.0.11
CurrentVersionCode: 11

Note that neither AutoUpdateMode nor UpdateCheckMode contain a tag name pattern here.

The date tag is generated from the last commit's UTC date

Ah, OK – my bad (I'm not an Android dev, so please forgive my misreading). Still, we'll need literal values, as CheckUpdates does not execute gradle but just performs RegEx matching.

Yes, take a look at [`UpdateCheckData`](https://f-droid.org/en/docs/Build_Metadata_Reference/#UpdateCheckData). Since a recent update, that can be used in conjunction with tags and files inside the repo itself. Example block from a Flutter app: ```yaml AutoUpdateMode: Version UpdateCheckMode: Tags UpdateCheckData: pubspec.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+ CurrentVersion: 0.0.11 CurrentVersionCode: 11 ``` Note that neither `AutoUpdateMode` nor `UpdateCheckMode` contain a tag name pattern here. > The date tag is generated from the last commit's UTC date Ah, OK – my bad (I'm not an Android dev, so please forgive my misreading). Still, we'll need literal values, as CheckUpdates does not execute gradle but just performs RegEx matching.

biodranik commented

2021-07-09 20:06:19 +00:00

(Migrated from github.com)

That's the reason why we didn't yet automate builds with F-Droid :) Any solution which won't break our existing workflow for Google Play is appreciated.

IzzySoft commented

2021-07-09 23:17:34 +00:00

(Migrated from github.com)

Well, putting together the pieces from above and constructing an example: right before creating the tag, you could generate a version.txt file with a single line, e.g.

version: 2021.06.23-9-gc3f05a1646-FDroid+21062309

That could then be matched by

AutoUpdateMode: Version
UpdateCheckMode: Tags
UpdateCheckData: version.txt|version:\s.+\+(\d+)|.|version:\s(.+)\+
CurrentVersion: 2021.06.23-9-gc3f05a1646-FDroid
CurrentVersionCode: 21062309

Only problem here is that, as you generate those values from the release commit hash, you have a hen-and-egg issue: you need to put in the commit hash before you commit (which is why it's not the best idea using the commit has in versionName). If you could do without that for the fdroid flavor this could work, as the date will hopefully be the same for the "commit before", so 2021.06.23-9-FDroid and 21062309 would match both. CheckUpdates would then look at the latest tag, scan version.txt with the RegEx specified, see the versionCode is higher than the latest it has locally, and build it. If the resulting APK then matches in versionCode and versionName, it will be published. No MRs needed.

Well, putting together the pieces from above and constructing an example: right before creating the tag, you could generate a `version.txt` file with a single line, e.g. version: 2021.06.23-9-gc3f05a1646-FDroid+21062309 That could then be matched by ```yaml AutoUpdateMode: Version UpdateCheckMode: Tags UpdateCheckData: version.txt|version:\s.+\+(\d+)|.|version:\s(.+)\+ CurrentVersion: 2021.06.23-9-gc3f05a1646-FDroid CurrentVersionCode: 21062309 ``` Only problem here is that, as you generate those values from the release commit hash, you have a hen-and-egg issue: you need to put in the commit hash before you commit (which is why it's not the best idea using the commit has in `versionName`). If you could do without that *for the `fdroid` flavor* this could work, as the date will hopefully be the same for the "commit before", so `2021.06.23-9-FDroid` and `21062309` would match both. CheckUpdates would then look at the latest tag, scan `version.txt` with the RegEx specified, see the `versionCode` is higher than the latest it has locally, and build it. If the resulting APK then matches in `versionCode` and `versionName`, it will be published. No MRs needed.

👍 2

rtsisyk commented

2021-08-08 07:42:33 +00:00

Owner

I've pushed #1033:

Gradle doesn't add git hash to version anymore.
Added Triple-T layout to android/src/fdroid/play/.
Tagged this commit in the master as 2021.08.08-7-android for testing.

I also took a quick look at https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/checkupdates.py#L146. I tested this code locally (see https://gist.github.com/rtsisyk/b04e1bbd6fa178eb473a0a07025a805c) and it seems to be working:

android % ./fdroid-checkupdates.py
DEBUG:root:UpdateCheckData found version 2021.08.08-7-FDroid (21080807)

@relan could you please help us with updating fdroidata? Please play with real checkupdates.py to check that it actually works.

AutoUpdateMode: Version
UpdateCheckMode: Tags
UpdateCheckData: src/fdroid/play/version.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+

I've pushed #1033: 1. Gradle doesn't add git hash to version anymore. 2. Added Triple-T layout to `android/src/fdroid/play/`. 3. Tagged this commit in the master as `2021.08.08-7-android` for testing. I also took a quick look at https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/checkupdates.py#L146. I tested this code locally (see https://gist.github.com/rtsisyk/b04e1bbd6fa178eb473a0a07025a805c) and it seems to be working: ``` android % ./fdroid-checkupdates.py DEBUG:root:UpdateCheckData found version 2021.08.08-7-FDroid (21080807) ``` @relan could you please help us with updating fdroidata? Please play with real checkupdates.py to check that it actually works. ``` AutoUpdateMode: Version UpdateCheckMode: Tags UpdateCheckData: src/fdroid/play/version.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+ ```

relan commented

2021-08-09 12:21:46 +00:00

(Migrated from github.com)

UpdateCheckData: src/fdroid/play/version.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+

The android component is missing in the path. Works fine after I changed the UCD line to

UpdateCheckData: android/src/fdroid/play/version.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+

So you can delete the test tag and I'll enable autoupdates in fdroiddata.

UpdateCheckData: src/fdroid/play/version.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+ The `android` component is missing in the path. Works fine after I changed the UCD line to UpdateCheckData: android/src/fdroid/play/version.yaml|version:\s.+\+(\d+)|.|version:\s(.+)\+ So you can delete the test tag and I'll enable autoupdates in fdroiddata.

👍 1 🚀 1

rtsisyk commented

2021-08-09 13:37:56 +00:00

Owner

[deleted] 2021.08.08-7-android

- [deleted] 2021.08.08-7-android

relan commented

2021-08-09 14:30:37 +00:00

(Migrated from github.com)

Done: 2b91d48fde

Done: https://gitlab.com/fdroid/fdroiddata/-/commit/2b91d48fde95cc7469115ba863b259936f617cca

biodranik commented

2021-08-31 11:26:08 +00:00

(Migrated from github.com)

Can someone help? There is no cmake 3.18.1 from Android SDK installed and fdroid build fails:

https://monitor.f-droid.org/builds/log/app.organicmaps/21081906

Can someone help? There is no cmake 3.18.1 from Android SDK installed and fdroid build fails: https://monitor.f-droid.org/builds/log/app.organicmaps/21081906

👍 1

relan commented

2021-09-01 08:11:05 +00:00

(Migrated from github.com)

Can someone help? There is no cmake 3.18.1 from Android SDK installed and fdroid build fails

Here it is: 482bfdad7b/metadata/app.organicmaps.yml (L316)

> Can someone help? There is no cmake 3.18.1 from Android SDK installed and fdroid build fails Here it is: https://gitlab.com/fdroid/fdroiddata/-/blob/482bfdad7b8cef0dbc3e7bd51db821410947ce5b/metadata/app.organicmaps.yml#L316

biodranik commented

2021-09-04 10:39:49 +00:00

(Migrated from github.com)

@relan it failed again: https://monitor.f-droid.org/builds/log/app.organicmaps/21090106#site-footer

Could it be due to missing Command Line Tools from Android SDK, as mentioned here: https://github.com/flutter/flutter/issues/56778#issuecomment-882000396 ?

@relan it failed again: https://monitor.f-droid.org/builds/log/app.organicmaps/21090106#site-footer Could it be due to missing Command Line Tools from Android SDK, as mentioned here: https://github.com/flutter/flutter/issues/56778#issuecomment-882000396 ?

relan commented

2021-09-04 13:56:39 +00:00

(Migrated from github.com)

it failed again

Ouch! Fixed in 4f60953dde. Tested that it builds now.

> it failed again Ouch! Fixed in https://gitlab.com/fdroid/fdroiddata/-/commit/4f60953ddebf3a98711a5aee147431acf2e3c540. Tested that it builds now.

rtsisyk commented

2022-01-08 10:51:36 +00:00

Owner

This one is done. Thanks all!

Rows
Columns

Automate F-Droid updates and metadata #825

Ideas