Generate Wikipedia articles from offline Wikipedia dump #3478

New issue

Closed

opened 2022-09-24 18:05:54 +00:00 by biodranik · 5 comments

biodranik commented

2022-09-24 18:05:54 +00:00

(Migrated from github.com)

There are several issues now with our current crawler implementation:

It is banned by Wiki servers because it DDoSes the API.
It takes a lot of time to get all summaries for several supported languages.
If we add more languages, the time increases even more.
As a result, now we are still using some old outdated articles we got in this spring.

A way better option is to download a dump of all Wiki articles for given languages and extract summaries from there directly.

Any volunteers for this task?

A list of supported languages and the output format can be checked in the existing implementation in tools/python/descriptions.

There are several issues now with our current crawler implementation: 1. It is banned by Wiki servers because it DDoSes the API. 2. It takes a lot of time to get all summaries for several supported languages. 3. If we add more languages, the time increases even more. 4. As a result, now we are still using some old outdated articles we got in this spring. A way better option is to download a dump of all Wiki articles for given languages and extract summaries from there directly. Any volunteers for this task? A list of supported languages and the output format can be checked in the existing implementation in `tools/python/descriptions`.

👍 6

az09 commented

2023-01-06 18:43:18 +00:00

(Migrated from github.com)

Can I help? ^_^

biodranik commented

2023-01-06 19:06:23 +00:00

(Migrated from github.com)

Some ideas about the workflow:

A separate script to update the wiki dumps to the latest version (looks like each language should be downloaded separately).
A separate script/tool to process the dump and extract articles only in required/supported languages.
pbzip2 (parallel bzip2) may be used to decompress/process dump faster.
A fast stream XML parser should be used to avoid unnecessary bz2 archives decompression.
Optionally, some timestamp check logic can be implemented to detect if the article was modified to avoid unnecessary file operations. Otherwise, a previous dump should be completely removed and regenerated again.
(Bonus for the future) Need to check if we can leave HTTP links to uploaded images, so later articles can optionally load/show images if there is an internet connection, and user has allowed it.

Some ideas about the workflow: 1. A separate script to update the wiki dumps to the latest version (looks like each language should be [downloaded separately](https://dumps.wikimedia.org/backup-index.html)). 2. A separate script/tool to process the dump and extract articles only in required/supported languages. 3. `pbzip2` (parallel bzip2) may be used to decompress/process dump faster. 4. A fast stream XML parser should be used to avoid unnecessary bz2 archives decompression. 5. Optionally, some timestamp check logic can be implemented to detect if the article was modified to avoid unnecessary file operations. Otherwise, a previous dump should be completely removed and regenerated again. 6. (Bonus for the future) Need to check if we can leave HTTP links to uploaded images, so later articles can optionally load/show images if there is an internet connection, and user has allowed it.

pastk commented

2023-01-07 13:46:10 +00:00

Member

Linking #2410

euf commented

2023-01-10 10:37:58 +00:00

(Migrated from github.com)

Since current workflow is in Python, a few libraries to consider:

Since current workflow is in Python, a few libraries to consider: - https://github.com/5j9/wikitextparser - https://github.com/earwig/mwparserfromhell

biodranik commented

2023-01-10 16:47:41 +00:00

(Migrated from github.com)

Python is not the fastest tool to quickly and efficiently process large bz2-ed dumps. There are faster tools/languages.

Python is not the _fastest_ tool to quickly and efficiently process large bz2-ed dumps. There are faster tools/languages.

This repo is archived. You cannot comment on issues.