organicmaps/wikiparser
13
Fork 0
Code Issues 13 Pull requests 2 Projects Releases Wiki Activity Actions

Remove doctype and html element

Browse source 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
This commit is contained in:
Evan Lloyd New-Schmidt 2023-08-10 16:27:10 -04:00 • committed by Evan Lloyd New-Schmidt
parent 58f32b43fd
commit 81783695d5
3 changed files with 184 additions and 184 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
src/html.rs
View file

@ -127,7 +127,7 @@ pub fn simplify_html(document: &mut Html, lang: &str) {
remove_empty_sections(document);
remove_comments(document);
remove_non_element_nodes(document);
expand_links(document);
@ -144,10 +144,14 @@ fn remove_ids(document: &mut Html, ids: impl IntoIterator<Item = NodeId>) {
}
}
fn remove_comments(document: &mut Html) {
fn remove_non_element_nodes(document: &mut Html) {
let mut to_remove = Vec::new();
for el in document.root_element().descendants() {
if el.value().is_comment() {
// `.root_element()` returns the first `Element` in the children of the
// root, which comments/doctypes are not.
// Use `root()` instead and `skip()` because `descendants` includes the
// node it is called on.
for el in document.tree.root().descendants().skip(1) {
if el.value().is_comment() || el.value().is_doctype() {
to_remove.push(el.id());
}
}
@ -234,7 +238,7 @@ fn final_expansions(document: &mut Html) {
.filter_map(ElementRef::wrap)
{
if (el.value().name() == "span" && el.value().attrs().next().is_none())
|| ["section", "body"].contains(&el.value().name())
|| ["section", "body", "html"].contains(&el.value().name())
{
to_expand.push(el.id());
}

166
tests/data/Q4185820-en/output.html
View file

@ -1,96 +1,94 @@
<!DOCTYPE html><html>
<p>
<span id="coordinates">
Coordinates: <span title="Maps, aerial photos, and other data for this location">53°06′11.4″N08°46′29.2″W</span> / <span title="Maps, aerial photos, and other data for this location">53.103167°N 8.774778°W</span> / 53.103167; -8.774778</span>
</p>
<p>
<b>
Thoor Ballylee Castle</b> (Irish<i>Túr Bhaile Uí Laí</i>) is a fortified, 15th-century Anglo-Normantower house built by the septsde Burgo, or Burke, near the town of Gort in County Galway, Ireland. It is also known as <i>Yeats' Tower</i> because it was once owned and inhabited by the poet William Butler Yeats.
</p>
<p>
It has been described as ‘the most important public building in Ireland’ by late Nobel laureate Seamus Heaney.
</p>
<h2 id="History">
History
</h2>
<p>
The castle was built in the 15th (or possibly 16th) century and originally formed part of the huge estates of the Earls of Clanricarde, from the de Burgo or Burke family.
</p>
<p>
The nearby four-arched bridge dates to around 1825. In 1837, the Carrig family was recorded as living in the castle. At the time of Griffith's Valuation (1857), Patrick Carrick was leasing a herd's house, castle and land at Ballylee, barony of Kiltartan, from William Henry Gregory. At the time, the property was valued at £5.
</p>
<p>
In the early 1900s, the castle/tower was still owned by the Gregory family and became part of nearby Coole Estate, home of Lady Augusta Gregory, Yeats’ lifelong friend. On the estate, Coole House, where Lady Gregory lived, was the centre for meetings for the Irish literary group, a group composed of a great number of preeminent figures of the day. Near this tower, in Coole Park, began the Irish Literary Revival.
</p>
<p>
Thoor Ballylee is also known today as <b>Yeats’ Tower</b>, because in 1916 (or 1917) Yeats purchased the property for the nominal sum of £35 because he was so enchanted with it and especially as it was located in a rural area. From 1921 to 1929, Yeats and his family lived there as it was his monument and symbol: In both aspects, it satisfied his desire for a rooted place in the countryside. The tower retained its original windows in the upper part. Yeats and his architect, Professor William A. Scott, restored the tower for the next two years and installed larger windows in the lower floors.
</p>
<p>
As he had an affinity for the Irish language, Yeats dropped the term "castle" in naming the property and replaced it with "<i>Thoor</i>" (<i>Túr</i>), the Irish word for "tower"; thus, the place has been known as Thoor Ballylee. For twelve years, Thoor Ballylee was Yeats’ summer home as it was his country retreat. In a letter to a friend, he wrote, "Everything is so beautiful that to go elsewhere is to leave beauty behind." Consequently, it is no wonder that Yeats was inspired and compelled to create literary works at Ballylee such as poems like <i>The Tower</i> and <i>Coole Park and Ballylee</i>.
</p>
<p>
In 1929, Ballylee was abandoned as the Yeats family moved out and it fell to disuse and ruin.
</p>
<p>
In 1951, a scene of John Ford's The Quiet Man in which John Wayne and Maureen O'Hara cross a river was shot next to Thoor Ballylee.
</p>
<p>
Mary Hanley (1914-1979) was the founder of the Kiltartan Society. A native of Carron, County Clare, Hanley founded the society in 1961 to foster interest in the literary history of the district, especially that of Lady Gregory, Edward Martyn and W.B. Yeats. She was responsible for the restoration of Thoor Ballylee (with the aid of Bord Fáilte and the Yeats family). At the time, the Office of Public Works was owner of the property. Hanley persuaded the poet Padraic Colum to open the castle on Sunday 20 June 1965, the centenary of Yeats’s birth, as <i>Yeats Tower</i> to appear as it was when he lived there and refitted as a Yeats museum containing a collection of first editions and items of furniture. The adjoining miller's cottage became a tea room and shop. This was later expanded by a newly constructed building in the back.
</p>
<h2 id="Today">
Today
</h2>
<p>
Due to its proximity to the Streamstown River, Thoor Ballylee is subject to sporadic flooding. This occurred notably in 1995 and in 2009/2010. In 2009, Thoor Ballylee was extensively damaged by flooding. For a while it appeared that due to the financial problems of the Irish government, no money would be available to repair it.
</p>
<p>
Thus only in February 2012 did work by Fáilte Ireland on restoring the tower begin, although no opening date was envisaged at the time. One of the forces behind the decision to repair the tower had been East Galway senator Lorraine Higgins, who argued that a reopened Yeats' Tower would be a boon to local tourism.
</p>
<p>
By February 2013 the tower had still not reopened. However, a private group — in cooperation with Fáilte Ireland — had engaged the services of Galway Rural Development, a make-work-scheme, for the maintenance work.
</p>
<p>
In 2014, a local community group the "Yeats Thoor Ballylee Society" leased Thoor Ballylee from Fáilte Ireland to develop it into a culture and education centre, in time for the Yeats 150th Anniversary in June 2015. The Society is cooperating with the National Yeats Steering Committee and the Yeats Society to ensure that Thoor Ballylee is an integrated part of the Yeats 2015 celebrations.
</p>
<p>
In early December 2015, Storm Desmond devastated parts of Ireland with flooding rain and damaging winds. Thoor Ballylee, and the adjacent cottage, were both damaged by several feet of flood water.
</p>
<h2 id="Architecture">
Architecture
</h2>
<p>
With four floors, the tower consists of one room on each floor that is connected by a spiral stone stairway built into the seven-foot thickness of the massive outer wall. Each floor has a window that overlooks the Streamstown River that flows alongside the tower. There is a small thatch cottage attached.
</p>
<p>
Yeats described the ground-floor chamber as "the pleasantest room I have yet seen, a great wide window opening over the river and a round arched door leading to the thatched hall". He also admired the mural stair, symbolically declaring "This winding, gyring, spiring treadmill of a stair is my ancestral stair; That Goldsmith and the Dean, Berkeley and Burke have traveled there."
</p>
<p>
There is a tablet on the wall that commemorates Yeats' sojourn:
</p>
<blockquote>
<p>
<span id="coordinates">
Coordinates: <span title="Maps, aerial photos, and other data for this location">53°06′11.4″N08°46′29.2″W</span> / <span title="Maps, aerial photos, and other data for this location">53.103167°N 8.774778°W</span> / 53.103167; -8.774778</span>
<i>
I, the poet William Yeats,</i>
</p>
<p>
<b>
Thoor Ballylee Castle</b> (Irish<i>Túr Bhaile Uí Laí</i>) is a fortified, 15th-century Anglo-Normantower house built by the septsde Burgo, or Burke, near the town of Gort in County Galway, Ireland. It is also known as <i>Yeats' Tower</i> because it was once owned and inhabited by the poet William Butler Yeats.
<i>
With old mill boards and sea-green slates,</i>
</p>
<p>
It has been described as ‘the most important public building in Ireland’ by late Nobel laureate Seamus Heaney.
</p>
<h2 id="History">
History
</h2>
<p>
The castle was built in the 15th (or possibly 16th) century and originally formed part of the huge estates of the Earls of Clanricarde, from the de Burgo or Burke family.
<i>
And smithy work from the Gort forge,</i>
</p>
<p>
The nearby four-arched bridge dates to around 1825. In 1837, the Carrig family was recorded as living in the castle. At the time of Griffith's Valuation (1857), Patrick Carrick was leasing a herd's house, castle and land at Ballylee, barony of Kiltartan, from William Henry Gregory. At the time, the property was valued at £5.
<i>
Restored this tower for my wife George.</i>
</p>
<p>
In the early 1900s, the castle/tower was still owned by the Gregory family and became part of nearby Coole Estate, home of Lady Augusta Gregory, Yeats’ lifelong friend. On the estate, Coole House, where Lady Gregory lived, was the centre for meetings for the Irish literary group, a group composed of a great number of preeminent figures of the day. Near this tower, in Coole Park, began the Irish Literary Revival.
<i>
And may these characters remain</i>
</p>
<p>
Thoor Ballylee is also known today as <b>Yeats’ Tower</b>, because in 1916 (or 1917) Yeats purchased the property for the nominal sum of £35 because he was so enchanted with it and especially as it was located in a rural area. From 1921 to 1929, Yeats and his family lived there as it was his monument and symbol: In both aspects, it satisfied his desire for a rooted place in the countryside. The tower retained its original windows in the upper part. Yeats and his architect, Professor William A. Scott, restored the tower for the next two years and installed larger windows in the lower floors.
<i>
When all is ruin once again.</i>
</p>
<p>
As he had an affinity for the Irish language, Yeats dropped the term "castle" in naming the property and replaced it with "<i>Thoor</i>" (<i>Túr</i>), the Irish word for "tower"; thus, the place has been known as Thoor Ballylee. For twelve years, Thoor Ballylee was Yeats’ summer home as it was his country retreat. In a letter to a friend, he wrote, "Everything is so beautiful that to go elsewhere is to leave beauty behind." Consequently, it is no wonder that Yeats was inspired and compelled to create literary works at Ballylee such as poems like <i>The Tower</i> and <i>Coole Park and Ballylee</i>.
</p>
<p>
In 1929, Ballylee was abandoned as the Yeats family moved out and it fell to disuse and ruin.
</p>
<p>
In 1951, a scene of John Ford's The Quiet Man in which John Wayne and Maureen O'Hara cross a river was shot next to Thoor Ballylee.
</p>
<p>
Mary Hanley (1914-1979) was the founder of the Kiltartan Society. A native of Carron, County Clare, Hanley founded the society in 1961 to foster interest in the literary history of the district, especially that of Lady Gregory, Edward Martyn and W.B. Yeats. She was responsible for the restoration of Thoor Ballylee (with the aid of Bord Fáilte and the Yeats family). At the time, the Office of Public Works was owner of the property. Hanley persuaded the poet Padraic Colum to open the castle on Sunday 20 June 1965, the centenary of Yeats’s birth, as <i>Yeats Tower</i> to appear as it was when he lived there and refitted as a Yeats museum containing a collection of first editions and items of furniture. The adjoining miller's cottage became a tea room and shop. This was later expanded by a newly constructed building in the back.
</p>
<h2 id="Today">
Today
</h2>
<p>
Due to its proximity to the Streamstown River, Thoor Ballylee is subject to sporadic flooding. This occurred notably in 1995 and in 2009/2010. In 2009, Thoor Ballylee was extensively damaged by flooding. For a while it appeared that due to the financial problems of the Irish government, no money would be available to repair it.
</p>
<p>
Thus only in February 2012 did work by Fáilte Ireland on restoring the tower begin, although no opening date was envisaged at the time. One of the forces behind the decision to repair the tower had been East Galway senator Lorraine Higgins, who argued that a reopened Yeats' Tower would be a boon to local tourism.
</p>
<p>
By February 2013 the tower had still not reopened. However, a private group — in cooperation with Fáilte Ireland — had engaged the services of Galway Rural Development, a make-work-scheme, for the maintenance work.
</p>
<p>
In 2014, a local community group the "Yeats Thoor Ballylee Society" leased Thoor Ballylee from Fáilte Ireland to develop it into a culture and education centre, in time for the Yeats 150th Anniversary in June 2015. The Society is cooperating with the National Yeats Steering Committee and the Yeats Society to ensure that Thoor Ballylee is an integrated part of the Yeats 2015 celebrations.
</p>
<p>
In early December 2015, Storm Desmond devastated parts of Ireland with flooding rain and damaging winds. Thoor Ballylee, and the adjacent cottage, were both damaged by several feet of flood water.
</p>
<h2 id="Architecture">
Architecture
</h2>
<p>
With four floors, the tower consists of one room on each floor that is connected by a spiral stone stairway built into the seven-foot thickness of the massive outer wall. Each floor has a window that overlooks the Streamstown River that flows alongside the tower. There is a small thatch cottage attached.
</p>
<p>
Yeats described the ground-floor chamber as "the pleasantest room I have yet seen, a great wide window opening over the river and a round arched door leading to the thatched hall". He also admired the mural stair, symbolically declaring "This winding, gyring, spiring treadmill of a stair is my ancestral stair; That Goldsmith and the Dean, Berkeley and Burke have traveled there."
</p>
<p>
There is a tablet on the wall that commemorates Yeats' sojourn:
</p>
<blockquote>
<p>
<i>
I, the poet William Yeats,</i>
</p>
<p>
<i>
With old mill boards and sea-green slates,</i>
</p>
<p>
<i>
And smithy work from the Gort forge,</i>
</p>
<p>
<i>
Restored this tower for my wife George.</i>
</p>
<p>
<i>
And may these characters remain</i>
</p>
<p>
<i>
When all is ruin once again.</i>
</p>
</blockquote>
</html>
</blockquote>

188
tests/data/Q748282-en/output.html
View file

@ -1,95 +1,93 @@
<!DOCTYPE html><html>
<p>
The <b>Crimean Mountains</b> (Crimean Tatar: <span lang="crh">Qırım dağları</span>; Ukrainian: <span lang="uk">Кримські гори</span>; Russian: <span lang="ru">Крымские горы</span>; Turkish: <i lang="tr">Yayla Dağları</i>) or <b>Yayla Mountains</b> are a range of mountains running parallel to the south-eastern coast of Crimea, between about 8–13 kilometers (5–8 miles) from the sea. Toward the west, the mountains drop steeply to the Black Sea, and to the east, they change slowly into a steppe landscape.
</p>
<p>
The Crimean Mountains consist of three subranges. The highest is the Main Range, which is subdivided into several yaylas or mountain plateaus (<i>yayla</i> or <i>yaylak</i> is Turkic for "alpine meadow"). They are:
</p>
<ul>
<li>
Baydar yayla
</li>
<li>
Ai-Petri yayla
</li>
<li>
Yalta yayla
</li>
<li>
Nikita yayla
</li>
<li>
Hurzuf yayla
</li>
<li>
Babugan yayla
</li>
<li>
Chatyr-Dag yayla
</li>
<li>
Dologorukovskaya (Subatkan) yayla
</li>
<li>
Demirci yayla
</li>
<li>
Qarabiy yayla
</li>
</ul>
<h2 id="Highest_peaks">
Highest peaks
</h2>
<p>
The Crimea's highest peak is the Roman-Kosh (Ukrainian: <span lang="uk">Роман-Кош</span>; Russian: <span lang="ru">Роман-Кош</span>, Crimean Tatar: <span lang="crh">Roman Qoş</span>) on the Babugan Yayla at 1,545 metres (5,069ft). Other important peaks over 1,200 metres include:
</p>
<ul>
<li>
Demir-Kapu (Ukrainian: <span lang="uk">Демір-Капу</span>, Russian: <span lang="ru">Демир-Капу</span>, Crimean Tatar: <span lang="crh">Demir Qapı</span>) 1,540 m in the Babugan Yayla;
</li>
<li>
Zeytin-Kosh (Ukrainian: <span lang="uk">Зейтин-Кош</span>; Russian: <span lang="ru">Зейтин-Кош</span>, Crimean Tatar: <span lang="crh">Zeytün Qoş</span>) 1,537 m in the Babugan Yayla;
</li>
<li>
Kemal-Egerek (Ukrainian: <span lang="uk">Кемаль-Егерек</span>, Russian: <span lang="ru">Кемаль-Эгерек</span>, Crimean Tatar: <span lang="crh">Kemal Egerek</span>) 1,529 m in the Babugan Yayla;
</li>
<li>
Eklizi-Burun (Ukrainian: <span lang="uk">Еклізі-Бурун</span>, Russian: <span lang="ru">Эклизи-Бурун</span>, Crimean Tatar: <span lang="crh">Eklizi Burun</span>) 1,527 m in the Chatyrdag Yayla;
</li>
<li>
Lapata (Ukrainian: <span lang="uk">Лапата</span>; Russian: <span lang="ru">Лапата</span>, Crimean Tatar: <span lang="crh">Lapata</span>) 1,406 m in the Yaltynska Yayla, Yalta Yaylası;
</li>
<li>
Northern Demirji (Ukrainian: <span lang="uk">Північний Демірджі</span>, Russian: <span lang="ru">Северный Демирджи</span>, Crimean Tatar: <span lang="crh">Şimaliy Demirci</span>) 1,356 m in the Demirci Yayla;
</li>
<li>
Ai-Petri (Ukrainian: <span lang="uk">Ай-Петрі</span>, Russian: <span lang="ru">Ай-Петри</span>, Crimean Tatar: <span lang="crh">Ay Petri</span>) 1,234 m in the Ay Petri Yaylası.
</li>
</ul>
<h2 id="Passes_and_rivers">
Passes and rivers
</h2>
<p>
The passes over the Crimean Mountains are: (from east to west)
</p>
<ul>
<li>
Angarskyi Pass (752m) near Perevalne, on a road from Alushta to Simferopol
</li>
<li>
Baydar Gate (503m) near Foros, connecting Baydar Valley and the sea coast
</li>
<li>
Laspi Pass (350m) near Cape Aya, on a road from Yalta to Sevastopol.
</li>
</ul>
<p>
Rivers of the Crimean Mountains include the Alma River, Chernaya River, and Salhir River on the northern slope and Uchan-su River on the southern slope which forms the Uchan-su waterfall, and the highest waterfall in Crimea.
</p>
<h2 id="History">
History
</h2>
<p>
Archaeologists have found the earliest anatomically modern humans in Europe in the Crimean Mountains' Buran-Kaya caves. The fossils are 32,000 years old, with the artifacts linked to the Gravettian culture. The fossils have cut marks suggesting a post-mortem defleshing ritual.
</p>
</html>
<p>
The <b>Crimean Mountains</b> (Crimean Tatar: <span lang="crh">Qırım dağları</span>; Ukrainian: <span lang="uk">Кримські гори</span>; Russian: <span lang="ru">Крымские горы</span>; Turkish: <i lang="tr">Yayla Dağları</i>) or <b>Yayla Mountains</b> are a range of mountains running parallel to the south-eastern coast of Crimea, between about 8–13 kilometers (5–8 miles) from the sea. Toward the west, the mountains drop steeply to the Black Sea, and to the east, they change slowly into a steppe landscape.
</p>
<p>
The Crimean Mountains consist of three subranges. The highest is the Main Range, which is subdivided into several yaylas or mountain plateaus (<i>yayla</i> or <i>yaylak</i> is Turkic for "alpine meadow"). They are:
</p>
<ul>
<li>
Baydar yayla
</li>
<li>
Ai-Petri yayla
</li>
<li>
Yalta yayla
</li>
<li>
Nikita yayla
</li>
<li>
Hurzuf yayla
</li>
<li>
Babugan yayla
</li>
<li>
Chatyr-Dag yayla
</li>
<li>
Dologorukovskaya (Subatkan) yayla
</li>
<li>
Demirci yayla
</li>
<li>
Qarabiy yayla
</li>
</ul>
<h2 id="Highest_peaks">
Highest peaks
</h2>
<p>
The Crimea's highest peak is the Roman-Kosh (Ukrainian: <span lang="uk">Роман-Кош</span>; Russian: <span lang="ru">Роман-Кош</span>, Crimean Tatar: <span lang="crh">Roman Qoş</span>) on the Babugan Yayla at 1,545 metres (5,069ft). Other important peaks over 1,200 metres include:
</p>
<ul>
<li>
Demir-Kapu (Ukrainian: <span lang="uk">Демір-Капу</span>, Russian: <span lang="ru">Демир-Капу</span>, Crimean Tatar: <span lang="crh">Demir Qapı</span>) 1,540 m in the Babugan Yayla;
</li>
<li>
Zeytin-Kosh (Ukrainian: <span lang="uk">Зейтин-Кош</span>; Russian: <span lang="ru">Зейтин-Кош</span>, Crimean Tatar: <span lang="crh">Zeytün Qoş</span>) 1,537 m in the Babugan Yayla;
</li>
<li>
Kemal-Egerek (Ukrainian: <span lang="uk">Кемаль-Егерек</span>, Russian: <span lang="ru">Кемаль-Эгерек</span>, Crimean Tatar: <span lang="crh">Kemal Egerek</span>) 1,529 m in the Babugan Yayla;
</li>
<li>
Eklizi-Burun (Ukrainian: <span lang="uk">Еклізі-Бурун</span>, Russian: <span lang="ru">Эклизи-Бурун</span>, Crimean Tatar: <span lang="crh">Eklizi Burun</span>) 1,527 m in the Chatyrdag Yayla;
</li>
<li>
Lapata (Ukrainian: <span lang="uk">Лапата</span>; Russian: <span lang="ru">Лапата</span>, Crimean Tatar: <span lang="crh">Lapata</span>) 1,406 m in the Yaltynska Yayla, Yalta Yaylası;
</li>
<li>
Northern Demirji (Ukrainian: <span lang="uk">Північний Демірджі</span>, Russian: <span lang="ru">Северный Демирджи</span>, Crimean Tatar: <span lang="crh">Şimaliy Demirci</span>) 1,356 m in the Demirci Yayla;
</li>
<li>
Ai-Petri (Ukrainian: <span lang="uk">Ай-Петрі</span>, Russian: <span lang="ru">Ай-Петри</span>, Crimean Tatar: <span lang="crh">Ay Petri</span>) 1,234 m in the Ay Petri Yaylası.
</li>
</ul>
<h2 id="Passes_and_rivers">
Passes and rivers
</h2>
<p>
The passes over the Crimean Mountains are: (from east to west)
</p>
<ul>
<li>
Angarskyi Pass (752m) near Perevalne, on a road from Alushta to Simferopol
</li>
<li>
Baydar Gate (503m) near Foros, connecting Baydar Valley and the sea coast
</li>
<li>
Laspi Pass (350m) near Cape Aya, on a road from Yalta to Sevastopol.
</li>
</ul>
<p>
Rivers of the Crimean Mountains include the Alma River, Chernaya River, and Salhir River on the northern slope and Uchan-su River on the southern slope which forms the Uchan-su waterfall, and the highest waterfall in Crimea.
</p>
<h2 id="History">
History
</h2>
<p>
Archaeologists have found the earliest anatomically modern humans in Europe in the Crimean Mountains' Buran-Kaya caves. The fossils are 32,000 years old, with the artifacts linked to the Gravettian culture. The fossils have cut marks suggesting a post-mortem defleshing ritual.
</p>