Merge pull request #357 from Sarthak-Shreshtha01/fix/language-from-any-meta

Find a content-language meta that is not the first meta

Author: benoit74

CHANGELOG.md

@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Carry a script's top-level `const`, `let` and `class` declarations across the wombat block, so other scripts on the page can still see them (#329)
 - Keep a rewritten `<script>`'s original `src` in `__wb_orig_src`, so wombat can return it from `getAttribute("src")` and bundlers that identify chunks by it (Next.js/Turbopack, Vite) still work (openzim/warc2zim#473)
 - Keep URL-encoded `#` characters of the path or querystring when rewriting document URIs, instead of dropping everything after them (#341)
+- Find the language of an HTML page in a `content-language` `<meta>` that is not the first `<meta>` of the page (#356)
 - Rewrite the `poster` attribute (e.g. of `<video>`) when rewriting HTML, like `src` and `href` (#339)
 - Do not collapse consecutive slashes `//` into a single `/` when normalizing URLs into ZIM paths, in both Python and JS rewriting (#340)
 - `Creator.finish()` no longer silently discards errors raised by libzim while finalizing the ZIM, so a failure during finalization is no longer reported as a successful run (openzim/warc2zim#462)

src/zimscraperlib/html.py

@@ -37,8 +37,9 @@ def find_language_in(content: str | BinaryIO | TextIO, mime_type: str) -> str:
     soup = BeautifulSoup(content, "lxml")
     for nodename, keylist in mapping.items():
         for key in keylist:
-            node = soup.find(nodename)
-            if node:
+            # all nodes, not only the first: the content-language <meta> is
+            # often not the first one (e.g. after <meta charset>)
+            for node in soup.find_all(nodename):
                 if not isinstance(
                     node,
                     element.Tag,  # pyright:ignore[reportUnnecessaryIsInstance]

tests/html/test_html.py

@@ -98,6 +98,14 @@ def test_find_language(tmp_path: pathlib.Path, html_page: str):
             ),
             "en-UK",
         ),
+        (
+            (
+                "<html><head><meta charset='utf-8'>"
+                "<meta http-equiv='content-language' content='en-UK'>"
+                "</head><body></body></html>"
+            ),
+            "en-UK",
+        ),
     ],
 )
 def test_find_language_order(html_string: str, expected_language: str):