normalising style attributes into `head/style` #32

petermr · 2023-01-14T22:39:33Z

petermr
Jan 14, 2023
Maintainer

typical styles

see test/resources/ipcc/Chapter06/mini.html. Content is:

<html>
<div style="" id="id709" marker="Executive Summary "><span style="font-family: TimesNewRomanPS-BoldMT; font-size: 14px;" id="id710">Executive Summary </span>
<div style="" id="id712"><span style="font-family: TimesNewRomanPS-BoldMT; font-size: 11px;" id="id713">Warming cannot be limited to well below 2&#176;C without rapid and deep reductions in energy system CO</span><span style="font-family: TimesNewRomanPS-BoldMT; font-size: 6px;" id="id715">2</span><span style="font-family: TimesNewRomanPS-BoldMT; font-size: 11px;" id="id716"> and GHG emissions. </span><span style="font-family: TimesNewRomanPSMT; font-size: 11px;" id="id717">In scenarios limiting likely warming to 1.5&#176;C with limited overshoot (likely below 2&#176;C), net energy system CO</span><span style="font-family: TimesNewRomanPSMT; font-size: 6px;" id="id719">2</span><span style="font-family: TimesNewRomanPSMT; font-size: 11px;" id="id720"> emissions (interquartile range) fall by 87% to 97%% (60% to 79%) in 2050. In 2030, in scenarios limiting warming to 1.5&#176;C with no or limited overshoot, net CO2 and GHG emissions fall by 35-51% and 38-52% respectively. In scenarios limiting warming to 1.5&#176;C with no or limited overshoot (likely below 2&#176;C), net electricity sector CO</span><span style="font-family: TimesNewRomanPSMT; font-size: 6px;" id="id724">2</span><span style="font-family: TimesNewRomanPSMT; font-size: 11px;" id="id725"> emissions reach zero globally between 2045 and 2055 (2050 and 2080) </span><span style="font-family: TimesNewRomanPS-ItalicMT; font-size: 11px;" id="id727">(high confidence)</span><span style="font-family: TimesNewRomanPSMT; font-size: 11px;" id="id728"> {6.7}  </span></div>
<div style="" id="id730"><span style="font-family: TimesNewRomanPS-BoldMT; font-size: 11px;" id="id731">Limiting warming to well below 2&#176;C will require substantial energy system changes over the next 30 years. This includes reduced fossil fuel consumption, increased production from low- and zero-carbon  energy  sources,  and  increased  use  of  electricity  and  alternative  energy  carriers.</span><span style="font-family: TimesNewRomanPSMT; font-size: 11px;" id="id734">  Coal consumption  without  CCS  falls  by  67%  to  82%  (interquartile  range)  in  2030  in  scenarios  limiting warming to 1.5&#176;C with no or limited overshoot. Oil and gas consumption fall more slowly. Low-carbon sources produce 93% to 97% of global electricity by 2050 in scenarios that limit likely warming to 2&#176;C or  below.  In  scenarios  limiting  warming  to  1.5&#176;C  with  no  or  limited  overshoot  (likely  below  2&#176;C), electricity supplies 48% to 58% (36% to 47%) of final energy in 2050, up from 20% in 2019. </span><span style="font-family: TimesNewRomanPS-ItalicMT; font-size: 11px;" id="id740">(high confidence)</span><span style="font-family: TimesNewRomanPSMT; font-size: 11px;" id="id742"> {6.7} </span></div>
...

Goal is to extract all styles into

<html>
  <head>
    <style> .s1 {font-family: TimesNewRomanPS-BoldMT; font-size: 14px;}</style>
    <style> .s2 {font-family: TimesNewRomanPS-BoldMT; font-size: 11px;}</style>
...

and removing from the element styles:

<html>
<div style="" id="id709" marker="Executive Summary "><span class="s1" id="id710">Executive Summary </span>
<div style="" id="id712"><span class="s2" id="id713">Warming cannot be limited to well below 2&#176;C without rapid and deep reductions in energy system CO</span><span style="font-family: TimesNewRomanPS-BoldMT; font-size: 6px;" id="id715">2</span><span style="font-family: TimesNewRomanPS-BoldMT; font-
...

petermr · 2023-01-14T22:42:56Z

petermr
Jan 14, 2023
Maintainer Author

tests in test/test_html.py

Tests we can extract styles from @style attributes into head/style elements

class TestCSSStyle:

    def test_extract_character_style(self):
        """
        Test extract character info and separate the rest
        creates 2 substyles
        """
        css = CSSStyle.create_css_style_from_css_string(
            "font-weight: bold; font-family: monospace; font-size: 13px; color: blue; bottom: 10px;")
        (extracted, retained) = font_style, rest_style = css.extract_substyles(
            [CSSStyle.FONT_STYLE, CSSStyle.FONT_WEIGHT, CSSStyle.FONT_FAMILY, CSSStyle.FONT_SIZE, CSSStyle.COLOR])
        assert extracted == CSSStyle.create_css_style_from_css_string(
            "font-weight: bold; font-family: monospace; font-size: 13px; color: blue;")
        assert retained == CSSStyle.create_css_style_from_css_string("bottom: 10px")

    def test_extract_text_styles_into_html_style(self):
        """
        Extracts named styled components into new styles and creates HtmlStyle
        """
        css = CSSStyle.create_css_style_from_css_string(
            "font-weight: bold; font-family: monospace; font-size: 13px; color: blue; bottom: 10px;")
        extracted_style, _ = css.extract_text_styles()
        class_locator = "l1"
        html_style_elem = extracted_style.create_html_style(class_locator)
        ss = lxml.etree.tostring(html_style_elem).decode("UTF-8")
        assert ss == "<style>l1 {font-weight: bold; font-family: monospace; font-size: 13px; color: blue;}</style>", f"expected {ss}"

    def test_extract_many_text_styles_into_html_style(self):
        """
        Extracts named styled components into new styles with locators and creates HtmlStyle
        """
        html_s = """
        <html>
          <head>
          </head>
          <body>
            <ul>
              <li style="font-weight: bold; font-family: monospace; font-size: 13px; color: blue; left: 10px;">foo</li>
              <li style="font-family: monospace; font-size: 13px; color: blue; left: 10px;">foo</li>
              <li style="font-weight: bold; font-family: monospace; font-size: 13px; color: blue; left: 10px;">foo</li>
              <li style="font-weight: bold; font-family: monospace; font-size: 10px; color: red; left: 10px;">foo</li>
              <li style="font-weight: bold; font-family: monospace; font-size: 13px; color: blue; left: 10px;">foo</li>
            </ul>
          </body>
        </html>
            """
        html_elem = lxml.etree.fromstring(html_s)
        html_tidy = CSSConverter().read_html_element(html_elem)
        head = html_tidy.xpath("/html/head")[0]
        style_atts = html_tidy.xpath(".//@style")
        assert len(style_atts) == 5
        for i, style_att in enumerate(style_atts):
            css_str = style_att.__str__()
            css = CSSStyle.create_css_style_from_css_string(css_str)
            extracted_style, _ = css.extract_text_styles()
            class_locator = f".s{i}"
            html_style_elem = extracted_style.create_html_style(class_locator)
            print (f"{lxml.etree.tostring(html_style_elem).decode('UTF-8')}")
            head.append(html_style_elem)
        print(f"ss {lxml.etree.tostring(head)} \n ... {lxml.etree.tostring(html_tidy)}")
        html_dir = Path(Resources.TEMP_DIR, "html")
        html_dir.mkdir(exist_ok=True)
        with open (str(Path(html_dir, "styles.html")), "w") as f:
            f.write(lxml.etree.tostring(html_tidy).decode('UTF-8'))

new class CSSConverter to do the whole process (?obsolete)

    def test_extract_styles_from_document(self):
        """
        start of IPCC WG3 Chapter06
        identifies all styles and extacts into <head><style>s
        """
        html_elem = lxml.etree.parse(str(Path(Resources.TEST_IPCC_CHAP06, "mini.html")))
        converter = CSSConverter()
        converter.read_html_element(html_elem)

final example (?works)

    def test_extract_styles_from_document_example(self):
        """
        start of IPCC WG3 Chapter06
        identifies all styles and extacts into <head><style>s and replaces @style with @class
        """
        html_dir = Path(Resources.TEMP_DIR, "html")
        html_dir.mkdir(exist_ok=True)
        html_elem = lxml.etree.parse(str(MINI_IPCC_PATH))
        html_elem = HtmlTidy.ensure_html_head_body(html_elem)
        assert len(html_elem.xpath("/html/head/style")) == 0, f"no head styles in original"
        assert len(html_elem.xpath("/html/body//*[not(normalize-space(@style))='']")) == 50,\
            f"raw document should have 50 elements with non-empty @style attributes"
        # this is so HTML browsers can see the initial file
        with open(str(Path(html_dir, "ipcc_styles0.html")), "wb") as f:
            f.write(lxml.etree.tostring(html_elem))

        HtmlStyle.extract_styles_and_normalize_classrefs(html_elem)

        with open(str(Path(html_dir, "ipcc_styles2.html")), "wb") as f:
            f.write(lxml.etree.tostring(html_elem))

        assert len(html_elem.xpath("/html/head/style")) == 7, f"7 head styles"
        assert len(html_elem.xpath("/html/body//*[@class]")) == 51,\
            f"new document should have 51 elements with @class attributes"

0 replies

petermr · 2023-01-19T17:08:47Z

petermr
Jan 19, 2023
Maintainer Author

`CSSStyle` and `HtmlStyle`

These do different things:

CSSStyle holds the value of a CSS style. In many cases this is a dictionary list, a: b; c: d; and CSSStyle has tools to manage this
HtmlStyle manages the <style> elements and their relationship to @style on elements.

extract text styles only from `CSSStyle` object

the engine

See https://github.com/petermr/pyami/blob/main/test/test_html.py (or checkout) test/test_html.py (line 789 - may change later)

    def test_extract_character_style(self):
        """
        Test extract character info and separate the rest
        creates 2 substyles
        """
        css = CSSStyle.create_css_style_from_css_string(
            "font-weight: bold; font-family: monospace; font-size: 13px; color: blue; bottom: 10px;")
        (extracted, retained) = font_style, rest_style = css.extract_substyles(
            [CSSStyle.FONT_STYLE, CSSStyle.FONT_WEIGHT, CSSStyle.FONT_FAMILY, CSSStyle.FONT_SIZE, CSSStyle.COLOR])
        assert type(extracted) is CSSStyle
        assert type(retained) is CSSStyle
        assert extracted == CSSStyle.create_css_style_from_css_string(
            "font-weight: bold; font-family: monospace; font-size: 13px; color: blue;")
        assert retained == CSSStyle.create_css_style_from_css_string("bottom: 10px")

integration into styles

See https://github.com/petermr/pyami/blob/main/test/test_html.py (or checkout) test/test_html.py (line 815 - may change later)

    def test_extract_text_styles_into_html_style(self):
        """
        Extracts text style components into new <style>, updated class, remaining style value
        """

Style values contains text-appearance (font-* and color)

        css = CSSStyle.create_css_style_from_css_string(
            "font-weight: bold; font-family: monospace; font-size: 13px; color: blue; bottom: 10px;")

This is the name of the style. In class attributes it occurs as s1; in <style> it occurs as .s1. We'll call the first class_name and the second class_ref

        class_name = "s1"
        old_class_name = "foo bar"

The old_class_name will be updated - see last line. This returns a 3-tuple (maybe later we'll objectify this in HtmlStyle) of the text components, the non-text components and the new class value.

        new_html_style_element, retained_style_string, html_class_val = \
            css.extract_text_styles_into_class(class_name, old_classstr=old_class_name)

Note that these are all (global) text attributes and can be moved to html/head/style

        assert new_html_style_element.text == ".s1 {font-weight: bold; font-family: monospace; font-size: 13px; color: blue;}"

These are non-text and specific to the element; (certainly coordinates); they must stay on it; This is what is left in the class attribute

        assert retained_style_string == "bottom: 10px;"

and the new class attribute. I tink we have to add the retained style string (or we lose coordinates). CHECK

        assert html_class_val == "foo bar s1"

0 replies

petermr · 2023-01-19T18:17:48Z

petermr
Jan 19, 2023
Maintainer Author

putting it all together

See https://github.com/petermr/pyami/blob/main/test/test_html.py (or checkout) test/test_html.py (line ~863)

    def test_extract_styles_from_mini_document_example(self):
        """
        start of IPCC WG3 Chapter06
        identifies all styles and extacts into <head><style>s
        """
        html_dir = Path(Resources.TEMP_DIR, "html")
        html_dir.mkdir(exist_ok=True)
        html_elem = lxml.etree.parse(str(MINI_IPCC_PATH))
        html_elem = HtmlTidy.ensure_html_head_body(html_elem)

        HtmlStyle.extract_styles_and_normalize_classrefs(html_elem)

This extracts styles from a raw document, normalizes them , adds to html/head/style and removes explicit text styles from document, replacing them with classrefs

0 replies

petermr · 2023-01-21T09:01:52Z

petermr
Jan 21, 2023
Maintainer Author

empirical styles

From a rather old HTML chapter (with missing chunks... The comments are my annotations
Some of the styles serve. multiple purposes

<head id="id1">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" id="id2"/>
<style classref=".s0">.s0 {font-family: TimesNewRomanPS-BoldMT; font-size: 14px;}</style>
    <style classref=".s1">.s1 {font-family: TimesNewRomanPSMT; font-size: 11px;}</style> <!--main text -->
    <style classref=".s10">.s10 {font-family: TimesNewRomanPSMT; font-size: 6px; }</style> <!-- sub/per/script-->
    <style classref=".s25">.s25 {font-family: TimesNewRomanPS-BoldMT; font-size: 6px;}</style> <!-- bold sub/per/script-->
    <style classref=".s101">.s101 {font-family: TimesNewRomanPS-BoldMT; font-size: 11px; background: #ffffdd;}</style> <!-- leading sentence-->
    <style classref=".s103">.s103 {font-family: TimesNewRomanPS-ItalicMT; font-size: 11px; background: #ffdddd;}</style> <!-- italic -->
    <style classref=".s128">.s128 {font-family: SymbolMT; font-size: 11px; background: #ddddff;}</style> <!-- symbol (one-off)-->
    <style classref=".s755">.s755 {font-family: Tahoma; font-size: 11px;}</style> <!-- degree -->
    <style classref=".s1000">.s1000 {font-family: TimesNewRomanPSMT; font-size: 9px; background: #aaaaff;}</style> <!-- tables -->
    <style classref=".s1096">.s1096 {font-family: TimesNewRomanPS-BoldItalicMT; font-size: 11px; background:#ffbbbb;}</style>
    <style classref=".s1108">.s1108 {font-family: TimesNewRomanPSMT; font-size: 8px; background: #ffddbb;}</style> <!-- table -->
    <style classref=".s1210">.s1210 {font-family: TimesNewRomanPS-BoldMT; font-size: 9px;}</style> <!-- tables?-->
    <style classref=".s1428">.s1428 {font-family: TimesNewRomanPS-BoldMT; font-size: 12px; background: #ffffaa}</style>
    <style classref=".s1463">.s1463 {font-family: TimesNewRomanPS-ItalicMT; font-size: 9px;}</style><!-- italic table -->
    <style classref=".s1481">.s1481 {font-family: TimesNewRomanPSMT; font-size: 12px;}</style> <!-- figure -->
    <style classref=".s1763">.s1763 {font-family: SymbolMT; font-size: 9px;}</style> <!-- bullet -->
    <style classref=".s1970">.s1970 {font-family: TimesNewRomanPS-ItalicMT; font-size: 6px;}</style> <!-- sub -->
    <style classref=".s2201">.s2201 {font-family: TimesNewRomanPS-BoldMT; font-size: 15px;}</style> <!-- header -->
    <style classref=".s2269">.s2269 {font-family: Calibri-Bold; font-size: 13px;}</style> <!-- corrigenda -->
    <style classref=".s2270">.s2270 {font-family: Calibri-Bold; font-size: 12px;}</style> <!-- corrigenda -->
    <style classref=".s2271">.s2271 {font-family: ArialMT; font-size: 10px;}</style> <!-- corrigenda -->
    <style classref=".s2273">.s2273 {font-family: Calibri-Bold; font-size: 10px;}</style> <!-- corrigenda -->
    <style classref=".s2274">.s2274 {font-family: Calibri; font-size: 10px;}</style>

These additional styles can be added automatically in most cases. They are usually included in multicomponent class_strings

    <style classref=".corrig4">.corrig4 {background: #aaffdd;}</style> <!--corrigenda -->
    <style classref=".corrig3">.corrig3 {background: #ddaaff;}</style> <!--corrigenda -->
    <style classref=".corrig2">.corrig2 {background: #ddffaa;}</style> <!--corrigenda -->
    <style classref=".corrig1">.corrig1 {background: #ffddaa;}</style> <!--corrigenda -->
    <style classref=".corrig">.corrig {background: #aaaaaa;}</style> <!--corrigenda -->
    <style classref=".large">.large {background: #aaffff;}</style> <!--header -->
    <style classref=".sub3">.sub3 {background: #aaffaa;}</style> <!--sub -->
    <style classref=".bull">.bull {background: #ff0000;}</style> <!--bullet/list -->
    <style classref=".fig2">.fig2 {font-style: italic; background: #aaffff;}</style> <!--figure -->
    <style classref=".fig1">.fig1 {font-style: italic; background: #ddffff;}</style> <!--figure -->
    <style classref=".tab1">.tab1 {font-style: italic; background: #ddffdd;}</style> <!--table -->
    <style classref=".fn1">.fn1 {font-style: italic; background: #ddffff;}</style> <!-- Footnote -->
    <style classref=".conf1">.conf1 {font-family: monospace; font-size: 6px; background: #ddffdd;}</style> <!--comment -->
    <style classref=".dec1">.dec1 {font-family: monospace; font-weight: bold; font-size: 14px; background: #bbffbb;}</style>
    <style classref=".dec2">.dec2 {font-family: monospace; font-weight: bold; font-size: 12px; background: #ffbbff;}</style>
    <style classref=".dec3">.dec3 {font-family: monospace; font-weight: bold; font-size: 10px; background: #bbffff;}</style>
    <style classref=".dec4">.dec4 {font-family: monospace; font-weight: bold; font-size: 9px; background: #bbbbff;}</style>
    </head>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

normalising style attributes into `head/style` #32

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

normalising style attributes into head/style #32

petermr Jan 14, 2023 Maintainer

typical styles

Replies: 4 comments

petermr Jan 14, 2023 Maintainer Author

tests in test/test_html.py

petermr Jan 19, 2023 Maintainer Author

CSSStyle and HtmlStyle

extract text styles only from CSSStyle object

the engine

integration into styles

petermr Jan 19, 2023 Maintainer Author

putting it all together

petermr Jan 21, 2023 Maintainer Author

empirical styles

normalising style attributes into `head/style` #32

petermr
Jan 14, 2023
Maintainer

petermr
Jan 14, 2023
Maintainer Author

petermr
Jan 19, 2023
Maintainer Author

`CSSStyle` and `HtmlStyle`

extract text styles only from `CSSStyle` object

petermr
Jan 19, 2023
Maintainer Author

petermr
Jan 21, 2023
Maintainer Author