Raw text and flow text #14

petermr · 2022-11-18T09:48:21Z

petermr
Nov 18, 2022
Maintainer

Some of the PDF parsers create characters with coordinates ("raw"). These display on separate lines in the HTML. they need to be joined into a single flowing paragraph ("flow").

petermr · 2022-11-18T09:51:55Z

petermr
Nov 18, 2022
Maintainer Author

Raw output

code

from test/test_pdf.py
uses:

AmiPage.create_html_pages

    def test_make_ami_pages_with_spans_from_charstream_ipcc_chap6(self):
        """The central AMI method to make HTML from PDF characters

        """
        output_stem = "chap6"
        page_nos = range(3, 13)
        # page_nos = [3 4 5 8 ]
        input_pdf = Path(IPCC_CHAP6_PDF)
        assert input_pdf.exists(), f"{input_pdf} should exist"
        bbox = BBox(xy_ranges=[[60, 999], [60, 790]])
        output_dir = Path(Resources.TEMP_DIR, "pdf")
        AmiPage.create_html_pages(bbox=bbox, input_pdf=input_pdf, output_dir=output_dir, output_stem=output_stem,
                                  range_list=[range(3, 8), range(129, 131)])
        assert output_dir.exists()
        assert Path(output_dir, f"{output_stem}_{5}.html").exists()

create HTML output with coordinates but no flow

<!DOCTYPE html>
<html>
	<head>
		<meta>
		<title></title>
	</head>
	<body>
		<div class="top">
			<div>
				<span style="font-size: 15.96 px;font-family: TimesNewRomanPS-BoldMT;" x0="189.29" x1="263.344" y="739.453">Chapter 6:</span>
			</div>
			<div>
				<span style="font-size: 15.96 px;font-family: TimesNewRomanPS-BoldMT;" x0="297.29" x1="410.14" y="739.453">Energy Systems</span>
			</div>
			<div>
				<span style="font-size: 11.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="206.58" y="698.995">Coordinating Lead Authors:</span> <span style="font-size: 11.04 px;font-family: TimesNewRomanPSMT;" x0="206.69" x1="505.9" y="698.995">Leon Clarke (the United States of America), Yi-Ming Wei (China)</span>
			</div>
...

			<div>
				<span style="font-size: 11.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="134.367" y="418.525">Date of Draft</span><span style="font-size: 11.04 px;font-family: TimesNewRomanPSMT;" x0="134.42" x1="193.01" y="418.525">: 27/11/2021</span>
			</div>
		</div>
	</body>
</html>

1 reply

petermr Nov 18, 2022
Maintainer Author

AmiPage.create_html_pages uses:

AmiPage.chars_to_spans

        with pdfplumber.open(input_pdf) as pdf:
            page_count = len(pdf.pages)
        for page_no in range(page_count):  # 0-based page_no
            page_1based = page_no + 1 # 1-based

            logging.debug(f"testing page {page_no}")
            # print(f"log level {logging.getLogger().level}")
        # for page_no in page_nos:
        #     logging.error(f"range list {range_list}")
        #     print(f"page 1-based {page_1based}; range list {range_list}")
            if not Util.range_list_contains_int(page_1based, range_list):
                continue
            html = AmiPage.chars_to_spans(bbox, input_pdf, page_no)
            output_html = Path(output_dir, f"{output_stem}_{page_1based}.html")
            with open(output_html, "wb") as f:
                f.write(lxml.etree.tostring(html))
                print(f" wrote html {output_html}")
                # assert output_html.exists()

The output (Raw) includes character coordinates and div but not flowing

	<body>
		<div class="top">
			<div>
				<span style="font-size: 14.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="196.52" y="753.667">Executive Summary</span>
			</div>
			<div>
				<span style="font-size: 11.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="526.094" y="727.435">Warming cannot be limited to well below 2°C without rapid and deep reductions in energy system</span>
			</div>
			<div>
				<span style="font-size: 11.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="88.533" y="713.515">CO</span>
			</div>

petermr · 2022-11-24T16:17:03Z

petermr
Nov 24, 2022
Maintainer Author

commandline

python -m py4ami.pyamix PDF 
--inpath /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17.pdf 
--outdir /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/

creates output for each page but NOT for total document

WARNING:root: *** ARG_DICT {'version': False, 'command': 'PDF', 'flow': True, 'footer': 80, 'header': 80, 'inpath': '/Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17.pdf', 'maxpage': 5, 'outdir': '/Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/', 'outstem': 'fulltext.flow', 'outform': 'html', 'pdf2html': 'pdfminer', 'pages': 'all_pages', 'resolution': 400}
WARNING:root: *** ARG_DICT {'version': False, 'command': 'PDF', 'flow': True, 'footer': 80, 'header': 80, 'inpath': '/Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17.pdf', 'maxpage': 5, 'outdir': '/Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/', 'outstem': 'Chapter17', 'outform': 'html', 'pdf2html': 'pdfminer', 'pages': 'all_pages', 'resolution': 400, 'outpath': PosixPath('/Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17.html')}
 wrote html /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/fulltext.flow_0.html
 wrote html /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/fulltext.flow_1.html
 ...
wrote html /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/fulltext.flow_98.html
 wrote html /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/fulltext.flow_99.html
(base) pm286macbook:pyami pm286$ python -m py4ami.pyamix PDF --inpath /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17.pdf

e.g.

<div><span style="font-size: 12.0 px;font-family: TimesNewRomanPSMT;" x0="72.024" x1="526.44" 
y="792.768">Final Government Distribution            Chapter 17 IPCC AR6 WGIII </span>
</div>
<div><span style="font-size: 12.0 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="242.33" 
y="35.328">Do Not Cite, Quote or Distribute </span>
<span style="font-size: 12.0 px;font-family: TimesNewRomanPSMT;" x0="286.75" x1="523.32" y="35.328">17-3  Total pages: 99</span>
</div>
<div><span style="font-size: 9.96 px;font-family: TimesNewRomanPSMT;" x0="523.44" x1="525.93" 
y="35.769"> </span>
</div>
<div><span style="font-size: 14.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="194.12" 
y="753.787">Executive summary </span>
</div>
<div><span style="font-size: 12.0 px;font-family: TimesNewRomanPSMT;" x0="48.0" x1="57.024" 
y="754.228">1 </span>
</div>
<div><span style="font-size: 12.0 px;font-family: TimesNewRomanPSMT;" x0="72.024" x1="57.024" 
y="740.068"> 2 </span>
</div>
<div><span style="font-size: 11.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="526.055" y="727.315">Accelerating climate actions and progress towards a just transition is essential to reducing climate </span>
</div>
<div><span style="font-size: 12.0 px;font-family: TimesNewRomanPSMT;" x0="48.0" x1="57.024" 
y="727.108">3 </span>
</div>
<div><span style="font-size: 11.04 px;font-family: TimesNewRomanPS-BoldMT;" x0="72.024" x1="526.083" 
y="713.395">risks and addressing sustainable development priorities, including water, food and human </span>
</div>
<div><span style="font-size: 12.0 px;font-family: TimesNewRomanPSMT;" x0="48.0" x1="57.024" y="713.188">4 </span>
</div>

Renders as:

Note the layout is determined by the divs not the coordinates. We have to join the lines and add divs when the y coord jumps

coordinate-based text

div -based lines
is LINE-based with constant Y across line, span when it changes style or size (may have hiccups at subscripts)

0 replies

petermr · 2022-11-24T18:35:48Z

petermr
Nov 24, 2022
Maintainer Author

Processing of command

def test_make_structured_html_cmdline_DEBUG(self):

...
       args = " PDF --inpath /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17.pdf" \
               " --outdir /Users/pm286/projects/semanticClimate/ipcc/ar6/wg3/Chapter17/raw/" \
               " --maxpage 4"

        pyami = PyAMI()
        pyami.run_command(args)

goes through

run_commands(self, args):
run_command(self, args):
parse_and_run_args(self, arglist):

substitutions

def run_arguments(self):

        self.wikipedia_lookup = WikidataLookup()
        self.logger.debug(f"commandline args {self.args}")
        subparser_type = self.args.get("command")
        logging.debug(f" COMMAND: {subparser_type} {self.args}")
        if not subparser_type:
            abstract_args = None
        elif subparser_type == "DICT":
            abstract_args = AmiDictArgs()
        elif subparser_type == "GUI":
            abstract_args = GUIArgs()
        elif subparser_type == "HTML":
            abstract_args = HTMLArgs()
        elif subparser_type == "PDF":
            abstract_args = PDFArgs()
        elif subparser_type == "PROJECT":
            abstract_args = ProjectArgs()

        if abstract_args:
            abstract_args.parse_and_process1(self.args)
        else:
            self.run_core_methods()

self.run_core_methods() runs help, version

abstract_args.parse_and_process1(self.args) is subclassed and runs subcommands, see below

PDF commands

0 replies

petermr · 2022-12-04T18:55:58Z

petermr
Dec 4, 2022
Maintainer Author

Flow example (imperfect, needs debugging but creates whole chapter)

THIS ONE NEARLY WORKS

example (being gradually transferred to new HtmlTidy class

test/test_pdf.PDFTest.test_make_ipcc_html_spans()

input arguments

debug calls removed for clarity.
Input messy - needs transferring to test.
Also was a pseudo production test

        sem_clim_dir = Path("/users/pm286", "projects", "semanticClimate")
        ipcc_dir = Path(sem_clim_dir, "ipcc", "ar6", "wg3")

        chapter = "Chapter04"
        pdf_args = PDFArgs() # this controls the args input
        chapter_dir = Path(ipcc_dir, chapter)
        pdf_args.arg_dict[INDIR] = chapter_dir
        inpath = Path(chapter, "fulltext.pdf")
        pdf_args.arg_dict[INPATH] = Path(chapter_dir, "fulltext.pdf")
        pdf_args.arg_dict[MAXPAGE] = 20 # not sure this works now
        pdf_args.arg_dict[OUTFORM] = "flow.html"
        outdir = Path(Resources.TEMP_DIR, "ipcc_html")
        pdf_args.arg_dict[OUTDIR] = outdir

chunks to remove

These are chunks to remove. Not sure they work. They are mainly inter-page cruft.

        pdf_args.unwanteds = {
            "chapter": {
                "xpath": ".//div/span",
                "regex": "^Chapter\\s+\\d+\\s*$"
            },
            "final_gov": {
                "xpath": ".//div/span",
                "regex": "^\\s*Final Government Distribution\\s*$"
            },
            "page": {
                "xpath": ".//div/a",
                "regex": "^\\s*Page\\s*\\d+\\s*$",
            },
            "wg3": {
                "xpath": ".//div/span",
                "regex": "^\\s*(IPCC AR6 WGIII)|(IPCC WGIII AR6)\\s*$",
            },
        }

convert and write (messy)

This dies several things controlled by args and class variables. Needs cleaning?

        _,_ = pdf_args.convert_write()

        self.process_args() # this populates the `arg_dict`
        if self.inpath is None:
            raise ValueError("No input path in convert_write()")

This is a messy routine - see below - needs more switches

        out_html = self.pdf_to_raw_then_raw_to_tidy(pdf_path=self.inpath, flow=self.flow)
        if out_html is None:
            raise ValueError(f" out_html is None")
        if self.outpath is None:
            print(f"no outpath given")
            return None
        with open(str(self.outpath), "w") as f:
            f.write(out_html)
            print(f"wrote outpath {self.outpath}")
        return self.outpath, out_html

0 replies

petermr · 2022-12-04T21:58:07Z

petermr
Dec 4, 2022
Maintainer Author

.pdf_to_raw_then_raw_to_tidy (messy)

    def pdf_to_raw_then_raw_to_tidy(self, pdf_path=None, flow=True):

This should be higher up it's the major module that creates raw html (i.e. with coordinates and unjoined lines)

        raw_html_element = PDFArgs.convert_pdf(path=pdf_path, fmt=self.outform, maxpages=self.maxpage)

    if raw_html_element is None:
        raise ValueError(f"null raw_html in convert_write()")

# program option
if not flow:
return raw_html_element

This is where HtmlTidy is created,. NOT CLEAR WHERE THE DIVING LINE IS BETWEEN PDFArgs and HtmTidy.
Maybe HtmlTidy should be set in PDFArgs?

    html_tidy = HtmlTidy()
    # might need a data tranfer object
    html_tidy.header = self.header
    html_tidy.footer = self.footer
    html_tidy.unwanteds = self.unwanteds
    html_tidy.outdir = self.outdir

    out_html_element = html_tidy.tidy_flow(raw_html_element)
    return out_html_element

## html_tidy.tidy_flow()

Probably should have `html_element` as a member.

It evolved from 2 indepedent writes of the method

def tidy_flow(self, raw_html):

    tree = lxml.etree.parse(StringIO(raw_html), lxml.etree.HTMLParser())
    result_elem = tree.getroot()

First historic chunk. Should the methods be in HtmlUtil or HtmlTidy? Not sure

      self.add_element(result_elem)
      self.add_descendant_element_to_remove("br")
      self.add_styles_to_remove(
          [
              "position",
              # "left",
              "border",
              "writing-mode",
              "width",  # if width is kept it disables flowing text
          ]
      )
      self.add_id = True
      self.add_empty_elements_to_remove(["span", "div"])
      self.remove_lh_line_numbers = True
      self.remove_large_fonted_elements = True
      marker_xpath = ".//div[a[@name]]"
      offset, pagesize, page_coords = HtmlUtil.find_constant_coordinate_markers(result_elem, marker_xpath)

these should be switchable by flags

      HtmlUtil.remove_headers_and_footers(result_elem, pagesize, self.header, self.footer, marker_xpath)
      HtmlUtil.remove_style_attribute(result_elem, "top")
      HtmlUtil.remove_style(result_elem, ["left", "height"])
      HtmlUtil.remove_unwanteds(result_elem, self.unwanteds)
      HtmlUtil.remove_newlines(result_elem)

The sections are document-type-dependent - i/e only work for WG3 of IPCC (I think) .
Definitely need to add decimal sectioning
`recs_by_section` controls the sectioning. `rec` is compiled regex. Probably stick with re as the compiler caches the compilation
/```
          HtmlTree.make_sections_and_output(result_elem, output_dir=self.outdir, recs_by_section=RECS_BY_SECTION)
          htmlstr = lxml.etree.tostring(result_elem).decode("UTF-8")

IDs are added sequently in the form idnnn...

        HtmlUtil.add_ids(result_elem)
# this can be moved
       HtmlUtil.remove_descendant_elements_by_tag("br", result_elem)
        HtmlUtil.remove_style(result_elem, [
            "position",
            # "left",
            "border",
            "writing-mode",
            "width",  # this disables flowing text
        ])
        HtmlUtil.remove_empty_elements(result_elem, ["span"])
        HtmlUtil.remove_empty_elements(result_elem, ["div"])
        HtmlUtil.remove_lh_line_numbers(result_elem) # not sure this works
        HtmlUtil.remove_large_fonted_elements(result_elem)
        marker_xpath = ".//div[a[@name]]"
        offset, pagesize, page_coords = HtmlUtil.find_constant_coordinate_markers(result_elem, marker_xpath)
        HtmlUtil.remove_headers_and_footers(result_elem, pagesize, self.header, self.footer, marker_xpath)
        HtmlUtil.remove_style_attribute(result_elem, "top")
        HtmlUtil.remove_style(result_elem, ["left", "height"])
        HtmlUtil.remove_unwanteds(result_elem, self.unwanteds)
        HtmlUtil.remove_newlines(result_elem)

sections are a major task. I think they are joined in the wrong order

        HtmlTree.make_sections_and_output(result_elem, output_dir=self.outdir, recs_by_section=RECS_BY_SECTION)
        htmlstr = lxml.etree.tostring(result_elem).decode("UTF-8")
        return htmlstr

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raw text and flow text #14

{{title}}

Replies: 5 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Raw text and flow text #14

petermr Nov 18, 2022 Maintainer

Replies: 5 comments · 1 reply

petermr Nov 18, 2022 Maintainer Author

Raw output

code

petermr Nov 18, 2022 Maintainer Author

petermr Nov 24, 2022 Maintainer Author

commandline

coordinate-based text

petermr Nov 24, 2022 Maintainer Author

Processing of command

PDF commands

petermr Dec 4, 2022 Maintainer Author

Flow example (imperfect, needs debugging but creates whole chapter)

input arguments

chunks to remove

convert and write (messy)

petermr Dec 4, 2022 Maintainer Author

.pdf_to_raw_then_raw_to_tidy (messy)

these should be switchable by flags

petermr
Nov 18, 2022
Maintainer

Replies: 5 comments 1 reply

petermr
Nov 18, 2022
Maintainer Author

petermr Nov 18, 2022
Maintainer Author

petermr
Nov 24, 2022
Maintainer Author

petermr
Nov 24, 2022
Maintainer Author

petermr
Dec 4, 2022
Maintainer Author

petermr
Dec 4, 2022
Maintainer Author