Add C++ autosummary extension #117

jbms · 2022-07-07T22:59:58Z

Fixes #92.

This is based on #116.

TODO:

jbms · 2022-07-07T23:15:22Z

This is somewhat working, but perhaps we can work together to refine it.

2bndy5 · 2022-07-07T23:35:48Z

documenting multiple signatures together (by leaving no space between them)

Is this related to overloads sharing the same docstring?

jbms · 2022-07-07T23:45:22Z

documenting multiple signatures together (by leaving no space between them)

Is this related to overloads sharing the same docstring?

Yes --- the result is a single cpp:function with multiple signatures.

I added an example of this: Array::operator[]

jbms · 2022-07-08T01:15:48Z

This is now updated to use types-clang rather than ones I generated myself. There are a few bugs in types-clang that required a cast as a workaround: tgockel/types-clang#2

2bndy5 · 2022-07-08T01:19:05Z

I also noticed that CursorKind.value isn't typed at all in that stub lib...

2bndy5

First impressions.

2bndy5 · 2022-07-08T03:29:59Z

docs/apidoc/cpp/apigen.rst

+
+.. rst:directive:: .. cpp-apigen-entity-summary:: entity-name
+
+   Generates a summary of a single Python entity.


Seems like a copy-n-paste artifact. 😄

Suggested change

Generates a summary of a single Python entity.

Generates a summary of a single C++ entity.

Yeah this example still needs to be fixed.

Also at the moment the entity is specified by the clang usr-derived id, but it should probably be something else friendlier.

I couldn't find any documentation about the USR generation, so I often look to the src code. Some USR ids use the line number, but IIRC that is only applied to locally scoped members (like param decls and vars).

sphinx_immaterial/apidoc/cpp/api_parser.py

2bndy5 · 2022-07-08T04:14:10Z

sphinx_immaterial/apidoc/cpp/api_parser.py

+            input_path,
+            unsaved_files=[(input_path, input_content)],
+            args=tuple(config.compiler_flags) + ("-ferror-limit=0",),
+            options=(  # TranslationUnit.PARSE_SKIP_FUNCTION_BODIES +


why not ignore the function bodies?

Should add a comment. If function bodies are skipped, the extent does not include the body. That interferes with detecting if there are multiple entities defined with no space between (so that they will all be documented together).

Oh ok. Yeah, a comment would've satisfied that inquiry.

sphinx_immaterial/apidoc/cpp/api_parser.py

2bndy5 · 2022-07-08T05:03:54Z

sphinx_immaterial/apidoc/cpp/api_parser.py

+    Macros matching any of these patterns are undocumented.
+    """
+
+    ignore_diagnostics: List[Pattern] = dataclasses.field(default_factory=list)


I think this could be made an arbitrary bool. In my research, I had a conf.py option to output these diags to the build log (defaulting to false) using the severity of the diag as a logging level. If users really want to use static analysis, then they can use clang-tidy separately.

Diagnostics can also be specified to the compiler_flags option (which are really just arbitrary clang args); the compilation database's path might be used in this way also.

I have found that any compiler errors are usually an indication that the result of parsing will be wrong, sometimes in subtle ways, and should normally not be ignored. However, for tensorstore I ran into some errors related to an undefined __builtin in a standard library header that I was not able to avoid, and did not seem to cause any problems, so I added this mechanism to suppress them.

I know that clang will fail fast if it can't find an included lib, thus no parsing at all. That's why I mentioned the compilation database path (-p I think) can be used when third party libs are involved.

I haven't yet looked at how you're normalizing the path relative to the conf.py path, but that might also be a factor when passing certain args to clang.

Paths are normalized using include_directory_map. We should probably try to support compilation databases, but one tricky thing about a compilation database is that it doesn't include the compiler built-in include paths. An alternative is to have the user preprocess the desired headers first using their normal compiler. The original file/line information is preserved by the added #line directives.

I think users can specify -working-directory to clang. Or, maybe we should provide an option to do it.

2bndy5 · 2022-07-08T06:17:58Z

Going forward, can we change NONITPICK to NO_NIT_PICK? I keep reading it as NON_IT_PICK 🤣

jbms · 2022-07-08T17:30:53Z

Going forward, can we change NONITPICK to NO_NIT_PICK? I keep reading it as NON_IT_PICK rofl

Makes sense --- or could be: nowarnxref or something --- nitpick is a sphinx-specific term that may be confusing to readers of the C++ source code.

2bndy5 · 2022-07-16T08:26:34Z

ok, I rebased this branch on latest main, and the CI is failing because it now tests for Windows and the fix for that is in my other branch (based on this one).

jbms · 2022-09-01T04:59:13Z

I wonder if we should merge this as an experimental feature, since this PR also contains the default-literal-role and highlight changes that also apply to Python and JSON.

2bndy5 · 2022-09-01T05:15:03Z

We could do that. I've been think on ways to refactor this ext. My priorities are

better parsing mechanism
specify an entity via user friendly implementation

The database and diagnostics are a lesser priority for me. And I assume unit testing will just develop organically.

jbms · 2022-09-01T05:25:59Z

To be clear, by better parsing mechanism, is that in regards to the doxygen syntax, or in regards to the C++ source itself?

For specifying an entity, we could allow it to be specified by its "object name", e.g. foo::bar::Baz or foo::bar::Baz[overload_id] if overloaded. Much more reasonable than the clang USR. Allowing it to be specified by signature, e.g. as supported by the C++ domain for cross references, would be significantly more challenging I think.

2bndy5 · 2022-09-01T05:36:49Z

To be clear, by better parsing mechanism, is that in regards to the doxygen syntax

Yes. I'm not sure how the C++ parsing would need improvement. I still want to provide a config option to turn off the doxygen syntax parsing, so the docstrings can be written in pure RST.

we could allow it to be specified by its "object name", e.g. foo::bar::Baz or foo::bar::Baz[overload_id] if overloaded

This was my thinking as well, but it would be more natural to use the parameter datatypes types for overloads (instead of a docstring specific ID). I find the ability to document multiple overloads with a single docstring very appealing - not even doxygen is capable of that.

jbms · 2022-09-01T05:53:46Z

To be clear, by better parsing mechanism, is that in regards to the doxygen syntax

Yes. I'm not sure how the C++ parsing would need improvement. I still want to provide a config option to turn off the doxygen syntax parsing, so the docstrings can be written in pure RST.

Sounds good.

we could allow it to be specified by its "object name", e.g. foo::bar::Baz or foo::bar::Baz[overload_id] if overloaded

This was my thinking as well, but it would be more natural to use the parameter datatypes types for overloads (instead of a docstring specific ID). I find the ability to document multiple overloads with a single docstring very appealing - not even doxygen is capable of that.

That could be done using the C++ domain symbol resolution used for cross-references, but is a bit tricky because currently the symbol tree only gets populated as the c++ domain directives are actually run, which is too late. We would need to somehow create a separate one and pre-populate it. I also think it is often inconvenient to specify the full signature if you have a lot of template or function parameters.

2bndy5 · 2022-09-01T06:02:36Z

currently the symbol tree only gets populated as the c++ domain directives are actually run

I noticed that. Is there a reason you don't just save the parsed API to a builder env var (on a config-inited event)?

jbms · 2022-09-01T06:08:14Z

There wasn't a need for the symbol tree except for normal xrefs so I just tried to keep it working closely to how it normally does.

Note that the c++ domain symbol lookup is slow if you have a lot of symbols in the same namespace because it just does a linear search over every name --- it would be better if it used a dict.

2bndy5 · 2022-09-01T06:14:17Z

Well I'm ok with this getting merged but there should be a warning on both config and demo doc pages. Possibly even a warning in the build log like you did for graphviz, but that seems a little excessive to me -- you went above and beyond my expectations there (again).

2bndy5 · 2022-09-01T06:17:24Z

Instead of a task list in the thread OP, you could use a github project to track the progress of this extension.

Also adds facilities for saving/restoring default role state.

… options This also ensures that the default roles and highlight language are restored after generated JSON/C++/Python object descriptions are inserted.

Co-authored-by: Brendan <2bndy5@gmail.com>

* fix cross-refs in docs * enable cpp.apigen verbosity and show number of found decls * adjust clang imports for quicker edits * fix windows path separator compensation * support `\` and `@` as cmd prefixes - support for param direction - support for retval cmd - add new strip_comment() to strip all forms of C++ comment syntax from comment tokens' text - use new strip_comment() instead of text.lstrip() This will also preserve consistent indentation that is needed for code-blocks (which isn't supported yet) - modify index_interva.h to test some of the new features * use explicit role for cross-refs * admonition importance of Linux path separators * change admonished text (per request) * change erroneous admonition text * allow blank docstr lines to get normalized - add multiline flag for re.sub(brief/details) call - ran black on api_parser.py * add some unit tests * add a blank line to array.h docstr * change array.h until we support multline comments The indentation removed here (on a subsequent line) is interpretted as a blockquote while still considered within the :param: field. This change also satisfies what the the Doxygen parser expects; meaning unexpected indentation is prohibited amidst single a multi-lined paragraph. * requested changes * try to get raw_comment first * update tests about comment stripping * only dedent multiline comments during stripping adjust tests about leading whitespace for single line comments * [no ci] remove outdated conditional statement * latest review requests - reverted xrefs in Config docs (now that roles are fixed) - removed old algorithm for parsing docstring line-by-line - added regex to remove non-docstring comments from a raw_comment - revised docstring parsing test with pytest.mark.paramtrize - added 2 tests to check for proper removal of non-docstring comments * return None when no raw_comment exists * replace non-doc comments with blank lines also added IDs to the growing parametrized test_comment_styles() * satisfy review request about demo src (`\returns`)

jbms · 2022-09-01T14:11:12Z

I added a warning that it is experimental.

2bndy5 · 2022-09-01T15:50:51Z

Did I mess up the rebase?

jbms · 2022-09-01T15:58:16Z

Did I mess up the rebase?

Not sure what happened with types-clang

2bndy5 · 2022-09-01T16:00:28Z

I thought it was weird when the rebased push didn't trigger the CI (only RTD's CI ran).

stale

setup.py

jbms requested a review from 2bndy5 July 7, 2022 23:00

jbms marked this pull request as draft July 7, 2022 23:00

jbms force-pushed the cpp-apigen branch 3 times, most recently from b16593f to 4266663 Compare July 7, 2022 23:07

2bndy5 mentioned this pull request Jul 7, 2022

Refactor cpp fixes in preparation for C++ autosummary extension #116

Merged

jbms force-pushed the cpp-apigen branch 2 times, most recently from 203fbe4 to 5852fdf Compare July 7, 2022 23:43

jbms force-pushed the cpp-apigen branch from 5852fdf to 204fb6c Compare July 8, 2022 01:14

jbms force-pushed the cpp-apigen branch from 204fb6c to 73d9284 Compare July 8, 2022 01:52

2bndy5 reviewed Jul 8, 2022

View reviewed changes

2bndy5 force-pushed the cpp-apigen branch from 5bc6f6b to 40b4a14 Compare July 16, 2022 08:22

2bndy5 force-pushed the cpp-apigen branch from c0675e9 to d26d812 Compare August 30, 2022 01:09

jbms and others added 7 commits September 1, 2022 07:10

Add default_literal_role

57b7b14

Add C++ apigen extension

fd71c37

Add default-literal-role and highlight-{push,pop} directives

2ad3bf1

Also adds facilities for saving/restoring default role state.

Add {python_apigen,cpp_apigen,json_schema}_rst_{prolog,epilog} config…

10e3ef5

… options This also ensures that the default roles and highlight language are restored after generated JSON/C++/Python object descriptions are inserted.

Apply suggestions from code review

5660e5f

Co-authored-by: Brendan <2bndy5@gmail.com>

Add warning that C++ apigen is experimental

0c7a21e

jbms force-pushed the cpp-apigen branch from d26d812 to 0c7a21e Compare September 1, 2022 14:10

2bndy5 previously approved these changes Sep 1, 2022

View reviewed changes

Add missing types-clang dependency

c303ce2

Add pydantic.mypy plugin

b8711e7

jbms marked this pull request as ready for review September 1, 2022 15:58

2bndy5 reviewed Sep 1, 2022

View reviewed changes

setup.py Outdated Show resolved Hide resolved

Change "libclang" extra -> "cpp"

cb473a0

jbms force-pushed the cpp-apigen branch from 8fc2ae7 to cb473a0 Compare September 1, 2022 16:17

2bndy5 approved these changes Sep 1, 2022

View reviewed changes

jbms merged commit 21f2aee into main Sep 1, 2022

jbms deleted the cpp-apigen branch September 1, 2022 17:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add C++ autosummary extension #117

Add C++ autosummary extension #117

jbms commented Jul 7, 2022 •

edited

Loading

jbms commented Jul 7, 2022

2bndy5 commented Jul 7, 2022

jbms commented Jul 7, 2022

jbms commented Jul 8, 2022

2bndy5 commented Jul 8, 2022

2bndy5 left a comment

2bndy5 Jul 8, 2022

jbms Jul 8, 2022

2bndy5 Jul 8, 2022

2bndy5 Jul 8, 2022

jbms Jul 8, 2022

2bndy5 Jul 8, 2022

2bndy5 Jul 8, 2022 •

edited

Loading

jbms Jul 8, 2022

2bndy5 Jul 8, 2022

jbms Jul 8, 2022

2bndy5 Jul 8, 2022

2bndy5 commented Jul 8, 2022

jbms commented Jul 8, 2022

2bndy5 commented Jul 16, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022 •

edited

Loading

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

2bndy5 commented Sep 1, 2022 •

edited

Loading

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022


		.. rst:directive:: .. cpp-apigen-entity-summary:: entity-name

		Generates a summary of a single Python entity.

	Generates a summary of a single Python entity.
	Generates a summary of a single C++ entity.

Add C++ autosummary extension #117

Add C++ autosummary extension #117

Conversation

jbms commented Jul 7, 2022 • edited Loading

jbms commented Jul 7, 2022

2bndy5 commented Jul 7, 2022

jbms commented Jul 7, 2022

jbms commented Jul 8, 2022

2bndy5 commented Jul 8, 2022

2bndy5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2bndy5 Jul 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2bndy5 commented Jul 8, 2022

jbms commented Jul 8, 2022

2bndy5 commented Jul 16, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022 • edited Loading

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

2bndy5 commented Sep 1, 2022 • edited Loading

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Sep 1, 2022

2bndy5 commented Sep 1, 2022

jbms commented Jul 7, 2022 •

edited

Loading

2bndy5 Jul 8, 2022 •

edited

Loading

2bndy5 commented Sep 1, 2022 •

edited

Loading

2bndy5 commented Sep 1, 2022 •

edited

Loading