Establishing generic parselets and functions #105

phorward · 2023-03-04T10:30:58Z

About

This PR is intended to coordinate generic parselets & functions in Tokay as established in #10.

To- do

Currently without usage. Still have to find a starting point.

The previous implementation did not have any real advantage, except making the code more complex at this point.

...for better distinction.

Unfinished intermediate commit.

Introduces ImlValue::Generic to describe a generic usage. Removes constants from ImlParselet and making it part of ImlValue::Parselet. Yet without functionality. Under further consideration.

This is yet another intermediate commit. Compiler::get() returns any ImlValue with a name. ImlOp and ImlValue will merge as well in their meaning.

Intermediate commit with a lot of stuff todo.

ImlProgram is a better name for the linker, to glue intermediate parts together into a program.

Squashed commit of the following: commit a5c658f Author: Jan Max Meyer <jmm@phorward.de> Date: Wed Apr 19 20:57:13 2023 +0200 ImlOp::compile_to_vec() commit 5946136 Author: Jan Max Meyer <jmm@phorward.de> Date: Wed Apr 19 20:10:44 2023 +0200 ImlValue::compile_load() & ImlValue::compile_call() commit 6566246 Author: Jan Max Meyer <jmm@phorward.de> Date: Tue Apr 18 20:33:41 2023 +0200 ImlProgram::finalize() improvements commit de8932e Author: Jan Max Meyer <jmm@phorward.de> Date: Tue Apr 18 18:42:31 2023 +0200 wip: Separated self-contained ImlProgram::finalize() commit 19acfce Author: Jan Max Meyer <jmm@phorward.de> Date: Tue Apr 18 00:36:55 2023 +0200 wip: Move finalization entirely into ImlProgram commit 45a1570 Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Apr 17 23:26:40 2023 +0200 wip: Change signature of ImlOp::compile commit 77e247f Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Apr 17 22:58:48 2023 +0200 wip: Rename Linker into ImlProgram ImlProgram is a better name for the linker, to glue intermediate parts together into a program. commit bef35e8 Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Apr 17 00:20:24 2023 +0200 . commit c12f1fc Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Apr 16 23:31:34 2023 +0200 wip: ImlValue::Shared Intermediate commit with a lot of stuff todo. commit d4ee396 Author: Jan Max Meyer <jmm@phorward.de> Date: Wed Apr 5 00:43:11 2023 +0200 wip: get clarification for data model This is yet another intermediate commit. Compiler::get() returns any ImlValue with a name. ImlOp and ImlValue will merge as well in their meaning. commit 0010a84 Author: Jan Max Meyer <jmm@phorward.de> Date: Wed Mar 22 13:33:51 2023 +0100 ImlValue::Generic & ImlParselet clean-up Introduces ImlValue::Generic to describe a generic usage. Removes constants from ImlParselet and making it part of ImlValue::Parselet. Yet without functionality. Under further consideration. commit 4eff813 Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Mar 20 15:33:21 2023 +0100 Substitute Option<ImlValue> by ImlValue::Void commit 50a5df8 Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Mar 20 12:31:17 2023 +0100 First traversal of value_generic/genarg Unfinished intermediate commit. commit 9e4bb42 Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Mar 19 16:21:40 2023 +0100 Rename ImlValue::value() to ImlValue::into_refvalue() commit 6f0d981 Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Mar 19 16:15:36 2023 +0100 Renaming iml-modules into appropriate filenames ...for better distinction. commit 19c8b1c Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Mar 19 15:09:58 2023 +0100 Merge ImlTarget into ImlValue The previous implementation did not have any real advantage, except making the code more complex at this point. commit 2b2d89d Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Mar 19 11:33:43 2023 +0100 Check for invalid parselet instances commit fe9a1bb Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Mar 13 22:56:18 2023 +0100 Let's use ImlValue::Parselet for instances as well Currently without usage. Still have to find a starting point. commit 43353b8 Author: Jan Max Meyer <jmm@phorward.de> Date: Sat Mar 11 12:29:50 2023 +0100 ImlValue::ParseletInstance? commit 3482a3d Merge: 2483e83 09e376c Author: Jan Max Meyer <jmm@phorward.de> Date: Mon Mar 6 18:49:22 2023 +0100 Merge branch 'main' into generic-parselets commit 2483e83 Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Mar 5 23:27:31 2023 +0100 temporarily disabling prelude commit 06d12fa Merge: 34e738e 0f1d00c Author: Jan Max Meyer <jmm@phorward.de> Date: Sat Mar 4 11:01:27 2023 +0100 Merge branch 'main' into generic-parselets commit 34e738e Merge: ae9a4b7 d51ee23 Author: Jan Max Meyer <jmm@phorward.de> Date: Thu Feb 23 00:29:11 2023 +0100 Merge branch 'main' into generic-parselets commit ae9a4b7 Author: Jan Max Meyer <jmm@phorward.de> Date: Sun Feb 19 00:16:48 2023 +0100 Some renamings commit 6b297f1 Author: Jan Max Meyer <jmm@phorward.de> Date: Sat Feb 18 22:26:51 2023 +0100 Unify naming for constants and signature commit 53b86f1 Author: Jan Max Meyer <jmm@phorward.de> Date: Sat Feb 18 22:23:53 2023 +0100 Use IndexMap for constants and signature

ImlSharedParselet can't run into infinite recursion when its recursively dumped.

The `+=`-operator uses `list_extend()`, which generates different results when P returns a list.

…generic-parselets

... and tested behavior with generated parselets. Broken, still WIP.

…t-recursion

Finally remove deprecated stuff as it is replaced by Tokay internals for now.

This is useful for debugging and error reporting

- Re-enables reporting of unused symbols - Improves reporting of unresolved values that are registered - Improves reporting of invalid generic configurations This commit also showed up that ImlValue::Shared is a bug problem, and was a conceptual mistake, making things heavy and blurry. Therefore, a huger rework of the ImlValue is scheduled, and the Iml*-stuff in general.

- Renamed `ImlParseletConfig` into `ImlParseletInstance` - Removed `ImlParselet.is_generic()` - Moved required generic argument checking into `ImlParselet.derive()`

phorward · 2023-10-22T18:28:13Z

This pull request is now ready to be merged.

I'm a little unhappy with the current ImlValue implementation, but this is the only way to adequately solve the problem at the moment. Anyway, this should be done better or differently, and I'm sure it will.

All that's missing now is some documentation to describe the new features.

By implementing build-ins with the help of Tokay itself, #120 will be another big milestone to significantly reduce the size of the generated code and thus make Tokay much more performant again like the previous solution. However, this step here has to be to raise Tokay to another level.

phorward added 26 commits February 18, 2023 22:23

Use IndexMap for constants and signature

53b86f1

Unify naming for constants and signature

6b297f1

Some renamings

ae9a4b7

Merge branch 'main' into generic-parselets

34e738e

Merge branch 'main' into generic-parselets

06d12fa

temporarily disabling prelude

2483e83

Merge branch 'main' into generic-parselets

3482a3d

ImlValue::ParseletInstance?

43353b8

Let's use ImlValue::Parselet for instances as well

fe9a1bb

Currently without usage. Still have to find a starting point.

Check for invalid parselet instances

2b2d89d

Merge ImlTarget into ImlValue

19c8b1c

The previous implementation did not have any real advantage, except making the code more complex at this point.

Renaming iml-modules into appropriate filenames

6f0d981

...for better distinction.

Rename ImlValue::value() to ImlValue::into_refvalue()

9e4bb42

First traversal of value_generic/genarg

50a5df8

Unfinished intermediate commit.

Substitute Option<ImlValue> by ImlValue::Void

4eff813

ImlValue::Generic & ImlParselet clean-up

0010a84

Introduces ImlValue::Generic to describe a generic usage. Removes constants from ImlParselet and making it part of ImlValue::Parselet. Yet without functionality. Under further consideration.

wip: get clarification for data model

d4ee396

This is yet another intermediate commit. Compiler::get() returns any ImlValue with a name. ImlOp and ImlValue will merge as well in their meaning.

wip: ImlValue::Shared

c12f1fc

Intermediate commit with a lot of stuff todo.

.

bef35e8

wip: Rename Linker into ImlProgram

77e247f

ImlProgram is a better name for the linker, to glue intermediate parts together into a program.

wip: Change signature of ImlOp::compile

45a1570

wip: Move finalization entirely into ImlProgram

19acfce

wip: Separated self-contained ImlProgram::finalize()

de8932e

ImlProgram::finalize() improvements

6566246

ImlValue::compile_load() & ImlValue::compile_call()

5946136

ImlOp::compile_to_vec()

a5c658f

phorward added 3 commits April 19, 2023 21:10

Merge branch 'main' into generic-parselets

0c14cd9

Merge branch 'main' into generic-parselets

089b122

Improving debug / ImlSharedParselet

a48e340

ImlSharedParselet can't run into infinite recursion when its recursively dumped.

phorward added 7 commits September 28, 2023 23:56

Fix preluded' Repeat to use list_push() instead of +=

92d9f59

The `+=`-operator uses `list_extend()`, which generates different results when P returns a list.

Merge remote-tracking branch 'phorward/generic-parselets-fixes' into …

216a222

…generic-parselets

Modifiers with assumed severity

b5a16c5

Improving context debug

dc8e24b

Fixing left-recursion detection, improving debug

cf5ed31

... and tested behavior with generated parselets. Broken, still WIP.

Modifying parselet_leftrec.tok to fulfill current capabilities on lef…

c708a59

…t-recursion

Removal of ImlOp::Repeat

d4ac089

Finally remove deprecated stuff as it is replaced by Tokay internals for now.

This was referenced Oct 5, 2023

Requirement for inlined parselets #120

Open

Implement builtin Keyword<P> #121

Closed

Implement indirect left-recursion #95

Open

Store name with compile-time local and global

307c7fc

This is useful for debugging and error reporting

phorward removed the tokay label Oct 8, 2023

phorward added 10 commits October 12, 2023 22:49

Fixing generic argument requirement error reporting

387dada

Replace ImlValue::Void by ImlValue::Unset

ff60c5f

Renamed ImlValue::Shared into ImlValue::Unresolved

6db705e

Removal of dead error reporting

5633b2c

Merge ImlValue::Local/Global into ImlValue::Variable

4857a96

Some renamings and docstring fixes

6bba5d2

Renamings and restructuring

755aaed

- Renamed `ImlParseletConfig` into `ImlParseletInstance` - Removed `ImlParselet.is_generic()` - Moved required generic argument checking into `ImlParselet.derive()`

Updating docstrings and adding tests

d438840

Fixing Expect to report correct offset

d4ba713

phorward mentioned this pull request Oct 22, 2023

Bug with Repeat<P>: Match in Pos<P> sequence fails without severity #123

Closed

Documentation for prelude.tok and fixing bug tokay-lang#123

03bdb0e

phorward marked this pull request as ready for review October 22, 2023 18:20

phorward mentioned this pull request Oct 23, 2023

feat: New list syntax with enforced ,-operator #100

Merged

phorward merged commit 2b75daf into tokay-lang:main Oct 23, 2023
3 checks passed

phorward deleted the generic-parselets branch December 25, 2023 20:40

phorward restored the generic-parselets branch December 25, 2023 20:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Establishing generic parselets and functions #105

Establishing generic parselets and functions #105

phorward commented Mar 4, 2023 •

edited

Loading

phorward commented Oct 22, 2023

Establishing generic parselets and functions #105

Establishing generic parselets and functions #105

Conversation

phorward commented Mar 4, 2023 • edited Loading

About

To- do

phorward commented Oct 22, 2023

phorward commented Mar 4, 2023 •

edited

Loading