Integration Tests for the SOM language

.github/workflows/ci.yml

@@ -3,6 +3,26 @@ name: Tests
 on: [push, pull_request]
 
 jobs:
+  python-style:
+    name: Python Style Checks
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Install Black, PyLint and Pytest
+        run: |
+          python -m pip install --upgrade pip
+          pip install black pylint pytest
+
+      - name: Run Black Check
+        run: |
+          black ./IntegrationTests --check --diff
+
+      - name: Run PyLint
+        run: |
+          pylint ./IntegrationTests/test_runner.py ./IntegrationTests/conftest.py
+
   test_soms:
     runs-on: ubuntu-24.04 # ubuntu-latest
     continue-on-error: ${{ matrix.not-up-to-date }}
@@ -166,6 +186,15 @@ jobs:
 
           echo "${{ matrix.som }} $SOM_TESTS"
           eval "${{ matrix.som }} $SOM_TESTS"
+
+      - name: Run Integration Tests
+        if: ${{ matrix.som != 'spec' }}
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytest
+          export EXECUTABLE="som-vm/${{ matrix.som }}"
+          export CLASSPATH=Smalltalk
+          pytest IntegrationTests
 
       # We currently test SomSom only on TruffleSOM
       - name: Test SomSom on TruffleSOM

IntegrationTests/README.md

@@ -0,0 +1,140 @@
+# SOM Integration Tests
+
+Most of the tests for the integration testing come from lang_tests of [yksom](https://github.com/softdevteam/yksom/tree/master/lang_tests). Tests are identified by their path from core-lib to test.som, this ensures there can be multiple tests named test.som in different directories.
+
+## Running the Integration Tests
+The tests can be run using pytest by simply running pytest in the base directory of any SOM implementation that includes a version of the core-library with IntegrationTests. It requires multiple python modules installed and environment variables set.
+
+### Simple Test Run
+```
+VM=./path-to-build CLASSPATH=./core-lib/Smalltalk python3 -m pytest
+```
+
+### Optionals
+A set of optionals have been created for this test suite which can be added.
+
+#### Executing the test_runner tests
+This optional flag will not execute any normal SOM tests but instead test the test_runner.
+```
+VM=./path-to-build CLASSPATH=./core-lib/Smalltalk python3 -m pytest -m tester
+```
+
+## Prerequisites
+- [PyYaml](https://pypi.org/project/PyYAML/)
+- [Pytest](https://pypi.org/project/pytest/)
+
+
+## Environment variables
+The environment variables are split into required and optional. Some optionals may be required for different implementations of SOM.
+
+#### VM
+This is the path from the current working directory to the executable VM of SOM.
+#### CLASSPATH
+The exact classpath required by SOM to find the Object class etc.
+#### TEST_EXCEPTIONS
+A yaml file which details the tags of tests. Specifically it labels tests that are expected to fail for one reason or another. **Give the whole path to the file**.
+#### GENERATE_REPORT
+Generates a yaml file which can be used as a **TEST_EXCEPTIONS** file. It will also include additional information about how many tests passed, which tests passed that were not expected to and which tests failed. **Give a full path from CWD to where it should be saved including .yaml**.
+
+## TEST_EXCEPTIONS (How to write a file)
+There are four tags that are currently supported by the SOM integration tests. All tags will run the tests still, other than do_not_run, but will not fail on test failure, a tagged test will cause the run to fail only when it passes unexpectedly. Check for example file IntegrationTests/test_tags.yaml.
+
+For a test to be given a tag specify it's location path like this:
+```
+known_failures:
+    core-lib/IntegrationTests/Tests/test.som
+```
+
+### known_failures
+Any test located in this tag is assumed to fail, it should only be used when another more suitable tag is not available.
+
+### failing_as_unspecified
+Any test located in this tag failed because SOM does not specify behaviour in this instance, this means that each implementation may treat this situation differently. *Example dividing by 0.*
+
+### unsupported
+Any test located here has a feature which is not suppoprted in this SOM.
+
+### do_not_run
+This test should not be ran ever as it causes an error in the python level code. The test may also cause a SOM level error but does not have to. (*This does not include Unicode errors, they are handled at runtime*)
+
+## How to write a new test
+For a test to be collected by Pytest it has to start with a comment, the comment should be structured with the expected output for either stderr or stdout.
+
+```
+"
+VM:
+    status: error
+    custom_classpath: ./core-lib/Examples/AreWeFastYet/Core:./core-lib/Smalltalk
+    case_sensitive: False
+    stdout:
+        1000
+        ...
+        2 is an integer
+    stderr:
+        ...
+        ERROR MESSAGE
+"
+```
+
+**When structuring a test all options must come before stderr and stdout**
+
+### Tags for structuring a test
+Below is a list of tags which structure how a test works.
+
+#### VM: 
+This is required as the base of the test structure and what allows the tests to be identified as an integration test.
+
+#### custom_classpath: 
+This allows for the specification of a custom classpath to be used. This is useful for loading different versions of classes with the same name. I.e. AWFY Vector instead of core-lib Vector. **The path to ./Smalltalk must still be specified after so that the Object class can be loaded**
+
+#### case_sensitive
+By default the tests are case insensitive (All outputs and expecteds are converted to be lower case) but by specifying True in case_sensitive that test can be checked as case_sensitive.
+
+#### stderr or stdout:
+This is your expected output, each new line will be a new "thing" to check for. Writing ... signifies a gap in checking, the output does not have to feature this gap but may do. For example:
+
+```
+stdout:
+    Hello, World
+    ...
+    Goodbye
+```
+This would be true for:
+```
+Hello, World
+Today is a Monday
+Goodbye
+
+/
+
+Hello, World
+Goodbye
+```
+
+### Understanding how the "..." works in test_runner
+There are situations where the ... is necessary for your output. Here are some example use cases, when they may be necessary and how to write the tests for it. As a preface the check_output will check a line as a whole so writing ... allows for a gap, a more precise check can be made by including as much of the expected output as possible.
+
+Line 1 in the below expected stdout says match on a whole line which has Hello, some other text as a gap then the word sample then whatever comes after on that line. Line 2 specifies that we must end with the word line. Whilst line 3 says somewhere in this line the word little must appear.
+
+#### Stdout
+```
+This is SOM++
+Hello, this is some sample output
+There is some more on this line
+And a little more here
+```
+
+#### Expected
+```
+VM:
+    status: success
+    case_sensitive: False
+    stdout:
+        Hello, ... sample ...
+        ... is ... this line
+        ... little ...
+```
+
+### When not to use "..."
+- When the word you are searching for is the end of the line do not do this "*word* ...".
+- When the word you are searching for is at the beginning of the line do not do this "... *word*"

IntegrationTests/SOMpp_test_tags.yaml

@@ -0,0 +1,111 @@
+known_failures:
+  - core-lib/IntegrationTests/Tests/mutate_superclass_method/test.som
+  - core-lib/IntegrationTests/Tests/instance_fields_overlap/test.som
+  - core-lib/IntegrationTests/Tests/fromstring.som
+  - core-lib/IntegrationTests/Tests/string_length.som
+  - core-lib/IntegrationTests/Tests/double2.som
+  - core-lib/IntegrationTests/Tests/double13.som
+  - core-lib/IntegrationTests/Tests/int26.som
+  - core-lib/IntegrationTests/Tests/integer_asdouble.som
+  - core-lib/IntegrationTests/Tests/int_modulus.som
+  - core-lib/IntegrationTests/Tests/int32.som
+  - core-lib/IntegrationTests/Tests/round.som
+  - core-lib/IntegrationTests/Tests/mutate_methods.som
+  - core-lib/IntegrationTests/Tests/double_asinteger.som
+  - core-lib/IntegrationTests/Tests/int_double_div.som
+  - core-lib/IntegrationTests/Tests/int30.som
+  - core-lib/IntegrationTests/Tests/int3.som
+  - core-lib/IntegrationTests/Tests/double_double_div.som
+  - core-lib/IntegrationTests/Tests/double_modulus.som
+  - core-lib/IntegrationTests/Tests/int24.som
+  - core-lib/IntegrationTests/Tests/double4.som
+  - core-lib/IntegrationTests/Tests/int19.som
+  - core-lib/IntegrationTests/Tests/system_global.som
+  - core-lib/IntegrationTests/Tests/perform_unknown.som
+  - core-lib/IntegrationTests/Tests/binary_super.som
+  - core-lib/IntegrationTests/Tests/shift_right.som
+  - core-lib/IntegrationTests/Tests/cos.som
+  - core-lib/IntegrationTests/Tests/array_literals.som
+  - core-lib/IntegrationTests/Tests/fromstring_double.som
+  - core-lib/IntegrationTests/Tests/sin.som
+  - core-lib/IntegrationTests/Tests/double1.som
+  - core-lib/IntegrationTests/Tests/int5.som
+  - core-lib/IntegrationTests/Tests/nested_backtrace2.som
+  - core-lib/IntegrationTests/Tests/int28.som
+  - core-lib/IntegrationTests/Tests/int21.som
+  - core-lib/IntegrationTests/Tests/int14.som
+  - core-lib/IntegrationTests/Tests/array_at_idx2_err.som
+  - core-lib/IntegrationTests/Tests/int10.som
+  - core-lib/IntegrationTests/Tests/int13.som
+  - core-lib/IntegrationTests/Tests/perform_string.som
+  - core-lib/IntegrationTests/Tests/double_double_div_err.som
+  - core-lib/IntegrationTests/Tests/unknown_field_write.som
+  - core-lib/IntegrationTests/Tests/array_at_idx0_err.som
+  - core-lib/IntegrationTests/Tests/str_escape_unknown.som
+  - core-lib/IntegrationTests/Tests/int9.som
+  - core-lib/IntegrationTests/Tests/int16.som
+  - core-lib/IntegrationTests/Tests/double5.som
+  - core-lib/IntegrationTests/Tests/system_global_lookup_string.som
+  - core-lib/IntegrationTests/Tests/double8.som
+  - core-lib/IntegrationTests/Tests/remainder_zero.som
+  - core-lib/IntegrationTests/Tests/perform_witharguments_wrong.som
+  - core-lib/IntegrationTests/Tests/load_string.som
+  - core-lib/IntegrationTests/Tests/shift_right_type_err.som
+  - core-lib/IntegrationTests/Tests/int22.som
+  - core-lib/IntegrationTests/Tests/double_double_div_zero_err3.som
+  - core-lib/IntegrationTests/Tests/system2.som
+  - core-lib/IntegrationTests/Tests/int_double_div_err.som
+  - core-lib/IntegrationTests/Tests/exit_double.som
+  - core-lib/IntegrationTests/Tests/inst_var_at_bad_idx.som
+  - core-lib/IntegrationTests/Tests/double7.som
+  - core-lib/IntegrationTests/Tests/array_at_negative_idx_err.som
+  - core-lib/IntegrationTests/Tests/double6.som
+  - core-lib/IntegrationTests/Tests/arbint_modulus_err.som
+  - core-lib/IntegrationTests/Tests/inst_var_at_put_bad_idx.som
+  - core-lib/IntegrationTests/Tests/exit_string.som
+  - core-lib/IntegrationTests/Tests/int_modulus_err.som
+  - core-lib/IntegrationTests/Tests/shift_right_too_big.som
+  - core-lib/IntegrationTests/Tests/int20.som
+  - core-lib/IntegrationTests/Tests/int8.som
+  - core-lib/IntegrationTests/Tests/fromstring_double_err.som
+  - core-lib/IntegrationTests/Tests/int_double_div_zero_err.som
+  - core-lib/IntegrationTests/Tests/exit_int_too_big.som
+  - core-lib/IntegrationTests/Tests/instance_fields_overlap2.som
+  - core-lib/IntegrationTests/Tests/fromstring_err.som
+  - core-lib/IntegrationTests/Tests/int31.som
+  - core-lib/IntegrationTests/Tests/int23.som
+  - core-lib/IntegrationTests/Tests/array_at_put_idx2_err.som
+  - core-lib/IntegrationTests/Tests/double_double_div_zero_err2.som
+  - core-lib/IntegrationTests/Tests/double11.som
+  - core-lib/IntegrationTests/Tests/double9.som
+  - core-lib/IntegrationTests/Tests/nested_backtrace1.som
+  - core-lib/IntegrationTests/Tests/int11.som
+  - core-lib/IntegrationTests/Tests/double_modulus_err.som
+  - core-lib/IntegrationTests/Tests/double3.som
+  - core-lib/IntegrationTests/Tests/double_double_div_zero_err1.som
+  - core-lib/IntegrationTests/Tests/array_at_put_idx0_err.som
+  - core-lib/IntegrationTests/Tests/int27.som
+  - core-lib/IntegrationTests/Tests/double_double_div_zero_err4.som
+  - core-lib/IntegrationTests/Tests/int12.som
+  - core-lib/IntegrationTests/Tests/arbint_double_div_err.som
+  - core-lib/IntegrationTests/Tests/arbint_double_div_zero_err.som
+  - core-lib/IntegrationTests/Tests/system_global_put_string.som
+  - core-lib/IntegrationTests/Tests/int25.som
+  - core-lib/IntegrationTests/Tests/array_at_idx_err.som
+  - core-lib/IntegrationTests/Tests/int17.som
+  - core-lib/IntegrationTests/Tests/int15.som
+  - core-lib/IntegrationTests/Tests/is_letters.som
+  - core-lib/IntegrationTests/Tests/obj2.som
+  - core-lib/IntegrationTests/Tests/to32bits.som
+  - core-lib/IntegrationTests/Tests/int4.som
+  - core-lib/IntegrationTests/Tests/int29.som
+  - core-lib/IntegrationTests/Tests/double12.som
+failing_as_unspecified:
+  - core-lib/IntegrationTests/Tests/ic1.som
+  - core-lib/IntegrationTests/Tests/mutate_fields.som
+  - core-lib/IntegrationTests/Tests/test_literals_limit_1.som
+  - core-lib/IntegrationTests/Tests/substring.som
+  - core-lib/IntegrationTests/Tests/call2.som
+unsupported:
+  - null
+do_not_run: