refactor app-simplefin.ts to make it easily testable

[tcrasset]

Changes

• splits app-simplefin.js into multiple components:

  • Adds classes representing objects from the Simplefin API so that they are easily usable in the service. These are taken from the Simplefin Protocol Docs
  • HTTPSClient which is responsible for calling any external URL, using Promises
  • SimplefinAPI which uses the injected http_client to call the simplefin API, and parse its response into one of the objects in models/
  • SimplefinService which uses the injected SimplefinAPI to do all the business logic for Actual. Currently, I haven't moved any of the business logic from app-simplefin.js, because I haven't tested these, so I did not want to break stuff. However, the end goal is to have everything except the routing in this service, to that we can test inner logic of the service, with calling our API, nor Simplefin's

   app.post(
     '/accounts',
     handleError(async (req, res) => {
       let accessKey = secretsService.get(SecretName.simplefin_accessKey);
   
       const now = new Date();
   
       try {
         const accounts = await simplefinService.getAccounts(
           accessKey,
           new Date(now.getFullYear(), now.getMonth(), 1),
           new Date(now.getFullYear(), now.getMonth() + 1, 1),
         );
   
       } catch (e) {
         if (e instanceof ServerDown){
           serverDown(e, res);
   
         }
   
         if (e instanceof InvalidToken){
           invalidToken(res);
         }
         return;
       }
   
       res.send({
         status: 'ok',
         data: {
           accounts: accounts.accounts,
         },
       });
     }),
   );

  • unit-tests all the previous components. The code uses dependency injection, so we can create fakes (not mocks) to replace the SimplefinAPI implementation with FakeSimplefinAPI when integration testing the SimplefinService. This will help us in the long run, so that we don't run into mocking hell where everything is mocked and we're not testing the actual code, but the mocks.

• cleans up the standard testing output:

  • add the --silent flag in jest.config.json so that the test output is not littered will all the transactions logging from app-gocardless
  • splits coverage from testing
  • disables the requests loggers
  • adds NODE_NO_WARNINGS=1 to disable the ExperimentalWarnings from VM modules coming from the --experimental-vm-modules flag. Currently, in Node 18 and 20, there is no way to disable this specific warning (or other warning as a matter of fact) directly from the node. [Source]. Instead, some people create jest setup scripts that mock console.warn and filter these warnings. I tried it, it didn't work, so I resorted to the nuclear option that disables all warnings. This is not ideal, but I'm not knowledgeable enough in the JS ecosystem to make it work.

  Now, a standard npm run test output looks all green and clean:

  

• allows us to run typescript files directly by calling npx tsx app.js. node app.js does not allow importing .ts files like simplefin-service.ts. I noticed that we were not running the files from the build/ folder either way (as we were running node app.js), so JavaScript code built from the Typescript code was not used. Might as well run typescript directly.

This required changing the Dockerfiles too.

• allows us to test typescript files directly using jest (which uses ts-node under the hood).

I admit, this was a mess to deal with, lots of flags changed in jest.config.json and tsconfig.json. I would have preferred changing only the code, but I desperately wanted to introduce typescript in the repo, and now it's done. I'm sure this can be improved.

I will need someone to manually test the simplefin integration, to make sure that the current refactor hasn't broken anything. I'm not able to do it, as I don't have a Simplefin Account.

Future

1. Move the business logic from app-simplefin.js to app-simplefin/services/simplefin-service.js. I prefer waiting until @psybers merges his PR Sync multiple accounts in a single SimpleFIN API call. #384 so that I don't break any existing/upcoming features.
2. We will need to, in the future, add integration tests that launches a server as a Dockerfile, and runs integration tests that target the whole chain, i.e. simulate the frontend. For example, we would simulate creating a budget, syncing, changing the key, syncing with the new key. Just a few tests, that cover the server's bread and butter. Not too many because they are expensive, but at least, when the tests pass, we have a better guarantee that the frontend will too.

[tcrasset]

Needed to update typescript to have certain flags in tsconfig.json work.

[tcrasset]

dependency injection, makes it easier to test

[tcrasset]

I tried to mock simplefinService to be able to test app-simplefin.js as an integration test, but I couldn't get jest.spyOn to work, so I didn't. I'm relying on someone checking out this branch and manually testing that I haven't broken anything.

[tcrasset]

I was able to test that, if the SimplefinAPI returns a valid account/transaction, the code in app-simplefin.js works. I just need someone to test with a real SimplefinAPI.

[tcrasset]

If no error is caught on line 100, results should always be set, as well as results.accounts (albeit it being an empty list) if there are any error. In that case, results.errors will have an error.

[tcrasset]

trans.posted is now a Date, so couldn't compare it to 0. Added a method on Transaction to do this.

[tcrasset]

The payee field appears nowhere in the Simplefin documentation, so I think newTrans.payeeName ends up undefined.

[tcrasset]

Hmm, seems this was changed not too long ago, so I guess the Simplefin docs are not quite up to date :/

[tcrasset]

I didn't want to change the logic further down in this function, so I just reverted the transacted_at to use seconds instead of Date objects. getTime() returns milliseconds, so needed to divide by 1000.

[tcrasset]

Separated the api call (fetchAccessKey()) from the state (context), so that the SimplefinAPI remains stateless, and the only state is in SimplefinContextData

[tcrasset]

One would reuse context when trying to fetch accounts right after getting an access key, as the access key would live in context, and could be reused for further calls to the SimplefinAPI.

The SimplefinContextData class is really a proof of concept here, some fields may need to be removed, e.g. method and port, but that can happen at a later date.

    const context = new SimplefinContextData(
      'POST',
      443,
      { 'Content-Length': 0 },
      base64Token,
    );

    this.simplefinApi.setContext(context);

	// Fetch access key
	const accessKey = this.simplefinApi.fetchAccessKey().then(() => context.accessKey)

	// Modify the context to add authentication
    context.parseAccessKey(accessKey);

	// Use the authentified context to make further calls
    const accounts = await this.simplefinApi
      .fetchAccounts(startDate, endDate)
      .then((accounts) => accounts);

[tcrasset]

Right now, I could only test that our API still works and returns invalid token errors in an end-to-end way.

[tcrasset]

I built all the docker images and ran them, making sure the server ran and was accessible with curl.

[tcrasset]

I've gone over the code in the frontend that uses the accounts from the server, and made sure that the variables used there had the same name as the ones in Account (from models/account.ts)

[psybers]

@tcrasset that other PR was merged in.