Skip to content

v7 #416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 104 commits into
base: main
Choose a base branch
from
Open

v7 #416

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
c5c8b0e
pnpm + stricli
bencmbrook Jul 8, 2025
2cb73a1
dist
bencmbrook Jul 8, 2025
33b640d
lib folder
bencmbrook Jul 8, 2025
d991421
test
bencmbrook Jul 8, 2025
18dfb42
rm mocha chai
bencmbrook Jul 8, 2025
ee529b0
rm mocha chai
bencmbrook Jul 8, 2025
099c5a3
alpha.2
bencmbrook Jul 8, 2025
ec7b4c4
dirname
bencmbrook Jul 8, 2025
9a8747f
run-vitest
bencmbrook Jul 8, 2025
9239dec
tsx
bencmbrook Jul 8, 2025
a0263d5
description
bencmbrook Jul 8, 2025
5d68c9c
start cmd
bencmbrook Jul 8, 2025
bed2c72
impl cli-pull
bencmbrook Jul 8, 2025
67026b8
esm, replace bluebird, lodash-es
bencmbrook Jul 8, 2025
f1deaf6
fix test
bencmbrook Jul 8, 2025
839fe41
vitest config
bencmbrook Jul 8, 2025
05b4fe7
specifiers
bencmbrook Jul 8, 2025
5473ca6
cli-push
bencmbrook Jul 8, 2025
9a908e5
derive-data-silos-from-data-flows
bencmbrook Jul 8, 2025
d7a389d
derive-data-silos-from-data-flows-cross-instance
bencmbrook Jul 8, 2025
81e98ff
discover-silos
bencmbrook Jul 8, 2025
311c8f4
pull-datapoints
bencmbrook Jul 8, 2025
b276111
cli-pull-unstructured-discovery-files
bencmbrook Jul 8, 2025
180fc88
cli-scan-packages
bencmbrook Jul 8, 2025
a4c7ea2
generate-api-keys
bencmbrook Jul 8, 2025
30fd738
build-xdi-sync-endpoint
bencmbrook Jul 8, 2025
5c8c813
consent-manager-service-json-to-yml
bencmbrook Jul 8, 2025
c516919
consent-managers-to-business-entities
bencmbrook Jul 8, 2025
0f233a6
pull-consent-metrics
bencmbrook Jul 8, 2025
9249423
pull-consent-preferences
bencmbrook Jul 8, 2025
08128e8
update-consent-manager
bencmbrook Jul 8, 2025
c8f2482
upload-consent-preferences
bencmbrook Jul 8, 2025
8006090
upload-cookies-from-csv
bencmbrook Jul 8, 2025
4b1bc2b
upload-data-flows-from-csv
bencmbrook Jul 8, 2025
95c0a08
upload-preferences
bencmbrook Jul 8, 2025
2e2dc7d
sync-ot
bencmbrook Jul 8, 2025
4688710
request-approve
bencmbrook Jul 8, 2025
a26ba89
request-cancel
bencmbrook Jul 8, 2025
2cf3e1e
cron-mark-identifiers-completed
bencmbrook Jul 8, 2025
ac41329
cron-pull-identifiers
bencmbrook Jul 8, 2025
f2b61db
request-download-files
bencmbrook Jul 8, 2025
d94e32f
enricher-restart
bencmbrook Jul 8, 2025
5f042d4
request-export
bencmbrook Jul 8, 2025
be61b4d
mark-silent
bencmbrook Jul 8, 2025
aab9dfe
notify-additional-time
bencmbrook Jul 8, 2025
77e41f1
preflight pull-identifiers
bencmbrook Jul 8, 2025
81fa359
manual-enrichment-push-identifiers
bencmbrook Jul 8, 2025
e857bd5
request-reject-unverified-identifiers
bencmbrook Jul 8, 2025
2ada1c7
request restart
bencmbrook Jul 8, 2025
a414efa
skip-preflight-jobs
bencmbrook Jul 8, 2025
ef262fd
mark-request-data-silos-completed
bencmbrook Jul 8, 2025
0694ef3
retry-request-data-silos
bencmbrook Jul 8, 2025
a86988e
skip-request-data-silos
bencmbrook Jul 8, 2025
71fb646
request-upload
bencmbrook Jul 8, 2025
02652af
alpha.3
bencmbrook Jul 8, 2025
a4f31b8
lint
bencmbrook Jul 8, 2025
49e3846
fix broken subpath imports which dont support esm exports
bencmbrook Jul 8, 2025
7867dbe
rm got subpath
bencmbrook Jul 8, 2025
3a9dbef
build readme
bencmbrook Jul 8, 2025
9baccd5
txt
bencmbrook Jul 8, 2025
5089f7d
parse
bencmbrook Jul 8, 2025
56bedd5
alpha.5
bencmbrook Jul 8, 2025
0f50c1a
readme. todo port important docs from old reference
bencmbrook Jul 10, 2025
06ed587
add old reference temporarily
bencmbrook Jul 10, 2025
2acfcb2
typesafe pullResources doc generation
bencmbrook Jul 10, 2025
bc28384
typesafe pullResources doc generation
bencmbrook Jul 10, 2025
11d8bb5
tr-pull docs
bencmbrook Jul 10, 2025
fe54ed0
fixes to pull/push docs being overly integration-specific
bencmbrook Jul 10, 2025
e6c72e8
structure
bencmbrook Jul 10, 2025
77cf660
additional docs structure
bencmbrook Jul 10, 2025
4c1336f
readme
bencmbrook Jul 10, 2025
55b07b0
push/pull readme done
bencmbrook Jul 10, 2025
5b41ee8
scan packages / silos
bencmbrook Jul 10, 2025
73e378c
readme
bencmbrook Jul 10, 2025
7ad01c0
consent pull/push
bencmbrook Jul 10, 2025
081cf25
upload cookies/data flows from csv
bencmbrook Jul 11, 2025
b34a35c
readme for generate api keys
bencmbrook Jul 11, 2025
98cb178
readme gen
bencmbrook Jul 11, 2025
8a04df9
merge main
bencmbrook Jul 11, 2025
29cecfb
Add tests for legacy command mapping and codebase structure
bencmbrook Jul 11, 2025
ff087f2
missing test file
bencmbrook Jul 11, 2025
efbe8bd
merge main
bencmbrook Jul 11, 2025
3c383e1
add undocumented pull-profiles cmd
bencmbrook Jul 11, 2025
d644513
fixes
bencmbrook Jul 11, 2025
18568a8
Move consent-manager-service-json-to-yml and consent-managers-to-busi…
bencmbrook Jul 11, 2025
6836427
alpha.6
bencmbrook Jul 11, 2025
f20a4c9
legacy command mapping update
bencmbrook Jul 11, 2025
e7ac9e9
alpha.7
bencmbrook Jul 11, 2025
3d17a7f
proxy
bencmbrook Jul 11, 2025
ff9ba2f
add readme to generate check
bencmbrook Jul 11, 2025
c0cb970
add pull-profiles to routes
bencmbrook Jul 11, 2025
f10c68a
alpha.8
bencmbrook Jul 11, 2025
a49ca57
Add Developers.md
bencmbrook Jul 11, 2025
9e2eb1b
update readme
bencmbrook Jul 11, 2025
0df91ab
7.0.0
bencmbrook Jul 11, 2025
0653f7e
add
bencmbrook Jul 11, 2025
953b652
rm console.log
bencmbrook Jul 11, 2025
1b3a967
readme nits
bencmbrook Jul 11, 2025
1b19f15
transcend.yml schema
bencmbrook Jul 11, 2025
3ac347a
readme nit
bencmbrook Jul 11, 2025
46a4241
doctoc the CHANGELOG
bencmbrook Jul 11, 2025
5a6528c
hide pull-identifiers cmd and regenerate docs
bencmbrook Jul 11, 2025
e6507c3
developers.md update
bencmbrook Jul 11, 2025
cd2c979
env defaults for URL and consentUrl
bencmbrook Jul 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
12 changes: 7 additions & 5 deletions .cursor/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
ARG VARIANT=16-bullseye
ARG VARIANT=18-bullseye
FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-${VARIANT}

USER root
Expand Down Expand Up @@ -32,8 +32,10 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
&& mkdir -p /root/.local/share \
&& tldr -u

# Install specific yarn version globally
RUN npm install -g "yarn@2.4.3"
# Install pnpm globally
RUN npm i -g corepack@latest
RUN corepack enable
RUN corepack install

# Install pre-commit globally (this doesn't need the repo to be present)
RUN pip3 install pre-commit
Expand All @@ -47,6 +49,6 @@ WORKDIR /home/ubuntu

# The agent will handle:
# 1. Cloning the repository
# 2. Running `yarn install`
# 3. Running `yarn tsc --build`
# 2. Running `pnpm install`
# 3. Running `pnpm exec tsc --build`
# 4. Setting up pre-commit hooks
24 changes: 12 additions & 12 deletions .cursor/environment.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,35 +3,35 @@
"context": ".",
"dockerfile": "Dockerfile"
},
"install": "yarn install",
"install": "pnpm install",
"terminals": [
{
"name": "yarn build",
"name": "pnpm build",
"description": "Build the project",
"command": "yarn build"
"command": "pnpm build"
},
{
"name": "yarn clean",
"name": "pnpm clean",
"description": "Clean the project",
"command": "yarn clean"
"command": "pnpm clean"
},
{
"name": "yarn lint",
"name": "pnpm lint",
"description": "Lint the project",
"command": "yarn lint"
"command": "pnpm lint"
},
{
"name": "yarn prepublish",
"name": "pnpm prepublish",
"description": "Pre-publish the project",
"command": "yarn prepublish"
"command": "pnpm prepublish"
},
{
"name": "yarn test",
"name": "pnpm test",
"description": "Test the project",
"command": "yarn test"
"command": "pnpm test"
},
{
"name": "pre-commit",
"name": "pnpm pre-commit",
"description": "Run pre-commit hooks on changed files",
"command": "pre-commit"
},
Expand Down
2 changes: 1 addition & 1 deletion .depcheckrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ignores: ["@yarnpkg/sdks", "depcheck", "prettier"]
ignores: ["depcheck", "prettier", "doctoc"]
2 changes: 1 addition & 1 deletion .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
ARG VARIANT=16-bullseye
ARG VARIANT=18-bullseye
FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-${VARIANT}

USER root
Expand Down
2 changes: 0 additions & 2 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@
"customizations": {
"vscode": {
"extensions": [
"arcanis.vscode-zipfs",
"slevesque.vscode-zipexplorer",
"dbaeumer.vscode-eslint",
"esbenp.prettier-vscode",
"bcanzanella.openmatchingfiles",
Expand Down
2 changes: 1 addition & 1 deletion .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ services:
context: ..
dockerfile: .devcontainer/Dockerfile
args:
VARIANT: 16-bullseye
VARIANT: 18-bullseye

volumes:
- ..:/workspaces/main:cached
Expand Down
2 changes: 0 additions & 2 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
.gitignore
.git/*
.yarn/*
**/docs/*
**/package.json
**/node_modules/*
**/build/*
**/dist/*
**/builds/*
**/coverage/*
**/.yarn/*
*.testjs
*.testts
examples/code-scanning/test-package-json/src/*
Expand Down
29 changes: 29 additions & 0 deletions .eslintrc.js → .eslintrc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ module.exports = {

/** handled by no-restricted-syntax */
'guard-for-in': 0,
'no-restricted-syntax': 0,
'no-await-in-loop': 0,

/**
* We prefer to use types instead of interfaces
Expand Down Expand Up @@ -249,6 +251,33 @@ module.exports = {
'@typescript-eslint/ban-types': ['error'],
'@typescript-eslint/no-explicit-any': ['error'],
},
overrides: [
{
files: ['src/commands/**/*'],
rules: {
'max-len': 'off',
'max-lines': 'off',
'jsdoc/require-jsdoc': 'off',
},
},
{
files: [
'**/*.test.ts',
'**/*.test.js',
'**/*.spec.ts',
'**/*.spec.js',
'./scripts/**/*',
],
rules: {
'import/no-extraneous-dependencies': [
'error',
{
devDependencies: true,
},
],
},
},
],
settings: {
/** Allow for typescript alias resolution */
'import/resolver': {
Expand Down
36 changes: 36 additions & 0 deletions .github/commands/dependencies/install_deps/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Install pnpm Dependencies
description: Installs pnpm Dependencies
runs:
using: 'composite'
steps:
# Install pnpm dependencies (see https://github.com/pnpm/action-setup?tab=readme-ov-file#use-cache-to-reduce-installation-time)
- uses: pnpm/action-setup@v4
name: Install pnpm
with:
run_install: false

- name: Install Node.js
uses: actions/setup-node@v4
with:
cache: 'pnpm'
node-version-file: '.nvmrc'
registry-url: https://registry.npmjs.org

- name: Get pnpm store directory
shell: bash
run: |
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV

- uses: actions/cache@v4
name: Setup pnpm cache
with:
path: ${{ env.STORE_PATH }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-

- name: Install dependencies
run: pnpm install
shell: bash
# Note: You don't need to run pnpm store prune at the end; post-action has already taken care of that.

83 changes: 27 additions & 56 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,55 +6,36 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Use Node.js 18.x
uses: actions/setup-node@v4
with:
node-version: 18.x
- run: yarn
- uses: ./.github/commands/dependencies/install_deps
- name: Build the typescript code
run: yarn build
run: pnpm build
- uses: actions/upload-artifact@v4
with:
name: build artifacts
path: build/
path: dist/

run-mocha:
run-vitest:
runs-on: ubuntu-latest
strategy:
matrix:
os: [ubuntu-latest]
node-version: [18.x]
steps:
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
- run: yarn
- run: yarn test
- uses: ./.github/commands/dependencies/install_deps
- run: pnpm test

run-depcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Use Node.js 18.x
uses: actions/setup-node@v4
with:
node-version: 18.x
- run: yarn
- run: yarn depcheck
- uses: ./.github/commands/dependencies/install_deps
- run: pnpm depcheck

generate-json-schema:
generate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Use Node.js 18.x
uses: actions/setup-node@v4
with:
node-version: 18.x
- run: yarn
- run: yarn script:transcend-json-schema
- run: yarn script:pathfinder-json-schema
- uses: ./.github/commands/dependencies/install_deps
- run: pnpm script:transcend-json-schema
- run: pnpm script:pathfinder-json-schema
- run: pnpm script:build-readme-docs
- name: Ensure no files changed
run: git diff --quiet HEAD || (git diff --raw && echo "Detected untracked changes in git after running command '${{ matrix.command }}'" && exit 1)

Expand All @@ -65,11 +46,7 @@ jobs:
with:
fetch-depth: 100 # need the history to do a changed files check below (source, origin)
- uses: actions/setup-python@v4
- name: Use Node.js 18.x
uses: actions/setup-node@v4
with:
node-version: 18.x
- run: yarn
- uses: ./.github/commands/dependencies/install_deps
- uses: pre-commit/action@v3.0.0
with:
extra_args: --source ${{ github.event.pull_request.base.sha || 'HEAD~1' }} --origin ${{ github.event.pull_request.head.sha || 'HEAD' }}
Expand All @@ -78,44 +55,38 @@ jobs:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
needs:
- run-mocha
- run-vitest
- run-depcheck
- build-and-upload-artifacts
- run-pre-commits
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '18.x'
- run: yarn
- uses: ./.github/commands/dependencies/install_deps
- name: Configure NPM authentication
run: |
yarn config set npmAlwaysAuth true
yarn config set npmAuthToken ${{ secrets.NPM_TOKEN }}
- name: Publish to yarn/npm
run: yarn npm publish
npm config set npmAlwaysAuth true
npm config set npmAuthToken ${{ secrets.NPM_TOKEN }}
- name: Publish to pnpm/npm
run: npm publish

build-to-github-packages:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
needs:
- run-mocha
- run-vitest
- run-depcheck
- build-and-upload-artifacts
- run-pre-commits
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '18.x'
- run: yarn
- uses: ./.github/commands/dependencies/install_deps
- name: Configure Github Packages authentication
run: |
yarn config set npmAlwaysAuth true
yarn config set npmRegistryServer https://npm.pkg.github.com
yarn config set npmPublishRegistry https://npm.pkg.github.com
yarn config set npmAuthToken ${{ secrets.GITHUB_TOKEN }}
npm config set npmAlwaysAuth true
npm config set npmRegistryServer https://npm.pkg.github.com
npm config set npmPublishRegistry https://npm.pkg.github.com
npm config set npmAuthToken ${{ secrets.GITHUB_TOKEN }}
- name: Publish to Github packages
run: yarn npm publish
run: npm publish
11 changes: 2 additions & 9 deletions .github/workflows/update_transcend_packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,10 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Use Node.js 18.x
uses: actions/setup-node@v4
with:
node-version: 18.x
- run: yarn
- uses: ./.github/commands/dependencies/install_deps

# Install
- run: yarn up @transcend-io/internationalization
- run: yarn up @transcend-io/privacy-types
- run: yarn up @transcend-io/persisted-state
- run: yarn up @transcend-io/type-utils
- run: pnpm up "@transcend-io/*" --latest

- name: Create Pull Request
id: cpr
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/version_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ jobs:
echo "$ALL_CHANGED_FILES";
echo ""
echo "Please bump the version in package.json using one of:";
echo " - yarn version patch (for bug fixes)";
echo " - yarn version minor (for new features)";
echo " - yarn version major (for breaking changes)";
echo " - npm version patch (for bug fixes)";
echo " - npm version minor (for new features)";
echo " - npm version major (for breaking changes)";
exit 1;
fi
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ cron-identifiers.csv
manual-enrichment-identifiers.csv
preference-management-upload-receipts.json
build/
dist/
privacy-request-upload-receipts/
dsr-files/

Expand Down
15 changes: 0 additions & 15 deletions .mocharc.js

This file was deleted.

Loading
Loading