Skip to content

Commit

Permalink
Merge pull request #872 from o1-labs/feature/workers-when-needed-new
Browse files Browse the repository at this point in the history
Start workers only when needed
  • Loading branch information
mitschabaude committed May 2, 2023
2 parents 10acfff + 030fd5e commit 2f5c661
Show file tree
Hide file tree
Showing 46 changed files with 718 additions and 1,067 deletions.
1 change: 1 addition & 0 deletions .github/workflows/doc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ jobs:

- name: run typedoc
run: |
git submodule update --init --recursive
npm ci
npx typedoc --tsconfig tsconfig.node.json src/index.ts
Expand Down
39 changes: 31 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,29 +15,52 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
_Security_ in case of vulnerabilities.
-->

## [Unreleased](https://github.com/o1-labs/snarkyjs/compare/97e393ed...HEAD)
## [Unreleased](https://github.com/o1-labs/snarkyjs/compare/bcc666f2...HEAD)

> No unreleased changes yet
## [0.10.0](https://github.com/o1-labs/snarkyjs/compare/97e393ed...bcc666f2)

### Breaking Changes

- All references to `actionsHash` are renamed to `actionState` to better mirror what is used in Mina protocol APIs https://github.com/o1-labs/snarkyjs/pull/833
- This change affects function parameters and returned object keys throughout the API
- No longer make `MayUseToken.InheritFromParent` the default `mayUseToken` value on the caller if one zkApp method calls another one; this removes the need to manually override `mayUseToken` in several known cases https://github.com/o1-labs/snarkyjs/pull/863
- Causes a breaking change to the verification key of deployed contracts that use zkApp composability

### Added

- `this.state.getAndAssertEquals()` as a shortcut for `let x = this.state.get(); this.state.assertEquals(x);` https://github.com/o1-labs/snarkyjs/pull/863
- also added `.getAndAssertEquals()` on `this.account` and `this.network` fields
- Support for fallback endpoints when making network requests, allowing users to provide an array of endpoints for GraphQL network requests. https://github.com/o1-labs/snarkyjs/pull/871
- Endpoints are checked for validity, and a "waterfall" approach is used to loop through fallback endpoints if needed.
- Endpoints are fetched two at a time, and the result returned from the faster response
- `reducer.forEach(actions, ...)` as a shortcut for `reducer.reduce()` when you don't need a `state` https://github.com/o1-labs/snarkyjs/pull/863
- New export `TokenId` which supersedes `Token.Id`; `TokenId.deriveId()` replaces `Token.Id.getId()` https://github.com/o1-labs/snarkyjs/pull/863
- Add `Permissions.allImpossible()` for the set of permissions where nothing is allowed (more convenient than `Permissions.default()` when you want to make most actions impossible) https://github.com/o1-labs/snarkyjs/pull/863

### Changes

- **Massive improvement of memory consumption**, thanks to a refactor of SnarkyJS' worker usage https://github.com/o1-labs/snarkyjs/pull/872
- Memory reduced by up to 10x; see [the PR](https://github.com/o1-labs/snarkyjs/pull/872) for details
- Side effect: `Circuit` API becomes async, for example `MyCircuit.prove(...)` becomes `await MyCircuit.prove(...)`
- Token APIs `this.token.{send,burn,mint}()` now accept an `AccountUpdate` or `SmartContract` as from / to input https://github.com/o1-labs/snarkyjs/pull/863
- Improve `Transaction.toPretty()` output by adding account update labels in most methods that create account updates https://github.com/o1-labs/snarkyjs/pull/863

### Deprecated

- Deprecate both `shutdown()` and `await isReady`, which are no longer needed https://github.com/o1-labs/snarkyjs/pull/872

### Changed

- Raises the limit of actions/events per transaction from 16 to 100, providing users with the ability to submit a larger number of events/actions in a single transaction. https://github.com/o1-labs/snarkyjs/pull/883.

### Fixed

- `SmartContract.deploy()` throws an error when no verification key is found https://github.com/o1-labs/snarkyjs/pull/885
- `SmartContract.deploy()` now throws an error when no verification key is found https://github.com/o1-labs/snarkyjs/pull/885
- The old, confusing behaviour was to silently not update the verification key (but still update some permissions to "proof", breaking the zkApp)

## [0.9.8](https://github.com/o1-labs/snarkyjs/compare/1a984089...97e393ed)

### Breaking Changes

- Renamed Variable: All references to `actionsHash` have been renamed to `actionState` to better mirror what is used in the Mina protocol implementation. https://github.com/o1-labs/snarkyjs/pull/833
- This change affects variables, function parameters, and object keys throughout the codebase.

### Fixed

- Fix fetching the `access` permission on accounts https://github.com/o1-labs/snarkyjs/pull/851
Expand Down
2 changes: 0 additions & 2 deletions MINA_COMMIT

This file was deleted.

9 changes: 5 additions & 4 deletions jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@ export default {
preset: 'ts-jest/presets/js-with-ts',
testEnvironment: 'node',
extensionsToTreatAsEsm: ['.ts'],
transformIgnorePatterns: ["node_modules/", "dist/node/"],
transformIgnorePatterns: ['node_modules/', 'dist/node/'],
globals: {
'ts-jest': {
useESM: true
}
}
useESM: true,
},
},
testTimeout: 1_000_000,
};
18 changes: 2 additions & 16 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "snarkyjs",
"description": "TypeScript framework for zk-SNARKs and zkApps",
"version": "0.9.8",
"version": "0.10.0",
"license": "Apache-2.0",
"homepage": "https://github.com/o1-labs/snarkyjs/",
"keywords": [
Expand Down Expand Up @@ -46,12 +46,13 @@
"type-check": "tsc --noEmit",
"dev": "npx tsc -p tsconfig.node.json && cp src/snarky.d.ts dist/node/snarky.d.ts",
"make": "make -C ../../.. snarkyjs",
"bindings": "cd ../../.. && ./scripts/update-snarkyjs-bindings.sh && cd src/lib/snarkyjs",
"build": "rimraf ./dist/node && npx tsc -p tsconfig.node.json && cp -r src/snarkyjs-bindings/compiled/node_bindings dist/node/_node_bindings && node src/build/buildNode.js && cp src/snarky.d.ts dist/node/snarky.d.ts",
"build:test": "npx tsc -p tsconfig.test.json && cp src/snarky.d.ts dist/node/snarky.d.ts",
"build:node": "npm run build",
"build:web": "rimraf ./dist/web && node src/build/buildWeb.js",
"build:examples": "rimraf ./dist/examples && npx tsc -p tsconfig.examples.json || exit 0",
"serve:web": "cp src/snarkyjs-bindings/compiled/web_bindings/server.js src/snarkyjs-bindings/compiled/web_bindings/index.html dist/web && node dist/web/server.js",
"serve:web": "cp src/snarkyjs-bindings/compiled/web_bindings/server.js src/snarkyjs-bindings/compiled/web_bindings/index.html src/examples/simple_zkapp.js dist/web && node dist/web/server.js",
"prepublish:web": "NODE_ENV=production node src/build/buildWeb.js",
"prepublish:node": "npm run build && NODE_ENV=production node src/build/buildNode.js",
"prepublish:both": "npm run prepublish:web && npm run prepublish:node",
Expand Down Expand Up @@ -94,7 +95,6 @@
"dependencies": {
"blakejs": "1.2.1",
"detect-gpu": "^5.0.5",
"env": "^0.0.2",
"isomorphic-fetch": "^3.0.0",
"js-sha256": "^0.9.0",
"reflect-metadata": "^0.1.13",
Expand Down
5 changes: 2 additions & 3 deletions src/build/buildNode.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import fs from 'node:fs/promises';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { exec } from 'node:child_process';
Expand Down Expand Up @@ -60,12 +59,12 @@ function makeNodeModulesExternal() {
}

function makeJsooExternal() {
let isJsoo = /bc.cjs$/;
let isJsoo = /(bc.cjs|plonk_wasm.cjs)$/;
return {
name: 'plugin-external',
setup(build) {
build.onResolve({ filter: isJsoo }, ({ path }) => ({
path: path.replace('../', './'),
path: path.replace('../../', './').replace('../', './'),
external: true,
}));
},
Expand Down
23 changes: 11 additions & 12 deletions src/build/buildWeb.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import glob from 'glob';
export { buildWeb };

const entry = './src/index.ts';
const target = 'es2021';
const target = 'es2022';

let nodePath = path.resolve(process.argv[1]);
let modulePath = path.resolve(fileURLToPath(import.meta.url));
Expand Down Expand Up @@ -54,7 +54,9 @@ async function buildWeb({ production }) {
'./src/snarkyjs-bindings/compiled/web_bindings/':
'./dist/web/web_bindings/',
'./src/snarky.d.ts': './dist/web/snarky.d.ts',
'./src/snarky/wrapper.web.js': './dist/web/snarky/wrapper.js',
'./src/snarkyjs-bindings/js/wrapper.web.js':
'./dist/web/snarkyjs-bindings/js/wrapper.js',
'./src/snarkyjs-bindings/js/web/': './dist/web/snarkyjs-bindings/js/web/',
});

await Promise.all([tscPromise, copyPromise]);
Expand Down Expand Up @@ -128,14 +130,10 @@ function execPromise(cmd) {
function rewriteWasmBindings(src) {
src = src
.replace("new URL('plonk_wasm_bg.wasm', import.meta.url)", 'wasmCode')
.replace('import.meta.url', '"/"')
.replace(
"import { startWorkers } from './snippets/wasm-bindgen-rayon-7afa899f36665473/src/workerHelpers.no-bundler.js';",
`import wasmCode from './plonk_wasm_bg.wasm';
let startWorkers;
`
);
return src;
.replace('import.meta.url', '"/"');
return `import wasmCode from './plonk_wasm_bg.wasm';
let startWorkers, terminateWorkers;
${src}`;
}
function rewriteBundledWasmBindings(src) {
let i = src.indexOf('export {');
Expand All @@ -147,12 +145,13 @@ function rewriteBundledWasmBindings(src) {
src = src.slice(0, i) + exportSlice;

src = src.replace('var startWorkers;\n', '');
return `import {startWorkers} from './workerHelpers.js'
src = src.replace('var terminateWorkers;\n', '');
return `import { startWorkers, terminateWorkers } from '../snarkyjs-bindings/js/web/worker-helpers.js'
export {plonkWasm as default};
function plonkWasm() {
${src}
}
plonkWasm.deps = [startWorkers]`;
plonkWasm.deps = [startWorkers, terminateWorkers]`;
}

function wasmPlugin() {
Expand Down
5 changes: 4 additions & 1 deletion src/build/extractJsooMethods.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ let classNames = [
}

let specJson = JSON.stringify(classList, null, 2);
await fs.writeFile('src/snarky/snarky-class-spec.json', specJson + '\n');
await fs.writeFile(
'src/snarkyjs-bindings/js/snarky-class-spec.json',
specJson + '\n'
);
// this could be useful for the browser version:
// await fs.writeFile('src/snarky-class-spec.js', 'export default ' + specJson);

Expand Down
27 changes: 27 additions & 0 deletions src/build/fix-wasm-bindings-node.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import fs from 'node:fs/promises';

const file = process.argv[2];

let src = await fs.readFile(file, 'utf8');

src = src.replace(
"imports['env'] = require('env');",
`
let { isMainThread, workerData } = require('worker_threads');
let env = {};
if (isMainThread) {
env.memory = new WebAssembly.Memory({
initial: 20,
maximum: 65536,
shared: true,
});
} else {
env.memory = workerData.memory;
}
imports['env'] = env;
`
);

await fs.writeFile(file, src, 'utf8');
5 changes: 0 additions & 5 deletions src/build/run.js
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,8 @@ if (!bundle) {
let module = await import(absPath);
if (main) await module.main();
if (runDefault) await module.default();
let { shutdown } = await import('../../dist/node/index.js');
shutdown();
} else {
let { isReady, shutdown } = await import('../../dist/node/index.js');
await isReady;
let module = await buildAndImport(filePath, { keepFile: !!keep });
if (main) await module.main();
if (runDefault) await module.default();
shutdown();
}
72 changes: 72 additions & 0 deletions src/examples/commonjs.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
/**
* Tests that snarkyjs can be imported and used from commonJS files
*/
let {
Field,
State,
PrivateKey,
SmartContract,
Mina,
AccountUpdate,
declareState,
declareMethods,
} = require('snarkyjs');

class SimpleZkapp extends SmartContract {
constructor(address) {
super(address);
this.x = State();
}

events = { update: Field };

init() {
super.init();
this.x.set(initialState);
}

update(y) {
this.emitEvent('update', y);
this.emitEvent('update', y);
this.account.balance.assertEquals(this.account.balance.get());
let x = this.x.get();
this.x.assertEquals(x);
this.x.set(x.add(y));
}
}
declareState(SimpleZkapp, { x: Field });
declareMethods(SimpleZkapp, { update: [Field] });

let Local = Mina.LocalBlockchain();
Mina.setActiveInstance(Local);

let feePayerKey = Local.testAccounts[0].privateKey;
let feePayer = Local.testAccounts[0].publicKey;

let zkappKey = PrivateKey.random();
let zkappAddress = zkappKey.toPublicKey();

let initialState = Field(1);
let zkapp = new SimpleZkapp(zkappAddress);

main();

async function main() {
console.log('compile');
await SimpleZkapp.compile();

console.log('deploy');
let tx = await Mina.transaction(feePayer, () => {
AccountUpdate.fundNewAccount(feePayer);
zkapp.deploy();
});
await tx.sign([feePayerKey, zkappKey]).send();

console.log('initial state: ' + zkapp.x.get());

console.log('update');
tx = await Mina.transaction(feePayer, () => zkapp.update(Field(3)));
await tx.prove();
await tx.sign([feePayerKey]).send();
console.log('final state: ' + zkapp.x.get());
}
9 changes: 5 additions & 4 deletions src/examples/ex00_preimage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,16 @@ class Main extends Circuit {
await isReady;

console.log('generating keypair...');
const kp = Main.generateKeypair();
const kp = await Main.generateKeypair();

const preimage = Field(1);
const hash = Poseidon.hash([preimage]);

console.log('prove...');
const pi = Main.prove([preimage], [hash], kp);
console.log('proof', pi);
const pi = await Main.prove([preimage], [hash], kp);

console.log('verify...');
let ok = Main.verify([hash], kp.verificationKey(), pi);
let ok = await Main.verify([hash], kp.verificationKey(), pi);
console.log('ok?', ok);

if (!ok) throw Error('verification failed');
Loading

0 comments on commit 2f5c661

Please sign in to comment.