Typegen Usage which worked with 9.14.2 doesn't work with 10.1.2

[rgoulter]

• I'm submitting a ...

  • Bug report
  • Feature request
  • Support request
  • Other

• What is the current behavior and expected behavior?

Code which worked with @polkadot/api v9.14.2 doesn't work with v10.1.2. -- Likely same as #5524 (comment)

Some clarification on what changes are needed would be useful.

Example code (& the commands used) can be found here: https://github.com/rgoulter/polkadot-js-typegen-efinity -- It's a pretty bare-bones usage of typegen, to generate the code, and includes a small example TypeScript program which uses the generated types.

The latest commit there updates the @polkadot/typegen to 10.1.2 (and in example/package.json, its @polkadot/api to 10.1.2).

I'm having trouble figuring out what changes I need to make to get it to work with 10.1.2.

What goes wrong:

When it runs polkadot-types-from-defs, it has problems loading the definitions.ts:

> ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package @efinity/api-augment/interfaces --input ./src/interfaces --endpoint ./metadata.json

ERROR: Unable to load user definitions: Unknown file extension ".ts" for /home/rgoulter/scratch/efinity-api-augment/src/interfaces/definitions.ts

When running polkadot-types-from-chain, some .ts code is generated, but the generated code then leads to errors when trying to build it.

> ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --endpoint ./metadata.json --output ./src/interfaces

Generating from metadata, 254,086 bytes
ERROR: No lookup definitions found: Unknown file extension ".ts" for /home/rgoulter/scratch/efinity-api-augment/src/interfaces/lookup.ts
2023-03-15 14:41:52        REGISTRY: Unknown signed extensions CheckFuelTank found, treating them as no-effect
/home/rgoulter/scratch/efinity-api-augment/src/interfaces/augment-api-consts.ts
        Generating
        Writing

...

Sample errors when trying to run tsc in example:

node_modules/@efinity/api-augment/src/interfaces/augment-api-consts.ts:166:17 - error TS2304: Cannot find name 'FrameSupportPalletId'.

166       palletId: FrameSupportPalletId & AugmentedConst<ApiType>;
                    ~~~~~~~~~~~~~~~~~~~~

...

Found 457 errors in 4 files.

Errors  Files
    11  node_modules/@efinity/api-augment/src/interfaces/augment-api-consts.ts:166
   142  node_modules/@efinity/api-augment/src/interfaces/augment-api-events.ts:43
    86  node_modules/@efinity/api-augment/src/interfaces/augment-api-query.ts:31
   218  node_modules/@efinity/api-augment/src/interfaces/augment-api-tx.ts:23

Problems running with ts-node-esm, too:

If instead of ts-node, I try ts-node-esm, polkadot-types-from-defs abruptly exits without an error message:

> ts-node-esm --skip-project node_modules/.bin/polkadot-types-from-defs --package @efinity/api-augment/interfaces --input ./src/interfaces --endpoint ./metadata.json

/home/rgoulter/scratch/efinity-api-augment/src/interfaces/types.ts
        Generating
        Extracting interfaces for default

If instead of ts-node, I try ts-node-esm, polkadot-types-from-chain is still unable to lookup the definitions:

> @efinity/[email protected] generate:meta
> ts-node-esm --skip-project node_modules/.bin/polkadot-types-from-chain --endpoint ./metadata.json --output ./src/interfaces

Generating from metadata, 254,086 bytes
ERROR: No lookup definitions found: Cannot read properties of undefined (reading 'default')
2023-03-15 14:45:04        REGISTRY: Unknown signed extensions CheckFuelTank found, treating them as no-effect
/
...

• What is the motivation for changing the behavior?

n/a

• Please tell us about your environment:

Linux.

$ node --version
v18.12.1

$ npx tsc --version
Version 4.9.4

[jacogr]

Ok, I found pages and pages of threads on google with regards to Unknown file extension ".ts" and ts-node. So at least we are not the first...

The quick fix seems to be (which mostly works for me on your reproduction) -

• Add "type": "module" to your package.json
• Add --esm to the ts-node flags, i.e ts-node --esm --skip-project node_modules/.bin/...

(In your specific example there is ERROR: No lookup definitions found: exports is not defined in ES module scope - which is due to the empty exports {} in defintions.ts)

Doing that makes it generate everything.

What is weird here is that you would expect, when running as ESM, you would not need --esm flags and change the module type for the generator part. So basically it would seem that 10.x is more breaking that planned as a major. The behavior seems to be that ts-node internally compiled to cjs without the --esm flag and then fails on the dynamic import(...) where it cannot resolve. So it is a bit of damned-if-you-do and damned-if-you-don't based on my quick reading of things.

(Swapping the ts-node --skip-project node_modules/.bin/... command for node --loader ts-node/esm node_modules/.bin/... also works, but it also needs the "type": "module" definition it would seem - otherwise it just stops without a trace after the first generation)

... Addendum: the above doesn't quite work, it still missing loading definitions & lookup. The exports is not defined in ES module scope is actually thrown by ts-node. So it "moves" in the right direction, but not "solved".

[rgoulter]

... Doing that makes it generate everything. ...

Part of the trouble is that, if polkadot-types-from-chain shows some ERROR related to lookup definitions, the generated TS code isn't usable.

e.g. when there's some error related to lookup definitions, the generated augment-api-consts.ts is:

import '@polkadot/api-base/types/consts';

import type { ApiTypes, AugmentedConst } from '@polkadot/api-base/types';
import type { Bytes, Option, U8aFixed, bool, u128, u16, u32, u64, u8 } from '@polkadot/types-codec';
import type { Codec } from '@polkadot/types-codec/types';
import type { AccountId32, Perbill, Permill } from '@polkadot/types/interfaces/runtime';

export type __AugmentedConst<ApiType extends ApiTypes> = AugmentedConst<ApiType>;

declare module '@polkadot/api-base/types/consts' {

whereas it 'should' look more like:

import '@polkadot/api-base/types/consts';

import type { ApiTypes, AugmentedConst } from '@polkadot/api-base/types';
import type { Bytes, Option, U8aFixed, bool, u128, u16, u32, u64, u8 } from '@polkadot/types-codec';
import type { Codec } from '@polkadot/types-codec/types';
import type { AccountId32, Perbill, Permill } from '@polkadot/types/interfaces/runtime';
import type {
    ...
    FrameSupportPalletId,
    ...
} from '@polkadot/types/lookup';

export type __AugmentedConst<ApiType extends ApiTypes> = AugmentedConst<ApiType>;

declare module '@polkadot/api-base/types/consts' {

And because of the missing import { ... } from @polkadot/types/lookup, the code which relies on it breaks.

---

Rather, in the example, the problem isn't so much generating the code, so much as polkadot-types-from-chain being able to read the lookup types properly.

(In my example, I can ignore polkadot-types-from-defs being unable to load definitions.ts & just keep running that with ts-node; but it does seem untidy to ignore the ERROR).

I can kinda-sorta work around it by directly modifying the node_modules/@polkadot/typegen/fromChain.js, and running with ts-node-esm (which is the same as adding --esm):

    try {
        const lookPath = assertFile(path.join(outputPath, 'lookup.ts'));
        const lookCont = await import(lookPath);
        customLookupDefinitions = {
            rpc: {},
            types: lookCont.module.default
        };
    }

to have lookCont.default.default instead of lookCont.default.default; but, I can't find out "why" the result of await import(...) would have .module or .default.

[jacogr]

.module should not be there (PR incoming).

.default will be there for .e.g lookup where it does an export default ...

[jacogr]

10.1.3 patch release is out that fixes the (completely incorrect), .module. specifiers. That has been tested against the above repo (with module & loader flags as per the PR just above that introduced the changes)

[montogeek]

I got this error:

With a definitions.ts like:

export { default as identity } from './identity/definitions';

Logging usersDefs shows:

[Module: null prototype] { default: { identity: { types: [Object] } } }

So I changed it back to userDefs = defCont.default;

It works for a bit, but then at the end throws this error:

disconnected, code: '1005' reason: ''

Logging the error, a property says that wasClean: true, but the good news is that I get the expected types.

[montogeek]

Another new problem in 10.1.2 and 10.1.3 is that types.ts is now exporting from a JS file:
In 9.10.5

export * from './identity/types';

In 10.1.2 and 10.1.3

export * from './identity/types.js';

This breaks the build, I am using the --esm option, will try with a different loader.
Update: Same result.

[jacogr]

That should really not break the build at all. It work in both TS moduleResultion: node and moduleResolution: nodenext (which is why it has been added)

EDIT: I do get an endless stream of errors in the above repo if I remove the tsonfig.json I added with { "compilerOptions": { "module": "esnext", "target": "esnext" } }

[jacogr]

@montogeek On your export { default as identity } from './identity/definitions'; issue - I actually added a couple of those in my tests as well, there is no default export for me on those. (Logging the keys of the object should show ['identity', ...])

[montogeek]

What Node version are you using?

[jacogr]

v18.14.0

[jacogr]

Something is weird here with ts-node and the options, so not sure what to fiddle with.

See the 2 commands and the outputs -

> ts-node --esm --skipProject ./node_modules/@polkadot/typegen/scripts/polkadot-types-from-defs.mjs --package @efinity/api-augment/interfaces --input ./src/interfaces --endpoint ./metadata.json

ERROR: Unable to load user definitions: exports is not defined in ES module scope
<somewhere>/polkadot-js-typegen-efinity/src/interfaces/types.ts
	Generating
	Writing

<somewhere>/polkadot-js-typegen-efinity/src/interfaces/augment-types.ts
	Generating
	Writing

2023-03-15 14:42:58        REGISTRY: Unknown signed extensions CheckFuelTank found, treating them as no-effect
<somewhere>/polkadot-js-typegen-efinity/src/interfaces/lookup.ts
	Generating
	Writing

And then via loader (no exports message as above, actual imported definition is generated) -

> node --loader ts-node/esm ./node_modules/@polkadot/typegen/scripts/polkadot-types-from-defs.mjs --package @efinity/api-augment/interfaces --input ./src/interfaces --endpoint ./metadata.json

(node:27028) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
<somewhere>/polkadot-js-typegen-efinity/src/interfaces/types.ts
	Generating
	Extracting interfaces for something
	Writing

<somewhere>/polkadot-js-typegen-efinity/src/interfaces/augment-types.ts
	Generating
	Writing

2023-03-15 14:43:31        REGISTRY: Unknown signed extensions CheckFuelTank found, treating them as no-effect
<somewhere>/polkadot-js-typegen-efinity/src/interfaces/lookup.ts
	Generating
	Writing

Have no clue what the difference would be between ts-node --esm ... and node --loader ts-node/esm

[rgoulter]

exports is not defined in ES module scope

This output seems to indicate it's reading CommonJS code, when it's expecting to read an ESM module.

And tsc without any compiler options will emit CommonJS code.
And running ts-node with --skip-project will ignore any tsconfig.json.

rgoulter/polkadot-js-typegen-efinity@6e193452fc180efdc8d275c341839e4bdd70c4d1HH is enough to be able to run the commands in the README, using @polkadot/typegen 10.1.3.

I'd be happy for the issue to be closed; or maybe it can catch other problems people are having with changes since 9.14.2.

[timbrinded]

EDIT: The below was fixed by adding the magic line: "experimentalSpecifierResolution": "node" to my ts-node config in tsconfig.json

e.g.

  "ts-node": {
    "experimentalSpecifierResolution": "node"
  }

---

I've moved our typegen project over to ESM and i've run into the same issue as reported here (with no workaround found yet).

exports is not defined in ES module scope which means that not getting any of the type imports automatically added to the interfaces. (just on this branch, master is non ESM and generates defs as expected)

https://github.com/PureStake/moonbeam/tree/typescript-api-2201/typescript-api

❯ npm run generate:defs

> @moonbeam-network/[email protected] generate:defs
> npm run generate:defs:moonbase && npm run generate:defs:moonriver && npm run generate:defs:moonbeam


> @moonbeam-network/[email protected] generate:defs:moonbase
> ts-node --esm --skip-project node_modules/.bin/polkadot-types-from-defs --package @moonbeam/api-augment/moonbase/interfaces --input ./src/moonbase/interfaces --endpoint ./metadata-moonbase.json

ERROR: Unable to load user definitions: exports is not defined in ES module scope
/home/.../workspace/moonbeam/moonbeam/typescript-api/src/moonbase/interfaces/types.ts
	Generating
	Writing

/home/.../workspace/moonbeam/moonbeam/typescript-api/src/moonbase/interfaces/augment-types.ts
	Generating
	Writing

/home/.../workspace/moonbeam/moonbeam/typescript-api/src/moonbase/interfaces/lookup.ts
	Generating
	Writing

[jacogr]

I'm not convinced that ts-node --esm actually works as expected for this. My outputs in #5540 (comment) above matches what you are seeing, however this...

node --loader ts-node/esm node_modules/.bin/polkadot-types-from-defs --package @moonbeam/...

did the trick above (for me).

EDIT: ... or the EDIT in the comment preceding mine, experimentalSpecifierResolution to the ts-node config

[timbrinded]

@jacogr the reason the above probably works for you is that you must have set --experimental-specifier-resolution=node option on node somewhere.

Or have it as an environment variable i.e.NODE_OPTIONS=--experimental-specifier-resolution=node

In any case, glad this is sorted!

[gdethier]

@jacogr By applying the option 1 described here, I am indeed able to generate definitions. However, chain types are not generated (polkadot-types-from-chain), all I get is this:

(node:41574) ExperimentalWarning: Custom ESM Loaders is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
connected
Generating from metadata, 105,931 bytes
disconnected, code: '1005' reason: ''

After a couple of hours debugging, I figured out that the process is actually killed because of the 1005 event code (see this line) and type generation is simply not executed.

A workaround is to comment out the line. It does not solve the unexpected event code issue, but at least types are properly generated.

[jacogr]

Indeed. Looking at the above code link promise is actually not properly tracked/handled -

• it does resolve when a response is received (all good)
• it does not reject onerror, rather exits (same with close)

So basically that should be adjusted to only resolve/reject once (onmessage/onerror/onclose) aka when we have a response, we don't care if things go wrong with the connection so should not perform any exit inside there. Said differently whichever of there we find once should determine the Promise result... once.

(It seems it was expecting all RPCs nodes to always behave and that assumption doesn't always hold)

[MrishoLukamba]

Am still getting errors in lookup.ts as it is not generated as ES module I think and cannot export the types to other modules hence breaking the code.
@jacogr

[0xverin]

In version 10.1.2, the export method has changed：
export * from './identity/types.js';

in version 9.14.2:
export * from './identity/types';

which disrupts the structure. I have tried all the mentioned methods above, but none of them successfully resolve the issue.

litentry/heima#1703

[devdanco]

I have created PR for docs on how we generate these types. I hope that it will be beneficial to someone.

[polkadot-js-bot]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.