Discussion: New “ESM by default” mode

@LiviaMedeiros

Building off of #49295 (comment), we’re considering a new mode where all of the current places where Node defaults to CommonJS would instead default to ESM. I think this should be enabled by flag, and once it ships we can potentially ship a separate binary where it’s enabled by default; and/or we can someday make the new mode Node’s default in a semver-major change.

The use cases solved by such a mode are (and I’ll edit this comment to update this list as people think of more):

I want to write ESM syntax by default without needing to opt into it, such as via package.json file or file extension or --input-type=module.
I want to write what are typically called shell scripts in ESM JavaScript, where I can have a single extensionless file anywhere on my disk that uses ESM syntax and it doesn’t need a nearby package.json file or a symlink or wrapper file in order to run as ESM.
I want consistency in how the CLI handles references to files; currently --import accepts URL strings but the main entry point must be a path string (and therefore can’t be a data: URL or https: URL, such as to work with --experimental-network-imports).
If this mode becomes enabled by default in a future version of Node, when using that version of Node I want to be able to start a new project and write ESM syntax without needing to take any steps to opt into enabling ESM support.

To handle these use cases, the changes that this mode should make are (and likewise I’ll edit this to reflect consensus):

When a .js file is outside of any defined package.json scope, because there are no package.json files in its folder or any folders above it, it will be treated as ESM JavaScript.
When a .js file is in a package scope where its nearest parent package.json lacks a "type" field, it will be interpreted per the conditions we establish in Discussion: In an ESM-first mode, how should a package.json file with no type field be handled? #49494: when the package scope is under a folder named node_modules, the file will be treated as CommonJS; otherwise it will be treated as ESM.
When an extensionless file is outside of any defined package.json scope, it can be run as an entry point or imported per the conditions we establish in Discussion: In an ESM-first mode, how should extensionless entry points be handled? #49431: if the file begins with the Wasm magic bytes, it will be evaluated as Wasm, and likewise for any other future file types with defined headers that don’t conflict with JavaScript; otherwise it will be evaluated as ESM JavaScript. (Still subject to --experimental-wasm-modules until that stabilizes.)
Input from STDIN or --eval would be treated as ESM unless --input-type=commonjs is passed. Or in other words, the default value of --input-type would flip from commonjs to module.
The CLI will parse the main entry point as an URL string, similar to how the value of --import is parsed. process.argv[1] will not be parsed by the CommonJS loader.

This new flag will be named per the discussion in #49541.

Prior art: #32394, #49295, #49407

@LiviaMedeiros @nodejs/loaders @nodejs/wasi @nodejs/tsc

As discussed in #49295 (comment), there are many potential solutions, the simplest of which is to just crawl through subfolders adding "type": "commonjs" to package.json files that lack it. I don’t think this needs to be a worry or something we need to deal with at this point.

We are not in agreement. The ecosystem concern is something that needs to be at the center of this design. Changing the default of how a module from node_modules is evaluated is so massively breaking that should be off the table. A node.js that cannot run most of the registry is close to useless.
The way this is outlined will create python2 / python3 break, which we should be avoiding at all cost.

Note that I'm not opposed of changing the default when a package.json is not available. I think it would be a significant improvement for shell scripts and the like.. I wish I had it in quite a few occasions. Also: "type": "module" should be the default on npm init at some point in the close future.

We are not in agreement. The ecosystem concern is something that needs to be at the center of this design. Changing the default of how a module from node_modules is evaluated is so massively breaking that should be off the table. A node.js that cannot run most of the registry is close to useless.
The way this is outlined will create python2 / python3 break, which we should be avoiding at all cost.

A new flag will not cause a break. Full stop. There is no breaking change proposed here.

The proposed behavior under the new flag will require an extra patch step until package managers catch up and insert the missing type field during package installation. This is not a big change; users can do it themselves via something like npm install && npx add-missing-type-fields. Calling this a Python 2/3 break is extreme hyperbole. I expect that package managers will begin doing it automatically once we ship the flag and they can test against this mode; or package managers will do it automatically when run by Node in this new mode, or at the very least offer an option to do so, saving users from needing to run the patch command. This is not a big deal. No package maintainers will need to update anything if they don’t want to. Even package managers don’t need to update, since we can publish an add-missing-type-fields script that users can run via npx to do the crawl and patch. We could even have Node throw an error upon encountering a typeless package.json: Error: ./node_modules/foo/package.json lacks a "type" field. Patch it by running npx add-missing-type-fields.

In #49295 (comment) and following comments we discussed the pros and cons of preserving the “missing type field still means CommonJS” behavior in the new mode. Doing so would mean avoiding the need for this patch step and/or avoiding package managers needing to update, but ultimately we wouldn’t be achieving the central goal of making ESM the default: it still wouldn’t be the default if a lack of type field still means CommonJS. And since the fix is so simple, that someone could probably implement in a Bash one-liner or a dozen lines of JavaScript, I think the patch approach is better as it allows us to get to a full ESM default without caveats. And once package managers catch up there won’t be any work for users to do.

There are other approaches. We could treat node_modules as a special case, where in this new mode, subfolders of node_modules preserve the “missing type = CommonJS” behavior. Another idea is that in this new mode, a missing type could be treated as a fallthrough, and Node continues to search up the tree for a package.json with a type, and the user could put a package.json with "type": "commonjs" in the node_modules root folder. I’m sure there are other variations people can think of. Personally though I think it’s better to have the same behavior on both sides of the mode, and just have the flag flip the default: it’s simpler, easier to reason about, and easier for other tools to follow. Yes it means running a patch script, at least at first. That’s a small price to pay, in my opinion, and short-term.

Also: "type": "module" should be the default on npm init at some point in the close future.

Or at the very least, it should start including "type": "commonjs" today and support npm init --module ASAP for the ESM version. But these things are a bit out of scope. Many things about making ESM the default get easier with a supportive package manager, but I want to try to design this so that such buy-in isn’t required, so that we don’t need to depend on every popular package manager signing on to a particular new design.

A new flag will not cause a break. Full stop. There is no breaking change proposed here.

What's the point of creating a mode of running Node.js that cannot run the majority of the registry?

I expect the .js to become recognized as an ESM by default some day, and this behavior as depicted here is unfeasible due to the massive ecosystem breakage.

The proposed behavior under the new flag will require an extra patch step until package managers catch up and insert the missing type field during package installation. This is not a big change; users can do it themselves via something like npm install && npx add-missing-type-fields. Calling this a Python 2/3 break is extreme hyperbole. I expect that package managers will begin doing it automatically once we ship the flag and they can test against this mode; or package managers will do it automatically when run by Node in this new mode, or at the very least offer an option to do so, saving users from needing to run the patch command. This is not a big deal. No package maintainers will need to update anything if they don’t want to.

I think this developer experience is unfeasible. What will happen when new packages that assumes are running in esm-first will be mixed with commonjs packages? Are users supposed to edit them by hand? This would worsen Node.js developer experience significantly and it is a very big deal, considering that the majority of the registry is (still) commonjs.

Moreover, this new mode will break quite a lot of npx and npm create commands, because individual packages would need to be fixed manually.

The ecosystem is already complex enough, and this new mode will actually worsen the situation. We can avoid completely avoid this friction and having "type": "module" there is by far a minor problem as long as it's the default.

Personally though I think it’s better to have the same behavior on both sides of the mode, and just have the flag flip the default: it’s simpler, easier to reason about, and easier for other tools to follow.

I agree. The default should not be changed actually: it has no friction whatsoever for developers. Let's not break everyone.

It seems like there's some confusions there.

Shebangs

IMHO most of linked prior discussions are a bit mislead.

Shebang does support arguments on its own. It works just fine as #!/usr/bin/node --esm.
EDIT: there is limitation of only one argument, so it can't be used together with any other option
There is a practice of using env(1) in shebangs that basically works as #!$(which node)
env historically has limitation of not being able to pass arguments, which is fixed in modern GNU coreutils and FreeBSD.
Minimalistic distributions that use e.g. busybox may not have split-string feature in their env.

The solution for portable enough shebang without separate binary is to write #!/usr/bin/node --esm, simple as that.

I don't recall any issues with that; even though very same thing always applied to shebangs with --experimental-wasm-modules, --allow-fs-write, or any other option.

Separate binaries

There is a difference between separate binary that is compiler with different source code and/or different compile options, and a copy of the very same binary that has different name.

The first comes with downsides such as increased build time and increased release size.

The second (which is proposed in #49407) comes with restriction that the second binary must have specific filename.

For common real life example of it, one may run find "$(git --exec-path)" -samefile "$(which git)".
If this prints about 137 filepaths, you found it: all the git something commands are apparently separate files referring to same inode, AKA hardlinks.
If it doesn't, ls -l "$(git --exec-path)" will probably show a waste of space with copies.
Symlinks would also work, as long as we don't resolve them (e.g. process.execPath).

The implementation comes with almost no overhead, and node-esm doesn't even have to be included in release build; it's a job for the installer that places node in /usr/bin/.

Breaking change

No, adding new optional command line flag that is disabled by default does not break anything for anyone by itself.
It's just a feature that will instantly benefit anyone who already prefers ESM-first code.

And yes, blindly running everything in this mode with Node.js ecosystem that we have right now is likely to break.

This is not only intentional, this is the intent. We must be able to enable it and see what happens. We must ship it so any developer can enable it and see how it works with their projects. We must ship it so any user can enable it and see what breaks in the packages they use. We must ship it so any maintainer can see which dependencies are breaking.

Only after that, we can decide if it's feasible or not to give everyone time to migrate, then introduce --cjs flag and maybe deprecate the default-cjs behaviour, then give everyone more time to migrate, then maybe change the default to ESM-first.

This is not only intentional, this is the intent. We must be able to enable it and see what happens. We must ship it so any developer can enable it and see how it works with their projects. We must ship it so any user can enable it and see what breaks in the packages they use. We must ship it so any maintainer can see which dependencies are breaking.

Only after that, we can decide if it's feasible or not to give everyone time to migrate, then introduce --cjs flag and maybe deprecate the default-cjs behaviour, then give everyone more time to migrate, then maybe change the default to ESM-first.

This is where we are not in agreement, and any plans that include massive ecosystem breakage should be avoided. I'm strongly -1 on this proposal as currently framed. Moreover, I think any plan involving the discontinuation of commonjs should be avoided.

Shebang does support arguments on its own. It works just fine as #!/usr/bin/node --esm.

To be precise, shebang on Linux supports only a single argument. Anything beyond that needs a modern env.

One thing not mentioned here as well is tooling needing out of band info for flags/separate binaries like this; tsconfig/eslintrc/babelrc etc. would need to account for this flag since it wouldn't be statically analyzable and/or can be varied per entrypoint or in multiple modes for a given directory:

/bin/newer -> #!/usr/bin/env node-esm or node --default-operating-type=esm
/bin/older -> #!/usr/bin/env node

These need not be in shebangs either, these can be sources out of band just by invocation of the executable, package.json#scripts, etc. This tooling problem should probably be considered with them in this discussion as going completely out of sync of tools was a struggle for exports but that was at least statically analyzable.

This is where we are not in agreement, and any plans that include massive ecosystem breakage should be avoided. I'm strongly -1 on this proposal as currently framed. Moreover, I think any plan involving the discontinuation of commonjs should be avoided.

Any suggestions then?

@LiviaMedeiros

@LiviaMedeiros you are demonstrating that passing a single argument works, it would be more interesting to show wether passing a second argument doesn't work.

This is where we are not in agreement, and any plans that include massive ecosystem breakage should be avoided. I'm strongly -1 on this proposal as currently framed. Moreover, I think any plan involving the discontinuation of commonjs should be avoided.

Any suggestions then?

I guess we'd need to find a way to either restrict the effect of this flag to dev-only (a bit like the Buffer constructor deprecation warning is emitted only outside of node_modules folder), or hide it behind a build flag.

Any suggestions then?

I think asking the package managers to add "type": "module" when initializing new projects achieve most of the same results.
As I mentioned earlier, not changing the default "type" in package.json will avoid most if not all of the ecosystem breakage.

@tniessen

@tniessen @aduh95 my bad, shebang indeed interprets parameters single argument with spaces. Thanks for pointing this out!

I guess we'd need to find a way to either restrict the effect of this flag to dev-only (a bit like the Buffer constructor deprecation warning is emitted only outside of node_modules folder), or hide it behind a build flag.

Dev-only might work, I guess we can sacrifice performance for that on the first stage.

I think asking the package managers to add "type": "module" when initializing new projects achieve most of the same results.

I think this part is feasible on its own, e.g. marking existing packages without type as using fallback mechanism that should default to commonjs, and setting deadline after which the default fallback value changes to module and/or "type": "module" gets explicitly set by default.

What about files outside of scope? How can we launch non-symlink /usr/local/bin/something.js or /usr/local/bin/something as ESM?

@mcollina

As I mentioned earlier, not changing the default “type” in package.json will avoid most if not all of the ecosystem breakage.

@mcollina If I’m understanding your full comments correctly, this was the only part that you opposed, right?

I have conflicting thoughts on this myself. On the one hand, the more that the flag breaks, the less adoption it’ll get and the longer the timeline to potentially making it the default. On the other hand, like @LiviaMedeiros mentioned, the more that it breaks behind a flag, the more potential gain: if people try out the changes and like them, that builds momentum for the ecosystem adapting.

So I guess the question is, what’s the benefit of changing how “typeless package.jsons” are treated? The only one I can think of is symmetry between the on and off versions of this flag, which is another way of saying user expectations. I can’t think of any use cases enabled by flipping this default. So sure, we can preserve “no type = CommonJS” if we want, and it makes the flag a bit harder to explain but it avoids a set of other problems, the things I was saying would need a patching script for. And maybe that’s what we should do, I don’t know.

There’s no way this flag won’t land without --experimental- in its name. So I also think that we’re free to land the most ambitious version first, just to see how people react and use it, and I’m sure there will be changes before the flag leaves experimental status (long before it becomes a new Node default). We could even split out the “treat typeless package.json as CommonJS” part into a separate flag, so that the primary flag enables everything else and another experimental flag enables that particular behavior, to see if the typeless change has benefits of its own. I don’t think we should be putting down hard blocks against these types of experimentation.

What will happen when new packages that assumes are running in esm-first will be mixed with commonjs packages?

I was thinking that in the new mode, type would be required in package.json files when they’re present. So anyone publishing a package assuming ESM first would still be forced to add "type": "module", keeping it compatible with the legacy mode.

Though I acknowledge that this breaks the symmetry, so maybe there’s no value in flipping how typeless scopes are treated. It would be an unfortunate “this is always this way for legacy reasons” thing.

One thing not mentioned here as well is tooling needing out of band info for flags/separate binaries like this; tsconfig/eslintrc/babelrc etc. would need to account for this flag since it wouldn’t be statically analyzable

I don’t think this is much of an issue. Most tools that I’m familiar with have moved away from the .babelrc pattern into ones like vite.config.js where the file extension behaves like any other file in that scope. And again, we can ship an experimental flag and see which tools are broken under it, and see what those tools say when they open issues with us; would there be a path for them to get their tool working in the new mode, or not? This is part of the point of the experiment.

@GeoffreyBooth

@GeoffreyBooth they interpret the type of files. I'm not concerned about tsconfig.json being interpreted in a new way but about TS not being able to determine the type of a module without "yet another config option". Even with more config options the entrypoint of a program could alter the type of a file to superimpose the 2 modes of operation.

@mcollina

@mcollina If I’m understanding your full comments correctly, this was the only part that you opposed, right?

Yes. There are a lot of technical details in the other points that needs to be ironed out, but nothing major.

The goal should be that the vast majority of modules should work as-is in this new mode. The ecosystem still downloads very old stuff all the time, with pinned dependencies.

I have conflicting thoughts on this myself. On the one hand, the more that the flag breaks, the less adoption it’ll get and the longer the timeline to potentially making it the default. On the other hand, like @LiviaMedeiros mentioned, the more that it breaks behind a flag, the more potential gain: if people try out the changes and like them, that builds momentum for the ecosystem adapting.

What is the gain exactly in this case? Changing the default type in package.json will generate a lot of pain. This is the part that I do not understand, why should we break people modules?

So I guess the question is, what’s the benefit of changing how “typeless package.jsons” are treated? The only one I can think of is symmetry between the on and off versions of this flag, which is another way of saying user expectations. I can’t think of any use cases enabled by flipping this default. So sure, we can preserve “no type = CommonJS” if we want, and it makes the flag a bit harder to explain but it avoids a set of other problems, the things I was saying would need a patching script for. And maybe that’s what we should do, I don’t know.

Essentially I'm prioritizing stability of the ecosystem. I don't think it makes the flag harder to explain, as doing that keep most packages running in the same way.

@mcollina

Changing the default type in package.json will generate a lot of pain.

I don’t think it would necessarily generate a lot of pain. Here’s a script that goes through and fixes missing "type" fields in all the package.json files in the current folder tree:

import { cwd } from 'node:process'
import { opendir, readFile, writeFile } from 'node:fs/promises'

let patched = 0

for await (const entry of await opendir(cwd(), { recursive: true })) {
  if (entry.isFile() && entry.name === 'package.json') {
    const packageJson = JSON.parse(await readFile(entry.path, 'utf8'))
    if (!('type' in packageJson)) {
      packageJson.type = 'commonjs'
      await writeFile(entry.path, JSON.stringify(packageJson, null, 2))
      console.log('Patched', entry.path)
      patched++
    }
  }
}

console.log(`Added "type" field to ${patched} package.json files that lacked it.`)

We could publish this to npm so that users could run it as an npx command like npx patch-typeless-packages and voilà, installing packages in the new ESM-first mode becomes npm install && npx patch-typeless-packages. A bit clunky? Yes. Python 2/3? No.

Alternatively I can see package managers taking the position that they disapprove of the tactic of patching packages after installation; that what users run should be what authors published, verbatim. And so no package manager builds in support for a script like the one I just wrote, even as an option. Hopefully they do something to ease the pain, though, like maybe adding an option to error on attempting to install type-less package. Now we’re at least in the same universe as Python 2/3, in that package authors would be pressured to update their packages to add this field, but it’s literally a one-line change that they need to make and it doesn’t require any coding knowledge or testing. So there would be some churn, but it’s not an ecosystem earthquake.

So basically we know how painful it will be on Day 1 when we release this experimental flag, but not how painful it would be on the day we want to make it Node’s new default; that depends on how the ecosystem reacts and adapts. And we can always revise this behavior while the flag is experimental.

This is the part that I do not understand, why should we break people modules?

I think a better way of putting it is, is the clunkiness/churn/pain worth changing how type-less package.json files are interpreted? @mcollina seems to think that the pain level is high and the benefit of the change minimal to nonexistent, which makes the decision easy. I’m more of the view that the pain level is low—not negligible, but low, and hopefully decreasing to negligible with ecosystem support—but the benefit is also low. If the pain can be eliminated, then even a low benefit becomes worthwhile.

I think the strongest argument goes something like this: Currently, all of Node’s server-side competitors support ESM out-of-the-box by default with no configuration required. (And TypeScript, for that matter, but that’s another story.) We’re losing the “new user” UX story because compared to Bun or Deno, it takes a few steps before users can just get going with writing code in Node that uses import and export. Having npm init add "type": "module" by default would help a lot, sure, but that’s out of our control and npm has rejected all such suggestions repeatedly. Maybe that would change if this new mode started making "type" a required field, adding some pressure on them, but who knows. But that’s ultimately the ideal: everything “just works,” both installing any package as well as writing ESM even in a typeless package. In other words, all gain and no pain. We can get there if we get some support from package managers, but if not then we’ll need to compromise.

I +1 for ESM by default, and want to see it in NodeJS 20 ASAP.

If there are breaking changes for exists code, why not keep using NodeJS 18 LTS?

I don't see any reason why we would need to change the type-less behaviour from the current commonjs when we can just get npm init and equivalents to include "type": "module" by default. If the json property is being set by default there's really no reason for us to risk breaking old modules to change the type-less default. Just go forward with getting package managers to always include the type in the future and assume it's old and therefore commonjs if not present.

As for running without a package.json, given the shebang problem I think it would be simplest and cleanest to just require the mjs extension, which already works today.

when we can just get npm init and equivalents to include "type": "module" by default.

We can’t, that’s part of the problem. It was proposed and rejected in 2019, then proposed and rejected again in 2021, then proposed again in 2022 (still open) and again in 2023. From what I can tell most voices seem to be in favor, though there’s one strongly opposed commenter who is extremely prolific in campaigning against the change. Though npm's track record on this relatively minor issue leaves me pessimistic in terms of how supportive a partner they will be in helping us migrate to ESM by default.

Just go forward with getting package managers to always include the type in the future and assume it’s old and therefore commonjs if not present.

It might be worth asking the npm registry folks if they might start requiring the type field on publish into the registry. I assume a package.json file itself is already required? Then at least going forward past the “turn on” date for requiring type, any new packages published won’t be presenting this issue.

As for running without a package.json, given the shebang problem I think it would be simplest and cleanest to just require the mjs extension, which already works today.

The request isn’t to do something that’s possible today, the request is to be able to run extensionless shebang scripts as ESM. This has been an ongoing and persistent user request over many years, and I don’t think we can just wave it away with the current workarounds like symlinks that can kinda-sorta-but-not-quite satisfy the user request.

@ljharb

Seemed to me like @ljharb multiple times stated that inclusion of the type field was a good idea. Just that it should be "commonjs" until ESM and the tooling to support it is properly ready. I would not interpret any of that as blocking adding the type field. Seems to me like it's just work no one has stepped up to do, and some have been unreasonable about expecting to make ESM the default there before the ecosystem was ready for that. We can start by always adding "type": "commonjs". Once all the core ESM tooling like loaders are stable we can start asking the user if they want to change from that default to "module", along with some explanation of the implications of doing so. Eventually, when the ecosystem support is there, we can think about switching that to the default.

If we want to be able to run a shebang file itself as ESM, we could just have an alias entrypoint like node-esm which runs node --input-type=module.

Encouraging people to patch third party code is a nightmare waiting to happen; my project is only supported when untouched, but if node encouraged any package patching - even just the type field - maintainers would rapidly burn out from all the confusing support requests.

I think there's a much bigger potential for breakage with pursuing a default "type" change - however, is there a reason not to pursue specifically a setting for extensionless files? That would allow it to support wasm, CJS, ESM, JSON, etc.

For example - a package.json flag, and/or a node flag, that asks "what extension should we pretend extensionless files have?" - and defaults to "cjs", for back compat. Thoughts?

I spun off the question of how to handle package.json files that lack a type field into a new issue: #49494. I updated the top post to point to that issue instead of suggesting a particular behavior for that condition. If people don’t mind, let’s please move the discussion of that question over there.

So aside from that question and the other larger one (#49431), are there any other open questions we need to address? Am I missing any other behaviors that this new mode should enable? Are there any other use cases we’d like this new mode to cover?

Since the motivating use case seems to be “i want an extensionless file to have ESM”, what about an issue for a very focused solution for that use case, which i suggested in #49432 (comment) ?

@LiviaMedeiros

The discussions in the various issues seems to have quieted down, and it seems like we might have a consensus. @LiviaMedeiros since you seem eager to make this happen, do you mind putting together a PR? It would be a combination of several of the recent PRs that you posted. I updated the top post to reflect my guess at the consensuses from the various threads. If you could put those together in a single PR, behind the flag with whatever name gets chosen in #49541, then I think we might have something that can land.

@GeoffreyBooth

@GeoffreyBooth can you please respond to #49432 (comment)? i don't believe there is indeed consensus yet.

I don't see any overall consensus here, either. In particular, on these two points:

Flag or separate binary.
IMHO both, because each has pros and cons. But maybe just one is enough.
Option to provide main entry point as URL.
Not discussed at all, although it might be a very controversial feature because it would need to mess with argv[1].

on these two points:

They were part of the original proposal and no one objected, so I'm assuming that everyone is okay with these aspects.

The separate binary we already know is controversial from your other PR, so I would leave that part out. We can start with the flag and then revisit the binary option.

The second binary seems unfeasible as it would double our install and download side and almost double the work for our releasers. But given it's not me to have to do that work, I leave that objection to the releasers.

@mcollina

The second binary seems unfeasible as it would double our install and download side and almost double the work for our releasers. But given it's not me to have to do that work, I leave that objection to the releasers.

Ok FTR, as one of the @nodejs/releasers I object for the same reasons as @mcollina.

The second binary could just be a script that calls the original one with all the same input args and additionally the input-type arg. That wouldn't be twice the size at all.

The CLI will parse the main entry point as an URL string, similar to how the value of --import is parsed.

I still object this as absolutely not feasible in combination with other parts of this proposal.

Please provide at least an example of how can it work.

this as absolutely not feasible in combination with other parts of this proposal.

I don't understand the concern. In #49295 you implemented this along with other ESM-first behaviors. Wouldn't we do it just like that?

The command line option proposed in #49295 has different form (--module <URL>) which can't work with shebang by design.
How is this solved here?

How is this solved here?

This proposal doesn't directly solve the shebang case but it gets us close. Once what is proposed is built, we have two options:

Make this mode Node's default, thereby not needing a flag for ESM scripts but creating the inverse problem of needing a flag for CommonJS scripts. Maybe users can install two versions of Node on their machines like how people have python2 / python3 if they really want to support both syntaxes as shell scripts; Node could decide that we only support ESM for this use case moving forward.
We figure out a way to make a second binary that the build and release team approves of, that doesn't double the size of Node etc.

The second option is better but possibly unachievable. The first option we know we can do and it's better than the status quo, in my opinion.

I want to write what are typically called shell scripts in ESM JavaScript, where I can have a single extensionless file anywhere on my disk that uses ESM syntax and it doesn’t need a nearby package.json file or a symlink or wrapper file in order to run as ESM.

I want consistency in how the CLI handles references to files; currently --import accepts URL strings but the main entry point must be a path string (and therefore can’t be a data: URL or https: URL, such as to work with --experimental-network-imports).

Please provide an example of how these two would work in this mode. Probably an extensionless ESM file that can be launched with node binary (separate or with flag, anything goes), launched directly (assuming we run it on e.g. Ubuntu with latest coreutils), and launched over https (assuming we configured web server to provide all appropriate headers).
For simplicity we can assume that it's a stray file: no package.jsons around, no node_modules/, and no subsequent imports.

I'm mainly interested in import.meta.url (i.e. have some ?a=b#c in URL) and process.argv (i.e. have firstArg secondArg, too) values in each case, so it would be nice to demonstrate them whenever applicable.

I’m mainly interested in import.meta.url (i.e. have some ?a=b#c in URL) and process.argv (i.e. have firstArg secondArg, too) values in each case, so it would be nice to demonstrate them whenever applicable.

What would be the desired UX be for query parameters paired with an extensionless file? Like if you had /usr/local/bin/extless, where extless is a JavaScript file with a Node shebang, I assume you’re running this via just extless; if you try to run it like extless?a=b#c the shell is going to complain because it can’t find a file with that name. If you want to pass parameters to an extensionless script, I think you need to use regular CLI arguments like extless --a=b c. Right?

Please provide an example of how these two would work in this mode. Probably an extensionless ESM file that can be launched with node binary (separate or with flag, anything goes), launched directly (assuming we run it on e.g. Ubuntu with latest coreutils), and launched over https (assuming we configured web server to provide all appropriate headers).

I’m not sure I’m following what you’re asking for. You want to run an extensionless file . . . that’s served over HTTPS? Do you mean you want to run a command that begins with node, like node https://a.tld/script? I assume you’re not trying to run just https://a.tld/script as the entire command and you’re expecting your shell to download and evaluate that?

I created a file named extless and ran chmod +x on it and gave it these contents:

#!/usr/bin/env -S node --enable-source-maps --no-experimental-fetch
console.log(typeof fetch)

Then I ran docker run --volume "$PWD":/app --workdir /app --interactive --tty --entrypoint bash node:20 to open a shell within the Node 20 Docker container. Running ./extless prints undefined. If I remove the --no-experimental-fetch, it prints function.

So at least in distros with modern env, that allow env -S, we can pass flags via shebang. Yes there’s still the problem of the distros lacking this support, and I’m not aware of a solution for them other than a new binary; but I wouldn’t want to deny this flag to distros that can take advantage of it, just because there are other platforms that can’t.

If I change the second line to console.log(process.argv) and run it as ./extless foo bar, it prints:

[ '/usr/local/bin/node', '/app/extless', 'foo', 'bar' ]

So this is an extensionless file that can be launched with the Node binary and launched directly, which answers the first two parts of your request. For the third, “launched over https (assuming we configured web server to provide all appropriate headers)” I assume you mean running via something like this?

node --experimental-type=module --experimental-network-imports https://example.com/extless

In this new mode, yes, I would expect this to work. It would ignore the shebang line in the file, since node is the entry point rather than bash or some other shell, and so you’d have to include whatever Node flags you need as part of the command.

It would be simpler test with data: URLs:

node --experimental-type=module 'data:text/javascript,console.log("hi!")'

I would expect this to work, like how this command works today:

node --import 'data:text/javascript,console.log("hi!")' --input-type=module --eval ''

The import.meta.url of a data: URL is just the entire data: URL. I would assume that for an https: URL it’s the https: URL, but that’s a question for --experimental-network-imports.

Does this answer your questions?

No, for starters I need to see both import.meta.url and process.argv for all three cases. The code should be at least

#!<shebang in any form that would enable this mode>
console.log(import.meta.url, process.argv);

Examples that don't have searchstring, hashstring and CLI arguments don't demonstrate how it works.
Examples that don't print or assert the target values don't demonstrate it as well.

I need to see both import.meta.url and process.argv for all three cases.

It's not a goal of this PR to have specific values for those. Why not implement it and see what they are? I don't know offhand what they should be, but they don't make much difference to me. The use cases at the top are satisfied no matter what these values are.

Do you have specific needs for particular values? Are there use cases satisfied by some values and not others?

For example, unset/unpredictable/inconsistent argv will break all programs that take more than zero arguments.

For example, unset/unpredictable/inconsistent argv will break all programs that take more than zero arguments.

Can you list the potential commands you have questions about, and what outputs you’re looking for, with an explanation of why?

For example, unset/unpredictable/inconsistent argv will break all programs that take more than zero arguments.

`argv`

As I wrote in #49432 (comment), any arguments on the shebang line are ignored.

With a file named extless containing:

#!/usr/bin/env -S node --enable-source-maps --no-experimental-fetch
console.log(process.argv)

Running extless foo bar in current Node prints argv of [ '/usr/local/bin/node', '/app/extless', 'foo', 'bar' ]. This argv would be the same in the new mode.
Running node extless foo bar in current Node prints argv of [ '/usr/local/bin/node', '/app/extless', 'foo', 'bar' ]. This argv would be the same in the new mode.
Current Node can’t run an extensionless ESM file or run an HTTPS entry point, but we can approximate how Node would behave if it could by running --import <extensionless file or HTTPS URL> --eval ''. Running NODE_TLS_REJECT_UNAUTHORIZED=0 node --experimental-network-imports --import https://localhost/extless --eval '' foo bar, where a local server serves that same file with valid headers, in current Node prints argv of [ '/usr/local/bin/node', '/app/extless', 'foo', 'bar' ]. The argv would be the same in the new mode, including if it were run as the main entry point instead of via --import.

To do the last test I set up a localhost server via:

import { createServer } from 'node:https'
import { readFileSync } from 'node:fs'

const options = {
  key: readFileSync('./key.pem'),
  cert: readFileSync('./cert.pem'),
}

const content = readFileSync('./extless')

createServer(options, (req, res) => {
  res.setHeader('Content-Type', 'text/javascript')
  res.end(content)
}).listen(443)

`import.meta.url`

With files named extless and test.mjs each containing:

#!/usr/bin/env -S node --enable-source-maps --no-experimental-fetch
console.log(import.meta.url)

Running NODE_TLS_REJECT_UNAUTHORIZED=0 node --experimental-network-imports --import 'https://localhost/extless?foo=bar' --eval '' prints https://localhost/extless?foo=bar. This import.meta.url would be the same in the new mode, including if it were run as the main entry point instead of via --import.
Current Node can’t run an extensionless file as ESM, either as the entry point or in --import, so using test.mjs and running via node --import 'file:///app/test.mjs?foo=bar' --eval '' prints file:///app/test.mjs?foo=bar. This import.meta.url would be the same in the new mode, including if it were run as the main entry point instead of via --import, and if extensionless URLs were permitted as entry points then node file:///app/extless?foo=bar would print file:///app/extless?foo=bar. For relative URLs, running node --import './test.mjs?foo=bar' --eval '' in current Node prints file:///app/test.mjs?foo=bar; and if extensionless URLs were permitted as entry points then node ./extless?foo=bar would print file:///app/extless?foo=bar.
Running test.mjs directly as ./test.mjs prints file:///app/test.mjs. This import.meta.url would be the same in the new mode, and if extensionless files were treated as ESM then running ./extless would print file:///app/extless. There would be no way to pass query parameters to a Node script run via this method.

Adding tsc-agenda for visibility and to discuss shipping as part of 21.0.0. We should probably have some kind of PR push to publicize this flag debuting in 21.0.0, especially if we also want to announce making it enabled by default in a future version such as 22.0.0. There's no PR yet much less a landed one, but as 21.0.0 is coming up fast I figured we should start talking about these logistics.

I probably should have made it more clear that I need proof that both features work in combination (because the proposal in this issue claims it's possible to do both under same flag). Why the examples are split?

As I wrote in #49432 (comment), any arguments on the shebang line are ignored.

Does that mean that node must somehow detect if the file was launched using shebang, and throw MODULE_NOT_FOUND in this case (unless search === '' && hash === '') even though running shebang line in shell would work just fine? That sounds like a solid footgun.

Running NODE_TLS_REJECT_UNAUTHORIZED=0 node --experimental-network-imports --import https://localhost/extless --eval '' foo bar, where a local server serves that same file with valid headers, in current Node prints argv of [ '/usr/local/bin/node', '/app/extless', 'foo', 'bar' ].

This is very interesting. How was the value of process.argv[1] ('/app/extless') calculated here?

Since the overall proposed mechanism seems to be "interpret argument that always had to be path as urlstring and in some cases as path", all counterarguments from #46009 and #48994 seem to be applicable here. IIRC the consensus in those issues was that we absolutely mustn't have such mechanism.

Moreover, the switch itself adds a few huge security issues.
Any path that contains # will be basically truncated.
Any path that contains ? will be truncated and the rest of it will be passed as parameters.
Any path sanitizing will stop working because of percent-encoding backdoor.

There also are less important problems (for example, Windows support and specifiers starting with #), but this is already enough to tell that what proposed here is not feasible.

I probably should have made it more clear that I need proof that both features work in combination

I really don't understand what you're trying to achieve.

If parsing the entry as URL isn't possible for some reason, we drop that aspect. It's only here because you requested it.

Sorry if the exact objection reason was confusing.

The CLI will parse the main entry point as an URL string, similar to how the value of --import is parsed.

I'm objecting having this part here.
I don't think "parsing main entry point as URL" as in "parsing the first positional argument as URL" is achievable - at least, in any way that doesn't introduce a bunch of breaking changes, terrible DX/UX and security problems.

Node.js indeed should support providing main entry point as URL, similar to how --import works (except for "module specifiers", maybe); however it should be done in a different way. #49295 proposed one of possible ways to do it (for the record, the draft has bugs with argv; but they are fixable), but if this proposal (specifically the "default flip" part) lands, it will allow to support URLs in less controversial way as independent feature.
Therefore, I think it's safe to remove this part from this proposal.

Aside from that, I agree that main point as URL is very ESM-ish feature. If there is a feasible approach to combine it with this proposal, I'll be glad to lift this objection.

I agree that main point as URL is very ESM-ish feature. If there is a feasible approach to combine it with this proposal, I’ll be glad to lift this objection.

Doing the exercise to see how current Node behaves led to me discovering the pattern of --import <URL> --eval '', which essentially is a way to define the entry point as an URL string, albeit in a workaround way. Perhaps a better approach would be to permit --import to live by itself, so like node --import ./entry.js?foo=bar or node --import file:///app/entry.js?foo=bar etc, without the need for --eval '' or a file path entry point. Then we achieve the goal of allowing an URL string entry point, without breaking the current expectations of the main entry being parsed as a path string.

This is something we could do in a standalone PR. Just as --eval currently somehow tells Node not to expect an entry point CLI argument, if --import is passed without --eval or --print or an entry point CLI argument, then --eval '' is inferred. It would be the same end result, and it’s simple enough to explain to users: node entry.js parses entry.js as a path string; node --import entry.js parses entry.js as an URL string.

I think it was already discussed before. --import doesn't (and shouldn't) launch something as main entry point. The difference is hard to observe right now, but it will become obvious once we have import.meta.main and modules that utilize it. Just imagine what would happen to python if if __name__ == "__main__": stopped working.
Also IMHO we should preserve ability to launch REPL with imported polyfills, it's handy.

Perhaps there are ways to improve "noop main module" usecases so we don't have to use --eval ';', and there are ways to parse main entry point as URL such as introducing --entry-url; but it probably should go to #46009 or #49204.

If we want to combine this with ESM-first flag, there should be either a reason due to impossibility of implementing ESM-first flag without it, or a reason to exclude CJS-first projects from having this feature.

a reason to exclude CJS-first projects from having this feature.

The reason is that changing how the main entry point is parsed would be a breaking change. It also doesn't make sense for CJS-first projects to load via URL, since all CommonJS modules are referenced via path.

--import doesn’t (and shouldn’t) launch something as main entry point. The difference is hard to observe right now, but it will become obvious once we have import.meta.main and modules that utilize it.

The same logic that detects --import without --eval or some other entry point could easily conclude “hey, that input passed to --import is the only entry point! It gets import.meta.main.” And if someone does node --import a.js --import b.js and still no other entry point or input, we would define via documentation that the last --import value is the one deemed to be the “main,” because traditionally --import and --require are used to load things prior to the “main” entry point, hence the main entry point comes last in the list.

When an extensionless file is outside of any defined package.json scope, it can be run as an entry point per the conditions we establish in Discussion: In an ESM-first mode, how should extensionless entry points be handled? #49431: if the file begins with the Wasm magic bytes, it will be evaluated as Wasm, and likewise for any other future file types with defined headers that don’t conflict with JavaScript; otherwise it will be evaluated as ESM JavaScript. This support for extensionless files would be limited to entry points; they still would not be allowed in import.

Aside from URLs, I didn't see this part discussed nor explained here as well.

Why we should allow extensionless Wasm as main entry point, but not as imports? What are usecases for this?

Why we should allow extensionless Wasm as main entry point, but not as imports? What are usecases for this?

Sorry I forgot to update that per #49531 (comment). The reason I wrote that was because that’s how the original implementation was (limited to entry points and not imports) but upon investigation I don’t think we need to keep that limitation anymore. Updated the top post.

I want consistency in how the CLI handles references to files; currently --import accepts URL strings but the main entry point must be a path string (and therefore can’t be a data: URL or https: URL, such as to work with --experimental-network-imports).

This should probably be rephrased, there is consistency today, it's just not very intuitive how it works.

The CLI will parse the main entry point as an URL string, similar to how the value of --import is parsed.

How would that work? This is too vague, we should be more precise here, it raises more questions than it answers:

What value of argv[1] will be given to userland?
Would bare specifiers be interpreted as relative URLs?
How would that work with shebang? AFAIK the OS will always pass a path, not a URL.
We should explicitly set the goal of getting rid of some of the legacy behavior; e.g. the fact that we always pass argv[1] to CJS loader first should probably change under the new mode.

@aduh95

@aduh95 I’m thinking of changing that section per #49432 (comment), where we keep the current behavior for parsing of the entry point but start permitting --import to define an entry point, i.e. node --import entry.js as the entire command. Then users have a way of launching an URL entry point without needing to break how the current CLI entry argument is interpreted. Thoughts?

This behavior can be a PR of its own rather than part of the new flag.

That would be a breaking change, and we would need to find a new way to enable REPL. In any case, removing it from the proposal sounds good to me, but I would still like to see the legacy behavior (where argv[1] is passed to the monkey-patchable CJS loader first) go away as part of this proposal – otherwise we're never going to it.

That would be a breaking change, and we would need to find a new way to enable REPL.

I think you would enable the REPL via --interactive. Like node --import foo --interactive. Or just -i: https://nodejs.org/api/cli.html#-i—interactive. I think this is an allowable semver-major change, it’s not that big a deal to need to pass -i going forward if you want to also import a dependency into REPL mode.

Alternatively we can just document that to run a URL from the command line, it’s --import <URL> --eval '', which works today. It won’t provide a useful import.meta.main if we ever add that, but we currently don’t have it, so 🤷

In any case, removing it from the proposal sounds good to me, but I would still like to see the legacy behavior (where argv[1] is passed to the monkey-patchable CJS loader first) go away as part of this proposal – otherwise we’re never going to it.

Can you propose specific language I should use for that bullet point?

What value of argv[1] will be given to userland?

I don’t know, what would you recommend? If you run node --eval 'console.log(process.argv)' you get back just the path to the node executable, so we could do that, treating argv[1] as undefined. Or define it as a file path when the entry point resolves to a file on disk, and undefined otherwise. I don't know.

Would bare specifiers be interpreted as relative URLs?

I was assuming that we would follow how --import behaves, which I think is that bare specifiers resolve based on the resolution algorithm and relative URLs need to begin with ./ or ../.

How would that work with shebang? AFAIK the OS will always pass a path, not a URL.

See #49432 (comment).

We should explicitly set the goal of getting rid of some of the legacy behavior; e.g. the fact that we always pass argv[1] to CJS loader first should probably change under the new mode.

Fine by me, can you be more specific? I also feel like this can land as a follow-up, we don’t need to get every last detail in the first PR, so long as we get all the desired behaviors in the flag before it goes stable.

(

Something that is missing in the above is the repeated use of "extensionless". It should really be "extension-agnostic".

Might sound picky, but consider that it works perfectly fine to write a bash / perl / python / many others script called x.js. So it should also be the case for adding node+esm to the list. This implies that it should be possible to write x.js and run it as a module.

So having an extension determine how to run things in some default way is nice, but being able to specify how to run things via a flag or a different binary executable is needed.

)

This implies that it should be possible to write x.js and run it as a module.

I assume you mean that it should be possible to run x.js as a module when x.js lives outside of any module scope (i.e. there’s no package.json nearby or above it). This would become possible under this flag the same way that running extensionless files would become possible, for systems that support env -S. The new flag could be passed in the shebang line, and work for platforms with modern env.

@LiviaMedeiros

@LiviaMedeiros

The solution for portable enough shebang without separate binary is to write #!/usr/bin/node --esm, simple as that.

Just to mention it, it is by far not portable to expect node to live in /usr/bin. For example NixOS and other distributions would not work that way. This should not be recommended, we should recommend #!/usr/bin/env and need a solution for this.

@GeoffreyBooth

@GeoffreyBooth: Well, yeah, part of this is saying that you should be able to run x.js as a module "when ... scope ...".

But the underlying issue that I want to highlight is that there should be a mode where the decision on how to run the code should be independent of anything outside of the file contents (that is, also independent from the file name).

Perhaps a user story would demonstrate it better: I implement a script in {CJS,ESM}, and I call it foo{.js,.mjs,/whatever}. I should have a way to add a shebang line that makes it work fine independent of the name that I ended up choosing -- and it sounds like you're saying that this will be possible, so that's one good side.

The other good side which I think is important in this context is that it should not depend on the environment -- that is, also independent of any possible package.json sibling, or in any of the parent directories. The reason for that is simple: I want to hand out my script in a way that makes it work as expected, and not break because some else put it in a directory that has a package.json context. Otherwise, I need to also tell people to ensure that it is put in a place without a package.json in its FS context. An alternate description would be:

I want to write ESM syntax by default without needing to opt into it, such as via package.json file ...,
or opt out to it, such as via ensuring that no package.json file.

If that's possible, then I'm happy, and my side-comment is reduced to just making it clear that this is the case. (And I'll happily pre-apologize in this case for the extra noise (that was the reason I put that comment in parens...).)

It would definitely be nice to have (as i've proposed multiple times) a mapping in package.json from extension to "type", that defines it for that package scope. An empty string would work for extensionless and it'd automatically be forward compatible to anything else in the future, and would have been a much cleaner and more intuitive setting than the type field. I continue to think this is the only reasonable way forward.

@ljharb

@ljharb I'm actually making a point for being able to not depend at all on a package.json. What you're suggesting makes perfect sense for distributing a directory of scripts, but any dependency on an external file is a disaster when the intention is for scripts that you just drop into some .../bin directory (eg, a system package that wants to add some scripts into /usr/bin would be a disaster if it needs its own package.json living in the same place).

(Edit:) Also, a point I made elsewhere is that I should be able to take any script I wrote (suffixless or not) and change the language its implemented without renaming it. (Which is IMO one of the major advantages of unx-style shebang lines vs windows-style suffix registrations.)

Sure - but it would be pretty straightforward to design, implement, and explain a command line flag that can supply the same mapping.

Virtual package.json flag was not blocked last it came up (sadly I never made a PR) but was intended for testing not general usage #37848 . That would satisfy a lot of things without bespoke flags if this does seem to lean towards package.json level config. This might be important since 2 projects A and B may differ: /A/bin/a - expects ESM /A/node_modules/B/bin/b - expects CJS Where A invokes B via child process

…

On Sat, Sep 16, 2023, 5:16 PM Jordan Harband ***@***.***> wrote: Sure - but it would be pretty straightforward to design, implement, and explain a command line flag that can supply the same mapping. — Reply to this email directly, view it on GitHub <#49432 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABZJI63CLTIQOR3ATGO53TX2YQNLANCNFSM6AAAAAA4H2ZUY4> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

I’m actually making a point for being able to not depend at all on a package.json . . . I should be able to take any script I wrote (suffixless or not) and change the language its implemented without renaming it.

On a system with modern env, where you can do something like #!/usr/bin/env node --some-flag, this is possible via the module customization hooks and --import and data: URLs. You could write out a resolve hook that told Node to interpret whatever your custom filename was as a module. Convert that into a data: URL. Then your shebang line could be something like #!/usr/bin/env node --import 'data:text/javascript,import { register } from "node:module"; register("data:text/javascript,import { pathToFileURL } from \"node:url\"; export async function resolve(specifier, context, next) { if (specifier === \"myfilename\") { return { url: pathToFileURL(process.cwd() + \"myfilename\"); format: \"module\" } } return next(specifier, context) }")'. Yes it’s a mouthful, and I probably didn’t get it completely right without testing, but this approach should work.

Is it as clean and elegant as a bespoke flag for exactly that purpose? No, but not every use case needs a flag specifically for it, especially if the use case seems less common and there’s already some way to achieve it.

@GeoffreyBooth

@GeoffreyBooth -- I'm very familiar with these kind of hacks. The most recent one that I have is this node-esm which is used with a simple #!/usr/bin/env node-esm. It's good that it's at least possible, but the fact that it's far from "easy" makes it a problem.

I also completely agree that "not every use case needs a flag" -- but the bottom line, which I think is a perfectly reasonable thing to expect -- is to be able to write an ESM script that does not depend on its filename, and hopefully on its FS environment. Just like with almost every other (scripting) language.

And yes, the fact that it is possible to get it to work is a very good thing, and you could argue that I should just dump something like that in some corner of the interwebs so others can use it -- but because it's such a tricky thing means that people would (rightfully) suspect it, and therefore people will continuously reinvent this wheel.

(BTW, the reason I stuck my head in this thread is my hope that whatever happens, it would either implement something like that, or at least something that is close enough to make my code much shorter and maybe even obvious.)

In fact @aduh95 (who I should credit for the core of my hack) talked about the problem of python3 (and the usual fear of kernels that mistreat spaces on the shebang). And he also mentioned making it available on npm as a suggestion for that corner. But I think that this is important enough to leave as something that is better to come from within the project itself to gain traction and free people to focus on their own wheels.

Finally, I know that this is all a very sick bed, so I appreciate any steps to make it a little better. I'd be happy to contribute the above in a form that is acceptable, but I know that there's so much noise around it to the point that such a PR is likely going to be stillborn. (Some projects (eg, git) use a contrib directory for such in-flux work.)

is to be able to write an ESM script that does not depend on its filename, and hopefully on its FS environment.

This proposal aims to support this for extensionless files, as well as extensions that Node supports like .js, .mjs or .cjs. I don’t think “I want to be able to run an ESM JavaScript file named main.py” is a use case that we’re likely to support any better than via module customization hooks, or a user making a symlink; it’s just too niche. It may be the case that many scripting languages ignore file extensions, but Node never has, and Node is primarily for applications rather than shell scripts. Speaking from my own experience, I can’t think of any shell scripts I’ve ever run that weren’t either extensionless or with a .sh extension; I don’t think it’s much of a restriction in the real world to require either no extension or a .js/.mjs/.cjs extension. You can also have Node run JavaScript from within a shell script; I wrote an example of that at https://github.com/disney/meteor-base/blob/main/src/docker/connect-to-mongo.sh.

I don’t think “I want to be able to run an ESM JavaScript file named main.py” is a use case that we’re likely to support any better than via module customization hooks, or a user making a symlink; it’s just too niche.

Well, .py was an extreme example; much more likely is to revise an old foo.js to modules. But more generally, by treating suffixes in a special way, you're restricting the possible names of scripts. (A quick ls -d /{usr/,}bin/*.* would probably get you a bunch of names with .s that are almost all not serving as suffixes.)

You can also have Node run JavaScript from within a shell script; I wrote an example ...

Yeah, redirections are another hack that is not general, since it prohibits using stdin.

..., and Node is primarily for applications rather than shell scripts.

This looks like a side-comment, but IMO it points at the core of the issue, since it's a hint of a policy. This is in the sense that it could be used to reject such contributions. If this is the case, it would be best to point at such a policy than go around it, which would help reducing friction as in the other issue thread.

(Note that I'm not trying to say anything provocative here, I'm just saying that it would be easier on both sides if things are clear.)

it would be best to point at such a policy

https://nodejs.org/en/about:

As an asynchronous event-driven JavaScript runtime, Node.js is designed to build scalable network applications.

@aduh95

@aduh95 I've done some digging, and I think I have tentative answers for the questions you posed above regarding parsing the entry as a URL:

What value of argv[1] will be given to userland?

So today, running node app.js will yield an argv[1] along the lines of /var/www/app.js. Since it’s already a conversion of an input string into another string, I think it would be fine for something like node --type=module file:///var/www/app.js to yield a path string argv[1], such as in this case the same /var/www/app.js. Then we avoid breaking anything that’s expecting an absolute file path in argv[1].

The argv[1] value is unset for entries that don’t correspond to files on disk, like --eval. So I think for non-file: URLs, we would do the same. node --eval 'console.log(process.argv)' foo bar prints [ '/usr/local/bin/node', 'foo', 'bar' ], so I would expect node --experimental-type=module 'data:text/javascript,console.log(console.argv)' foo bar to also print [ '/usr/local/bin/node', 'foo', 'bar' ].

There’s also process.argv0 that stores the original value of argv[0] before it gets converted. We could make a similar process.argv1 to store the pre-conversion value of the entry point. So in the node --type=module file:///var/www/app.js example, process.argv[1] would be /var/www/app.js and process.argv1 would be file:///var/www/app.js. In the node --type=module 'data:text/javascript,console.log(3)' example, process.argv[1] would be undefined/missing and process.argv1 would be data:text/javascript,console.log(3).

Would bare specifiers be interpreted as relative URLs?

I think so. I’m not sure what the use case would be using the full resolution algorithm. Doing so would mean that you could never run node --type=module app.js, only node --type=module ./app.js, and that seems annoying. You could use a bare specifier entry via node --import pkg --eval '' if you wanted that, among other ways.

Disallowing bare specifiers would also be consistent with legacy --type=commonjs behavior, where argv[1] is parsed via path.resolve, not require.resolve.

How would that work with shebang? AFAIK the OS will always pass a path, not a URL.

I did some testing. At least on my Mac, the OS passes whatever string was used as the command. I made a file shebang and saved it in ~/Sites/node, and modified https://github.com/nodejs/node/blob/main/lib/internal/process/pre_execution.js to save a new property process.argv1 to preserve argv[1] before this file updates it. Within shebang I logged out process.argv1.

When run via ./shebang it prints './shebang'. When run via $PWD/shebang it prints '/Users/geoffrey/Sites/node/shebang'. So maybe the “OS passing the path” part you’re remembering is the path.resolve replacement we do in pre_execution.js.

At least on my system and maybe all POSIX systems, these pre-replacement strings are parseable as URLs: both new URL('./shebang', pathToFileURL(process.cwd()) and new URL('/Users/geoffrey/Sites/node/shebang', pathToFileURL(process.cwd()) return the same thing, an URL instance for file:///Users/Geoffrey/Sites/node/shebang which is what we want. So basically it seems like it's happenstance that for the shebang case, the strings sent from the OS work as relative URLs. Presumably this wouldn't work in Windows, but do people run shebang scripts in Windows?

We should explicitly set the goal of getting rid of some of the legacy behavior; e.g. the fact that we always pass argv[1] to CJS loader first should probably change under the new mode.

Agreed, I’ve updated the top post.

At least on my system and maybe all POSIX systems, these pre-replacement strings are parseable as URLs

I think we've already established that parsing paths as URLs is not an acceptable solution: it breaks as soon as the path contains %, #, or ?. We wouldn't accept a solution that "sometimes works, sometimes doesn't, sometimes loads an entirely different file".

I think we’ve already established that parsing paths as URLs is not an acceptable solution: it breaks as soon as the path contains %, #, or ?.

My examples above aren’t parsing paths, at least for the non-shebang case. They’re parsing relative URLs, based on the file URL of process.cwd(). We would define it as such. node ./file.js?foo=bar should work, as desired in #49295. It happens to be identical to what users in POSIX systems would write as paths, but it’s defined input as a relative URL. In Windows, node C:\folder\file.js would not work; Windows users would need to write with forward slashes, as either relative or absolute URLs. This would just be part of the specification, that we would document.

Just so we’re clear, my goal is to eventually allow node https://example.com/script.js and node 'data:text/javascript,console.log("hello")'. I think it’s okay to tell people something along the lines of “If you wish to run a file whose filename contains characters that requiring escaping in a URL, such as % or # or ?, you need to specify it using an absolute file URL such as node file:///path/to/wha%3F.js.” We can even have an error path where if the entry point isn’t found, Node checks to see if parsing as a path would have worked and if so, suggests the correct file: absolute URL instead.

So that leaves the shebang case. When I run ./shebang, the original argv[0] and argv[1] passed to Node from my OS, before Node does any replacements, are values like 'node' and './shebang'. The first value, of argv[0], is whatever executable I write in the shebang line; if I make the shebang file begin with #!/usr/bin/env /Users/geoffrey/Sites/node/node then argv[0] becomes '/Users/geoffrey/Sites/node/node'. It would seem that there’s no way to know, at least from argv alone, whether Node was launched directly or from a shebang script. (Is there a way to know via some other method?)

If I do mv shebang 'shebang?' and run ./shebang\? I get argv[1] before replacement of './shebang?', on my test branch. So this is perhaps one approach: if the original argv[1] contains characters that require escaping to be allowable in URLs, we could check to see if a file with those characters exists on disk; and if it does, error to tell people to use an absolute file URL with escaped characters. This would mean that we are disallowing shebang files whose filenames have special characters, since you can’t run a shebang file as an absolute file URL, but I think that’s fine. What do you think? Is there a better approach? I would much prefer a solution that allowed input to be URLs, even at the expense of special characters in shebang files, rather than requiring the entry to always be a file path. There’s always going to be a tradeoff there but I think there’s much more benefit to allowing URLs than allowing shebang filenames with special characters.

Folks subscribed to this issue, please tell me what you think. Which is a better UX under the new mode:

🚀 Only URLs are allowed as the main entry point:
- node entry.js?foo=bar
- node wha%3F.js
- node 'data:text/javascript,console.log("Hello")'
- node file:///app/start%20here.js
👀 The main entry point is still parsed as a path (though with explicit extension required) and if you want to run Node with an URL as the entry, you use --import with no entry point and you need to start with ./ for relative URLs:
- node --import ./entry.js?foo=bar
- node 'wha?.js'
- node --import 'data:text/javascript,console.log("Hello")'
- node '/app/start here.js'

Assume for the purposes of this question that both can be made to work: we can figure out ways to properly error on ambiguous special characters, etc.

I would follow the latter case and minimize the breakage. Please use paths, folks are used to use paths to manipulate files in the filesystem.

I think the terminology is mixed up. We don't "parse the main entry point" here, we parse first positional argument in something that points to main entry point.

My answer is: we absolutely must keep parsing first positional argument only as path, and we should be able to provide main entry point as URL in a different way than as first positional argument.

@LiviaMedeiros

I agree with @LiviaMedeiros and @mcollina.

We must not change how the first positional argument is interpreted by default. We don't want to break the use case and user expectations.

What we can do is create a new flag that either:

Takes an URL (or relative URL as defined in 🚀) as a parameter
Takes no parameter and changes the behavior of the first positional argument to 🚀.

Okay, so it sounds like people want the second option, where if you want to pass an entry point that’s an URL you do it via node --import ./entry.js?foo=bar. Currently that command launches the REPL, with entry.js loaded beforehand.

We could ship a semver-major change where node --import <entry URL> would behave the same as node --import <entry URL> --eval '' does today, i.e. running the entry URL. I feel like this could just ship on its own separate from the new flag, and land in 21.0.0? And it wouldn’t be backportable, but I don’t think it needs to be; people could just add --eval '' for Node <20. What do people think of this?

And just so we’re clear, it seems like we don’t want the first option, but are people asking for something other than the second option? I was planning to use --import for this to avoid the need to create yet another CLI flag; are people saying that they would prefer a new flag, like --entry-url?

I don't like the behaviour of not being able to enter the REPL with an --import flag. That seems like it would be confusing.

Personally I think what would make the most sense is an option to change how to interpret that first arg. So your --entry-url would just be a mode changer not a --entry-url=url form.

I don’t like the behaviour of not being able to enter the REPL with an --import flag. That seems like it would be confusing.

You would just need to pass -i or --interactive. Like node --import ./dependency.js?foo=bar --interactive would launch the REPL. This would a breaking change, but I'm not sure how many people it would affect.

Personally I think what would make the most sense is an option to change how to interpret that first arg. So your --entry-url would just be a mode changer not a --entry-url=url form.

I intended --entry-url to be a boolean that enables the new behavior for the parsing of the entry point. So like node --entry-url --enable-source-maps ./entry.js?foo=bar, where --entry-url doesn’t take a value.

The issue with --entry-url being a standalone option is that then it needs to work for CommonJS too, unless we just error if the entry is CommonJS?

The issue with --entry-url being a standalone option is that then it needs to work for CommonJS too, unless we just error if the entry is CommonJS?

It shouldn’t be an issue, the ESM loader is able to load CJS, why would we error?

It shouldn’t be an issue, the ESM loader is able to load CJS, why would we error?

I was thinking how would we handle something like node --entry-url ./entry.cjs?foo=bar but I guess you’re right, it would get handled however an import() of that would be handled currently, which I assume is along the lines of “convert the URL to a file path and import/require it.” So --entry-url would become another of those “if this is used, the user has opted into having the ESM loader handle the entry point” flags like --import and --loader.

Anyway getting back to the UX question first though, which do we want:

👀 --import is used for this case, like node --import ./entry.js, and if the user wants the REPL with --import they need to add -i or --interactive
❤️ --entry-url is a new boolean flag that causes the entry point string to be parsed as a URL and loaded by the ESM loader: node --entry-url --enable-source-maps ./entry.js?foo=bar

I know people are still debating the best approach for supporting URL entry points, but since one of the options under consideration is a semver-major change and the others aren’t, I opened a PR for the semver-major one in case that wins consensus: #49946. Let’s hopefully decide whether or not to go with that approach before the 21.0.0 cutoff on Tuesday.

I think it would be better to open spinoff issue so we can have consensus on more details.
For example, do we want to support relative URLs? Do we want to support bare specifiers? Do we want to support shortcut specifiers? Do we want to allow users to override mime-type? Should we support loading CJS with URL? Should we seek for pjsons for file: URLs?

For the UX question, it might be worth mentioning that third option: providing main entry URL as named argument.

As for when it should land, I see it as a feature tied to import.meta.main rather than feature tied to --default-type. If we are planning to have .main soon, I'd suggest to land this feature directly afterwards (if not in the same PR, assuming that import.meta.main isn't semver-major).
The reasons for that are that it's much harder (if possible at all) to test if it actually works as main entry point, and it would be unclear for users that there's difference between main entry point and import until we provide a mechanism to differentiate.

I think it would be better to open spinoff issue so we can have consensus on more details.

We can, but since the first PR implemented everything else I think the URL entry stuff is all that’s left to design, so maybe we can just finish it here?

For example, do we want to support relative URLs?

Yes. The input would be relative to the file URL of process.cwd().

Do we want to support bare specifiers?

I don’t think so. That’s what --import is for, or one could do node --entry-url data:text/javascript,import('bare-specifier').

Do we want to support shortcut specifiers?

What is a shortcut specifier?

Do we want to allow users to override mime-type?

From an HTTPS URL, you mean? I think not as part of the CLI, that can be achieved via a module customization hook.

Should we support loading CJS with URL? Should we seek for pjsons for file: URLs?

Yes and yes. It would be loaded by the ESM loader, the same as if you had referenced a relative or absolute file URL via import().

What is a shortcut specifier?

Subpath imports, the ones starting from #.

Do we want to allow users to override mime-type?

From an HTTPS URL, you mean? I think not as part of the CLI, that can be achieved via a module customization hook.

From any non-file: URL, to be precise.
For example, we could expand --input-type to enable node --input-type=commonjs --entry-url https://example.com/legacy.cjs and node --input-type=commonjs --entry-url 'data:text/javascript,console.log("am CJS");' (I don't think we should, but it's worth considering).

Subpath imports, the ones starting from #.

No, I think we shouldn’t implement the ESM resolution algorithm here. # would just mean a hash, like a URL.

From any non-file: URL, to be precise.

We already have --experimental-network-imports. The way data: and https: imports work currently is that Node uses the MIME types from the header of the resource. That can be overridden by the load module customization hook; I don’t think we want to provide another way to customize this.

ESBuild + TSC Demo

This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

Not stale

Not stale

Yes it is, the last comment was from almost 3 years ago. Maybe you meant "still relevant"?

I would argue that is no longer relevant, now that we have syntax detection enabled by default, I think the use-case is mostly fulfilled, and I don't think anyone is working on this. I don't see the point of keeping that issue open, closing as not planned.

Navigation Menu

Repository navigation

Discussion: New “ESM by default” mode #49432

Shebangs

Separate binaries

Breaking change

`argv`

`import.meta.url`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Discussion: New “ESM by default” mode #49432

Description

Activity

mcollina commented on Sep 1, 2023

GeoffreyBooth commented on Sep 1, 2023

mcollina commented on Sep 2, 2023

LiviaMedeiros commented on Sep 2, 2023

Shebangs

Separate binaries

Breaking change

mcollina commented on Sep 2, 2023

tniessen commented on Sep 2, 2023

bmeck commented on Sep 2, 2023

LiviaMedeiros commented on Sep 2, 2023

aduh95 commented on Sep 2, 2023

LiviaMedeiros commented on Sep 2, 2023

aduh95 commented on Sep 2, 2023

mcollina commented on Sep 2, 2023

LiviaMedeiros commented on Sep 2, 2023

GeoffreyBooth commented on Sep 2, 2023

bmeck commented on Sep 2, 2023

mcollina commented on Sep 3, 2023

GeoffreyBooth commented on Sep 4, 2023

muzuiget commented on Sep 4, 2023

Qard commented on Sep 4, 2023

GeoffreyBooth commented on Sep 4, 2023

Qard commented on Sep 4, 2023

ljharb commented on Sep 4, 2023

ljharb commented on Sep 4, 2023

GeoffreyBooth commented on Sep 5, 2023

ljharb commented on Sep 5, 2023

GeoffreyBooth commented on Sep 7, 2023

ljharb commented on Sep 7, 2023

LiviaMedeiros commented on Sep 8, 2023

GeoffreyBooth commented on Sep 8, 2023

mcollina commented on Sep 8, 2023

richardlau commented on Sep 8, 2023

Qard commented on Sep 9, 2023

LiviaMedeiros commented on Sep 9, 2023

GeoffreyBooth commented on Sep 9, 2023

LiviaMedeiros commented on Sep 9, 2023

GeoffreyBooth commented on Sep 9, 2023

LiviaMedeiros commented on Sep 9, 2023

GeoffreyBooth commented on Sep 9, 2023

LiviaMedeiros commented on Sep 10, 2023

GeoffreyBooth commented on Sep 10, 2023

LiviaMedeiros commented on Sep 10, 2023

GeoffreyBooth commented on Sep 10, 2023

GeoffreyBooth commented on Sep 11, 2023

argv

import.meta.url

GeoffreyBooth commented on Sep 11, 2023

LiviaMedeiros commented on Sep 11, 2023

GeoffreyBooth commented on Sep 11, 2023

LiviaMedeiros commented on Sep 11, 2023

GeoffreyBooth commented on Sep 11, 2023

LiviaMedeiros commented on Sep 11, 2023

GeoffreyBooth commented on Sep 11, 2023

LiviaMedeiros commented on Sep 12, 2023

GeoffreyBooth commented on Sep 12, 2023

aduh95 commented on Sep 12, 2023

GeoffreyBooth commented on Sep 12, 2023

aduh95 commented on Sep 12, 2023

GeoffreyBooth commented on Sep 12, 2023

GeoffreyBooth commented on Sep 12, 2023

elibarzilay commented on Sep 15, 2023

GeoffreyBooth commented on Sep 15, 2023

luxzeitlos commented on Sep 16, 2023

elibarzilay commented on Sep 16, 2023

ljharb commented on Sep 16, 2023

elibarzilay commented on Sep 16, 2023

ljharb commented on Sep 16, 2023

bmeck commented on Sep 16, 2023

GeoffreyBooth commented on Sep 17, 2023

elibarzilay commented on Sep 17, 2023

GeoffreyBooth commented on Sep 17, 2023

elibarzilay commented on Sep 17, 2023

GeoffreyBooth commented on Sep 17, 2023

GeoffreyBooth commented on Sep 25, 2023

aduh95 commented on Sep 25, 2023

`argv`

`import.meta.url`