Fix Dub package when cross-compiling

[jacob-carlborg]

There are two commits, I recommend reviewing them separately.

[dlang-bot]

Thanks for your pull request and interest in making D better, @jacob-carlborg! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the annotated coverage diff directly on GitHub with CodeCov's browser extension
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + dmd#9275"

[WalterBright]

There are two commits, I recommend reviewing them separately.

Please do them as 2 PRs, then.

[jacob-carlborg]

I don't see the problem of looking at the individual commits.

[thewilsonator]

DAutoTest

Running pre-generate commands for dmd:lexer...
/bin/sh: line 2: : command not found
Command failed with exit code 127: 
    "${DUB_EXE}" \
    --arch=${DUB_ARCH} \
    --single "${DUB_PACKAGE_DIR}config.d" \
    -- "${DUB_PACKAGE_DIR}generated/dub" \
    "${DUB_PACKAGE_DIR}VERSION" \
    /etc

Say what?

[jacob-carlborg]

Say what?

Hmm, I didn't know that DAutoTest was building the Dub package. But for Semaphore I instead get:

/bin/sh: 2: : Permission denied
Command failed with exit code 127: 
    "${DUB_EXE}" \
    --arch=${DUB_ARCH} \
    --single "${DUB_PACKAGE_DIR}config.d" \
    -- "${DUB_PACKAGE_DIR}generated/dub" \
    "${DUB_PACKAGE_DIR}VERSION" \
    /etc

[jacob-carlborg]

Say what?

Turns out that the $DUB_EXE variable was added in the release shipped with DMD 2.084.0 [1] (and the changelog is wrong). DAutoTest is using Dub from DMD 2.083.0. Can we update that?

[1] https://dlang.org/changelog/2.084.0.html#dubEnvVar

[jacob-carlborg]

BTW, this blocks: #8528.

[thewilsonator]

Can we update that?

cc @CyberShadow

[CyberShadow]

Can we update that?

I think it's coming from here:
https://github.com/dlang/dlang.org/blob/master/posix.mak#L183

[jacob-carlborg]

@thewilsonator @CyberShadow dlang/dlang.org#2557.

[CyberShadow]

Unless it is known for a fact that $DUB_EXE will not match $(which dub), I suggest using "$${DUB_EXE:-dub}" to allow this change to work with older versions of Dub.

[jacob-carlborg]

Unless it is known for a fact that $DUB_EXE will not match $(which dub)

If dub is not in the PATH then which dub will not work. But $DUB_EXE will, it uses thisExePath.

[CyberShadow]

If dub is not in the PATH then which dub will not work.

That's fine. If that's your only concern, I recommend that you apply the change I suggested above.

[jacob-carlborg]

Any equivalent for Windows?

[CyberShadow]

Not in its variable substitution syntax, as far as I know, so only as a separate command. Not updating the Windows command only means that Windows users will have no choice but to update to 2.084, which doesn't seem like a problem.

[jacob-carlborg]

I got this error following your advice:

/bin/sh: line 2: 29755{DUB_EXE:-dub}: command not found

[CyberShadow]

And $${DUB_EXE} (with two $) works? Is that some special Dub syntax? There should probably be only one $, with :-dub or without.

[jacob-carlborg]

And $${DUB_EXE} (with two $$) works? Is that some special Dub syntax?

Yes, it's special Dub syntax. Some variables will be substituted before the command is run.

[CyberShadow]

I can't find the implementation for parsing that syntax in Dub's source code. It looks like it's just using spawnShell, and everything points to that those variables are expanded by the shell, not Dub. Are you sure there are supposed to be two $?

Edit: Found something, looking into it.

[jacob-carlborg]

I can't find the implementation for parsing that syntax in Dub's source code. It looks like it's just using spawnShell, and everything points to that those variables are expanded by the shell, not Dub.

It must be doing that before calling spawnShell.

Are you sure there are supposed to be two $?

Yes, otherwise I get Invalid variable: DUB_EXE. See https://dub.pm/package-format-sdl.html#environment-variables: "To denote a literal dollar sign, use $$.".

[CyberShadow]

Oh. Looks like it's a Dub bug. Look at this function:

https://github.com/dlang/dub/blob/927c5e6c248f7e2098a303fd475decfd8d8cca0a/source/dub/project.d#L1197-L1211

Instead of looking for $ (minding escapes), it uses a regular expression to match what it thinks shell variable use looks like. The code looks wrong and overcomplicated, I guess I'll work on a fix.

Never mind my suggestion I guess...

[jacob-carlborg]

The issue comes from the fact that Dub uses the same variable syntax as Bash. On Windows [1] it's not a problem because the shell on Windows uses % instead of $.

[1] https://github.com/dlang/dmd/pull/9275/files#diff-406ece895a5b70a272cd76a2ef902836R43

[CyberShadow]

Yes, but that doesn't change what I wrote above. Its escaping logic is wrong.

[jacob-carlborg]

Ok.

[CyberShadow]

I guess I'll work on a fix.

Here it is: dlang/dub#1641

[PetarKirov]

@jacob-carlborg In order for the tests to pass on SemaphoreCI, replace

dmd/semaphoreci.sh

Lines 39 to 48 in 54a2a5a

	# FIXME: v2.082.0 has a broken DUB which fails the CI
	# Remove this when a fixed v2.082.1 is released
	# See https://github.com/dlang/dub/issues/1551
	if [ "$DMD" == "dmd" ]; then
	install_d "dmd-2.081.2"
	elif [ "$DMD" == "ldc" ]; then
	install_d "ldc-1.11.0"
	else
	install_d "$DMD"
	fi

with:

install_d "$DMD"

[jacob-carlborg]

@thewilsonator @ZombineDev all tests are passing, finally 😃.

[thewilsonator]

Nice I take it it is goo to go?

[jacob-carlborg]

Nice I take it it is goo to go?

Yes. I had to download Dub separately on Semaphore when using LDC 1.11.0. Because the version of Dub shipped with LDC 1.11.0 is too old and doesn't support the DUB_EXE environment variable. Using a later version of LDC cause some linker error.

[thewilsonator]

Can you report the linker error please?

[jacob-carlborg]

Sure.

[thewilsonator]

Thanks.

[jacob-carlborg]

Has already been reported: ldc-developers/ldc#2954.