A collection of utilities for assisting basic MediaWiki development tasks
These tools assume that there is a "mediawki" directory (DEV_MW_DIR), used directly by an IDE, for making changes and that this directory contains the following subdirectories:
- core: git checkout of mediawiki/core
- skins: directory of git checkouts of mediawiki/skin/*
- extensions: directory of git checkouts of mediawiki/extensions/*
These tools assume that there is another "mediawki" directory (SRV_MW_DIR), used by a webserver, for testing changes made in the IDE. This directory must be synchronized prior to each attempt to test functionality or run unit tests against it.
Set DEV_MW_DIR and SRV_MW_DIR in ~/.bashrc (or similar).
export DEV_MW_DIR=/home/aaron/PhpstormProjects/mediawiki
export SRV_MW_DIR=/srv/mediawiki
A set of programs useful for making WSL faster by avoiding 9P file system I/O. They work by deciding which binary to use based on the current working directory. They do not carefully inspect the program arguments to see what files are targeted. To avoid slowdown, avoid running them while the working directory is within an ntfs mount (e.g. /mnt/c/) when they need to operate on files inside a non-9P mount (e.g. an "ext4" mount in the WSL VM).
Some of the wrappers always uses the Linux version if an argument starts with /proc, /run, ect... The grep wrapper always uses the Linux version if the input is from a pipe (within WSL, launching an exe is slower than a Linux program).
- composer: uses the Win32 exe if applicable and available, and the Linux version otherwise
- find: uses the "Git for Windows" exe if applicable and available, and the Linux version otherwise
- git: uses the "Git for Windows" exe if applicable and available, and the Linux version otherwise
- git-review: uses the PIP3 Win32 exe if applicable and available, and the Linux version otherwise
- grep: uses the "Git for Windows" version if applicable and available, and the Linux version otherwise
These programs can be enabled by creating symlinks within your "~/bin" folder, e.g.:
mkdir -p ~/bin
ln -s /path/to/mediawiki-developer-tools/smart_utils/* ~/bin/
A set of programs that can be run from the "core" MediaWiki repo directory.
- mediawiki-phpunit: runs phpunit.php (directly or via paratest)
- dev_phpunit: wrapper; runs phpunit in the "development" directory
- dev_paratest: wrapper; runs paratest in the "development" directory
- srv_phpunit: wrapper; runs phpunit in the "server" directory
- srv_paratest: wrapper; runs paratest on the "server" directory
These programs can be enabled by creating symlinks within the "core" repo folder, e.g.:
cd /path/to/mediawiki/core
ln -s /path/to/mediawiki-developer-tools/core_utils/* .
The test runners depends on paratest, which can be installed via:
git clone https://github.com/paratestphp/paratest.git
cd paratest
git checkout 5.0.4
sed -i.bak 's/"php": "^/"php": ">=/' composer.json
sed -i.bak 's/phpstan-banned-code": "^/phpstan-banned-code": ">=/' composer.json
composer.phar update
ln -s $(realpath ./bin/paratest) ~/bin/paratest
A set of programs that can be run from either an "extensions" directory containing MediaWiki extension repos, or, a "skins" directory containing MediaWiki skin repos. These directories should not be the same directories as the ones that appears under the "core" repo. The proper place for these directories is under the "mediawiki" directory.
These programs can be enabled by creating symlinks within this extension folder, e.g.:
cd /path/to/mediawiki/extensions
ln -s /path/to/mediawiki-developer-tools/extension_utils/* .
cd /path/to/mediawiki/skins
ln -s /path/to/mediawiki-developer-tools/extension_utils/* .
Some git/git-review utilities are included for doing mass operations.
- git-all: use this to perform an action on all Gerrit-hosted extensions/skins for the current folder
- git-wmf: use this to perform an action on all Wikimedia deployed, Gerrit-hosted, extensions/skins for current folder
The mass repo actions include:
- sync: clone if nonexistant, stash any changes, update origin, hard reset to origin/master, and recursively update/init submodules
- update: clone and recursively update/init submodules if nonexistant, and update origin
- uncommitted: show diff bewteen upstream and the working tree (includes staged changes)
- commit: commit all changed files (tracked) and staged files (preferably added via the IDE)
- unpushed: show all commits after upstream
- push-direct: push the changes to the remote and hard reset to master
- push-review: push the changes for Gerrit review and hard reset to master
- reset: hard reset to master
- gc: garbage collect
Note: At least when using Windows, it's best to set SSH connection keep-alive to avoid TCP connection spam in TIME_WAIT that can lead to exhaustion. E.g.:
Host gerrit.wikimedia.org
KeepAlive yes
ServerAliveInterval 60
Alternatively, using pageant and setting git to use plink.exe via core.sshCommand also works for sharing TCP connections. Hardcoding the username in core.sshCommand means that SSH repo origin URLs do not need your username in them. Note that GitHub SSH URLs require the username "git", so be careful when setting core.sshCommand with --global.
git config core.sshCommand = "plink.exe -ssh -l YOUR_SSH_USERNAME -share -agent"
MediaWiki-Docker currently assumes that extensions are checked out under the extensions/ subdirectory of the mediawiki core directory rather than a sibling extensions/ directory. Likewise for skins and the skins/ directory. Making it work this way requires a few extra things.
In order for docker to work, it needs to be launched from the mediawiki directory, so that the container mounts a filesystem with access to the core, extensions, and skins directory. However, the MediaWiki-included docker-compose.yml file is under core. To work around this, go to the mediawiki directory, and create a symlink to this file:
ln -s core/docker-compose.yml docker-compose.yml
The mediawiki-jobrunner image expects that core will be the working directory for it's ENTRYPOINT, not mediawiki. Also, the mediawiki image is used for maintenance scripts, which often aspect core to be the working directory. To work around this, in the mediawiki directory, create a docker-compose.override.yml file with:
services:
mediawiki:
environment:
MW_INSTALL_PATH: /var/www/html/w/core
working_dir: /var/www/html/w/core
mediawiki-jobrunner:
environment:
MW_INSTALL_PATH: /var/www/html/w/core
working_dir: /var/www/html/w/core
If that file already exists, then just merge these values into the yaml by hand.
Some grep wrappers are included for searching MediaWiki extension repos, of which there are 1000+. The must be run in a folder called "extensions" that contains the extension repos.
- grep-all: searches all extension repos and groups output by repo
- grep-wmf: searches only extensions enabled on Wikimedia productions sites and groups output by repo
- grep-nonwmf: searches only extensions not enabled on Wikimedia productions sites and groups output by repo
A set of scripts are included for quickly setting up replication on a single machine using mariadb.
- setup_multi_mariadb: setup db1, db2, db3, db4 instances using the mariadb@ SystemD template
- enable_replicated_multi_mariadb: make the db2,db3,db4 instances replicate from db1
- import_stock_mariadb: import stock mariadb instance data into the db1 instance
These scripts can be found in the /maria subdirectory.
A set of programs that can be run from anywhere.
- sync-srv-mediawiki: This is a fast git/rsync based script to syncronize the MediaWiki directory used for serving content with the one used by your IDE(s) for development. This is mostly useful when the server runs inside WSL (with native file system access) but the IDE runs as a Win32 app that needs to heavily watch, stat, and read thousands of files (requiring native file system access).
- mmariadb: Run mariadb, with the first argument passed as the --defaults-group-suffix option
These programs can be enabled by creating symlinks within the "~/bin" folder, e.g.:
mkdir -p ~/bin
ln -s /path/to/mediawiki-developer-tools/utils/* ~/bin/
The bash_aliases script contains convenient shell terminal aliases for running common commands, e.g:
- YubiKey operations
- Git operations
- git-review operations
- Synchronizing the MediaWiki "server" directory to match the "development" directory
They can be enabled by editing ~/.bashrc, e.g.:
source /path/to/mediawiki-developer-tools/bash_aliases