diff --git a/.gitignore b/.gitignore index ba4d11c..a14d7e6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,9 +1,10 @@ *.pyc *~ *.swp -coverage/* -build/* +*.egg-info +coverage*/ +build/ mydatabase -docs/_build/* +docs/_build/ MANIFEST dist/ diff --git a/CHANGES.rst b/CHANGES.rst new file mode 100644 index 0000000..4f7eb73 --- /dev/null +++ b/CHANGES.rst @@ -0,0 +1,66 @@ +CHANGES +======= + +2.0 (unreleased) +------------------ + +- **Backward incompatible:** ``env_settings.py`` is required and contains + the settings that ``fabfile.py`` used to have. It is written in the normal + Django settings.py style of variable assignments. + +- **Backward incompatible:** ``tasks.py`` is required and replaces the + previously required ``fabfile.py``. It no longer defines Django settings + (now handled in ``env_settings.py``). + +- **Backward incompatible:** switched from Fabric to Invoke, so the ``fab`` + command is replaced with ``invoke``. + +- **Backward incompatible:** Lettuce testing support is gone. + +- **Backward incompatible:** the Sphinx documentation command ``docs`` is gone. + +- **Backward incompatible:** remove ``assertInContext``, ``assertNone``, + ``assertIsA`` and ``assertDoesNotHave``. Some of these duplicated native + Python methods and the potential confusion was greater than the benefit. + +- **Backward incompatible:** individual components need to specify the + ``fudge`` requirement if they need it. + +- **Backward incompatible:** Fix ``generate_random_users()`` and turn it + into a generator. + +- **Backward incompatible:** remove ``concrete`` decorators. The Armstrong + standard practice is to use "support" models when necessary in testing, + which are much easier to use and understand. + +- Setuptools is explictly used. This is not a backwards incompatible change + because anything installed with Pip was automatically and transparently + using Setuptools/Distribute *anyway*. We rely on setup kwargs that Distutils + doesn't support and that only worked because of Pip's behind the scenes swap. + This allowed us to remove boilerplate and better prepares us for Python 3 + and perhaps even more simplifying refactors. Functionally though, this + doesn't change anything. + +- Drop the atypical VirtualDjango in favor of the ``settings.configure()`` + Django bootstrapping method. + +- Bare minimum package requirements for as-fast-as-possible virtualenv + creation. Even Invoke is optional when running tests. Individual tasks + can specify package requirements and will nicely message their needs if + the package is not installed. + +- Run any Django ``manage.py`` command from import or the CLI with + component-specific settings bootstrapped in. + +- Run tests with arguments. Use any args that ``manage.py test`` accepts + to run only specific test cases, change output verbosity, etc. + +- Coverage testing is ready for multiple environments at once (like with Tox). + +- Use (and backport) the Django 1.6 test runner. This standardizes testing + in favor of the newest method so we don't need to be cognizant of the current + Django version as we test across multiple versions. Bonus: because the new + runner is explicit about test discovery, drop the ``TESTED_APP`` code. + +- New ``remove_armstrong`` task command to uninstall every Armstrong component + (except for ArmDev). diff --git a/MANIFEST.in b/MANIFEST.in index 1ab1acc..a47dcd7 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,4 +1,5 @@ +include LICENSE include README.rst +include CHANGES.rst include package.json -include armstrong/cli/templates/standard/requirements/*.txt prune build/* diff --git a/README.rst b/README.rst index 492ce0b..6841b8b 100644 --- a/README.rst +++ b/README.rst @@ -3,78 +3,184 @@ armstrong.dev Tools and such for handling development of Armstrong applications This package contains some of the various helpers needed to do development work -on the Armstrong packages. If you're not actively developing, or working with +on the Armstrong packages. If you're not actively developing, or working with development versions of Armstrong, you probably don't need this package. + +Installation & Configuration +---------------------------- +If you are just running tests for a component, Tox will grab everything it +needs including ArmDev. + +- ``pip install tox`` and run ``tox`` + +Otherwise: + +- ``pip install armstrong.dev invoke`` + +`Invoke`_ is not strictly required. ArmDev is as lean as possible to support +fast virtualenv creation so multi-environment testing tools like TravisCI +and Tox will complete ASAP. + +Many of the Invoke tasks have their own package requirements and they will +nicely notify you if something they require needs to be installed. + +**Optional Settings:** (Used in ``env_settings.py``) + +``COVERAGE_EXCLUDE_FILES = ['*/migrations/*']`` + A list of filename patterns for files to exclude during coverage testing. + Individual components are free to extend or replace this setting. + +.. _Invoke: http://docs.pyinvoke.org/en/latest/index.html + + Usage ----- +Most Armstrong components already have the necessary configuration to use these +Dev tools. Specifically, components need ``tasks.py`` and ``env_settings.py`` +files. Assuming these are present: -Create a `fabfile` (either `fabfile/__init__.py` or simply `fabfile.py`) in -your project and add the following:: +``invoke --list`` + to see a list of all available commands - from armstrong.dev.tasks import * +``invoke --help `` + for help on a specific command +Several of the tasks take an optional ``--extra`` argument that is used as a +catch-all way of passing arbitrary arguments to the underlying command. Invoke +cannot handle arbitrary args (like Fabric 1.x could) so this is our workaround. +Two general rules: 1) enclose multiple args in quotes 2) kwargs need to use +"=" with no spaces (our limitation, not Invoke's). Example: +``invoke test --extra "--verbosity=2 "`` - settings = { - "DEBUG": True, - # And so on with the keys being the name of the setting and the values - # the appropriate value. - } +``invoke install [--editable]`` + to "pip install" the component, by default as an `editable`_ install. For + a regular install, use ``--no-editable`` or ``--editable=False``. - main_app = "name.of.your.app" - tested_apps ("another_app", main_app, ) +``invoke test [--extra ...]`` + to run tests where --extra handles anything the normal Django + "manage.py test" command accepts. +``invoke coverage [--reportdir=] [--extra ...]`` + for running test coverage. --extra works the same as in "invoke test" passing + arbitrary args to the underlying test command. --reportdir is where the HTML + report will be created; by default this directory is named "coverage". -Now your fabfile will expose the various commands for setting up and running -your reusable app inside a virtualenv for testing, interacting with via the -shell, and even running a simple server. +``invoke managepy [--extra ...]`` + to run any Django "manage.py" command where --extra handles any arbitrary + args. Example: ``invoke managepy shell`` or + ``invoke managepy runserver --extra 9001`` -Type ``fab -l`` to see a list of all of the commands. +``invoke create_migration [--initial]`` + to create a South migration for the component. An "auto" migration is + default if the --initial flag is not used. +There are other commands as well, but these are the most useful. Remember +that individual components may provide additional Invoke tasks as well. So +run ``invoke --list`` to discover them all. -Installation ------------- -:: +.. _editable: http://pip.readthedocs.org/en/latest/reference/pip_install.html#editable-installs - name="armstrong.dev" - pip install -e git://github.com/armstrong/$name#egg=$name -**Note**: This currently relies on a development version of Fabric. This -requirement is set to be dropped once Fabric 1.1 is released. To ensure this -runs as expected, install the ``tswicegood/fabric`` fork of Fabric: +Component Setup +--------------- +If you are creating a new Armstrong component or updating one that uses the +pre-2.0 ArmDev, you'll need to create (or port to) these two files: -:: +1. Create a ``tasks.py`` and add the following:: - pip install -e git://github.com/tswicegood/fabric.git#egg=fabric + from armstrong.dev.tasks import * + # any additional Invoke commands + # ... -Contributing ------------- +2. Create an ``env_settings.py`` and add the following:: -* Create something awesome -- make the code better, add some functionality, - whatever (this is the hardest part). -* `Fork it`_ -* Create a topic branch to house your changes -* Get all of your commits in the new topic branch -* Submit a `pull request`_ + from armstrong.dev.default_settings import * + # any additional settings + # it's likely you'll need to extend the list of INSTALLED_APPS + # ... -License -------- -Copyright 2011 Bay Citizen and Texas Tribune +Not required but as long as you are reviewing the general state of things, +take care of these too! -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at +- Review the ``requirements`` files +- Review the TravisCI configuration +- Drop Lettuce tests and requirements +- Add a ``tox.ini`` file +- Review the README text and setup.py metadata +- Use Setuptools and fix any improper namespacing +- Stop shipping tests by moving tests/ to the root directory +- If the component uses logging, consider namespacing it with + ``logger = logging.getLogger(__name__)``. +- Add a ``CHANGES.rst`` file and include it in the MANIFEST +- Review ``.gitignore``. You might want to ignore these:: - http://www.apache.org/licenses/LICENSE-2.0 + .tox/ + coverage*/ + *.egg-info -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. +Notable changes in 2.0 +---------------------- +Setuptools is now explicitly used/required instead of Distutils. + +Invoke replaces Fabric for a leaner install without the SSH and crypto +stuff. Invoke is still pre-1.0 release so we might have some adjustment +to do later. + +This version offers an easier and more standard way to run a Django +environment with a component's specific settings, either from the +commandline or via import. + +It provides an "a la carte" requirements approach. Meaning that if you run an +Invoke command that needs a package that isn't installed, it will prompt you +to install it instead of requiring everything up-front. This allows for much +faster virtualenv creation (which saves considerable time in testing) and +doesn't pollute your virtualenv with packages for features you don't use. + +``test`` and ``coverage`` will work better with automated test tools like +TravisCI and Tox. These commands also now work like Django's native test +command so that you can pass arguments for running selective tests or +changing the output verbosity. + +Settings are now defined in the normal Django style in an ``env_settings.py`` +file instead of as a dict within the tasks file. It's not called "settings.py" +to make it clearer that these are settings for the development and testing +of this component, not necessarily values to copy/paste for incorporating +the component into other projects. + +The full list of changes and backward incompatibilties is available +in **CHANGES.rst**. + + +Contributing +------------ +Development occurs on Github. Participation is welcome! + +* Found a bug? File it on `Github Issues`_. Include as much detail as you + can and make sure to list the specific component since we use a centralized, + project-wide issue tracker. +* Have code to submit? Fork the repo, consolidate your changes on a topic + branch and create a `pull request`_. +* Questions, need help, discussion? Use our `Google Group`_ mailing list. + +.. _Github Issues: https://github.com/armstrong/armstrong/issues .. _pull request: http://help.github.com/pull-requests/ -.. _Fork it: http://help.github.com/forking/ +.. _Google Group: http://groups.google.com/group/armstrongcms + + +State of Project +---------------- +`Armstrong`_ is an open-source news platform that is freely available to any +organization. It is the result of a collaboration between the `Texas Tribune`_ +and `The Center for Investigative Reporting`_ and a grant from the +`John S. and James L. Knight Foundation`_. Armstrong is available as a +complete bundle and as individual, stand-alone components. + +.. _Armstrong: http://www.armstrongcms.org/ +.. _Texas Tribune: http://www.texastribune.org/ +.. _The Center for Investigative Reporting: http://cironline.org/ +.. _John S. and James L. Knight Foundation: http://www.knightfoundation.org/ diff --git a/armstrong/__init__.py b/armstrong/__init__.py index 3ad9513..de40ea7 100644 --- a/armstrong/__init__.py +++ b/armstrong/__init__.py @@ -1,2 +1 @@ -from pkgutil import extend_path -__path__ = extend_path(__path__, __name__) +__import__('pkg_resources').declare_namespace(__name__) diff --git a/armstrong/dev/__init__.py b/armstrong/dev/__init__.py index 3ad9513..e69de29 100644 --- a/armstrong/dev/__init__.py +++ b/armstrong/dev/__init__.py @@ -1,2 +0,0 @@ -from pkgutil import extend_path -__path__ = extend_path(__path__, __name__) diff --git a/armstrong/dev/default_settings.py b/armstrong/dev/default_settings.py new file mode 100644 index 0000000..19ead69 --- /dev/null +++ b/armstrong/dev/default_settings.py @@ -0,0 +1,53 @@ +""" +Default settings for Armstrong components running in a dev/test environment + +A component may (and might have to) override or supply additional settings +by creating an `env_settings.py` file in its root directory that imports +from this file. + + from armstrong.dev.default_settings import * + +""" +# Since we are using configure() we need to manually load the defaults +from django.conf.global_settings import * + +# Grab our package information +import json +package = json.load(open("./package.json")) +app_name = package['name'].rsplit('.', 1)[1] + +# +# Armstrong default settings +# +DEBUG = True +INSTALLED_APPS = [package['name']] +DATABASES = { + "default": { + "ENGINE": "django.db.backends.sqlite3", + "NAME": 'mydatabase' + } +} +TEST_RUNNER = "armstrong.dev.tests.runner.ArmstrongDiscoverRunner" + +COVERAGE_EXCLUDE_FILES = ['*/migrations/*'] + +# Add a DEBUG console "armstrong" logger +LOGGING = { + 'version': 1, + 'disable_existing_loggers': False, + 'formatters': { + 'basic': {'format': '%(levelname)s %(name)s--%(message)s'} + }, + 'handlers': { + 'console': { + 'class': 'logging.StreamHandler', + 'formatter': 'basic' + } + }, + 'loggers': { + 'armstrong': { + 'level': 'DEBUG', + 'handlers': ['console'] + } + } +} diff --git a/armstrong/dev/dev_django.py b/armstrong/dev/dev_django.py new file mode 100644 index 0000000..9822938 --- /dev/null +++ b/armstrong/dev/dev_django.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python +import sys +import threading +from functools import wraps + + +__all__ = ['run_django_cmd', 'run_django_cli', 'DjangoSettings'] + + +class DjangoSettings(object): + """ + Isolate settings import so it doesn't happen on module import. + + Not all of our tasks need Django so we only want to build up + our component environment's settings if and when they are needed. + This approach avoids unnecessary warnings if the settings aren't + available or Django isn't installed. + + Do this as a singleton to avoid trying our imports and running + configure() over and over, but note that the object returned is + still the typical ``django.conf.LazySettings`` and so settings + remain mutable. + + """ + _singleton_lock = threading.Lock() + _instance = None + + def __new__(cls, *args, **kwargs): + """Threadsafe singleton loading of settings""" + + if not cls._instance: + with cls._singleton_lock: + if not cls._instance: + cls._instance = cls.load_settings() + return cls._instance + + @staticmethod + def load_settings(): + try: + from django.conf import settings + except ImportError as e: + raise ImportError( + "%s. Check to see if Django is installed in your " + "virtualenv." % e) + + try: + import env_settings as package_settings + except ImportError as e: + print( + "Could not find component specific settings file. " + "Using armstrong.dev defaults...") + try: + from . import default_settings as package_settings + except ImportError as e: + raise ImportError( + "%s. Running a Django environment for this component " + "requires either an `env_settings.py` file or " + "`armstrong.dev.default_settings.py`." % e) + + # Setup the Django environment + if not settings.configured: + settings.configure(default_settings=package_settings) + + return settings + + +def load_django_settings(func): + @wraps(func) + def wrapper(*args, **kwargs): + DjangoSettings() + return func(*args, **kwargs) + return wrapper + + +# Import access +@load_django_settings +def run_django_cmd(cmd, *args, **kwargs): + from django.core.management import call_command + return call_command(cmd, *args, **kwargs) + + +# Commandline access +@load_django_settings +def run_django_cli(argv=None): + argv = argv or sys.argv + + from django.core.management import execute_from_command_line + execute_from_command_line(argv) + + +if __name__ == "__main__": + run_django_cli() diff --git a/armstrong/dev/tasks.py b/armstrong/dev/tasks.py new file mode 100644 index 0000000..6396f4e --- /dev/null +++ b/armstrong/dev/tasks.py @@ -0,0 +1,177 @@ +import sys +from os.path import dirname +from contextlib import contextmanager + +try: + from invoke import task, run +except ImportError: + sys.stderr.write("Tasks require Invoke: `pip install invoke`\n") + sys.exit(1) + +# Decorator keeps the function signature and argspec intact, which we +# need so @task can build out CLI arguments properly +from decorator import decorator + +from .dev_django import run_django_cmd, run_django_cli, DjangoSettings + + +__all__ = [ + "clean", "create_migration", "pep8", "managepy", + "coverage", "test", "install", "remove_armstrong"] + + +# Grab our package information +import json +package = json.load(open("./package.json")) + + +HELP_TEXT_MANAGEPY = 'any command that `manage.py` normally takes, including "help"' +HELP_TEXT_EXTRA = 'include any arguments this method can normally take. ' \ + 'multiple args need quotes, e.g. --extra "test1 test2 --verbosity=2"' +HELP_TEXT_REPORTS = 'directory to store coverage reports, default: "coverage"' + + +@decorator +def require_self(func, *args, **kwargs): + """Decorator to require that this component be installed""" + + try: + __import__(package['name']) + except ImportError: + sys.stderr.write( + "This component needs to be installed first. Run " + + "`invoke install`\n") + sys.exit(1) + return func(*args, **kwargs) + + +def require_pip_module(module): + """Decorator to check for a module and helpfully exit if it's not found""" + + def wrapper(func, *args, **kwargs): + try: + __import__(module) + except ImportError: + sys.stderr.write( + "`pip install %s` to enable this feature\n" % module) + sys.exit(1) + else: + return func(*args, **kwargs) + return decorator(wrapper) + + +@contextmanager +def html_coverage_report(report_directory=None): + package_parent = str(package['name'].rsplit('.', 1)[0]) # fromlist can't handle unicode + module = __import__(package['name'], fromlist=[package_parent]) + base_path = dirname(module.__file__) + + settings = DjangoSettings() + omit = getattr(settings, 'COVERAGE_EXCLUDE_FILES', None) + + import coverage as coverage_api + print("Coverage is covering: %s" % base_path) + cov = coverage_api.coverage(branch=True, source=[base_path], omit=omit) + + cov.start() + yield + cov.stop() + + # Write results + report_directory = report_directory or "coverage" + run('rm -rf ' + report_directory) + cov.html_report(directory=report_directory) + print("Coverage reports available in: %s " % report_directory) + + +@task +def clean(): + """Find and remove all .pyc and .pyo files""" + run('find . -name "*.py[co]" -exec rm {} \;') + + +@task +@require_self +@require_pip_module('south') +def create_migration(initial=False): + """Create a South migration for this project""" + + settings = DjangoSettings() + if 'south' not in (name.lower() for name in settings.INSTALLED_APPS): + print("Temporarily adding 'south' into INSTALLED_APPS.") + settings.INSTALLED_APPS.append('south') + + kwargs = dict(initial=True) if initial else dict(auto=True) + run_django_cmd('schemamigration', package['name'], **kwargs) + + +@task +@require_pip_module('pep8') +def pep8(): + """Run pep8 on all .py files in ./armstrong""" + run('find ./armstrong -name "*.py" | xargs pep8 --repeat') + + +@task(help=dict(extra=HELP_TEXT_EXTRA)) +@require_self +def test(extra=None): + """Test this component via `manage.py test`""" + return managepy('test', extra) + + +@task(help=dict(reportdir=HELP_TEXT_REPORTS, extra=HELP_TEXT_EXTRA)) +@require_self +@require_pip_module('coverage') +def coverage(reportdir=None, extra=None): + """Test this project with coverage reports""" + + try: + with html_coverage_report(reportdir): + return test(extra) + except (ImportError, EnvironmentError): + sys.exit(1) + + +@task(help=dict(cmd=HELP_TEXT_MANAGEPY, extra=HELP_TEXT_EXTRA)) +def managepy(cmd, extra=None): + """Run manage.py using this component's specific Django settings""" + + extra = extra.split() if extra else [] + run_django_cli(['invoke', cmd] + extra) + + +@task +def install(editable=True): + """Install this component (or remove and reinstall)""" + + try: + __import__(package['name']) + except ImportError: + pass + else: + run("pip uninstall --quiet -y %s" % package['name'], warn=True) + + cmd = "pip install --quiet " + cmd += "-e ." if editable else "." + + run(cmd, warn=True) + + +@task +def remove_armstrong(): + """Remove all Armstrong components (except for Dev) from this environment""" + + from pip.util import get_installed_distributions + pkgs = get_installed_distributions(local_only=True, include_editables=True) + apps = [pkg for pkg in pkgs + if pkg.key.startswith('armstrong') and pkg.key != 'armstrong.dev'] + + for app in apps: + run("pip uninstall -y %s" % app.key) + + if apps: + print( + "Note: this hasn't removed other dependencies installed by " + "these components. There's no substitute for a fresh virtualenv.") + else: + print("No Armstrong components to remove.") diff --git a/armstrong/dev/tasks/__init__.py b/armstrong/dev/tasks/__init__.py deleted file mode 100644 index d955ae4..0000000 --- a/armstrong/dev/tasks/__init__.py +++ /dev/null @@ -1,207 +0,0 @@ - -from pkgutil import extend_path -__path__ = extend_path(__path__, __name__) - - -from contextlib import contextmanager -try: - import coverage as coverage -except ImportError: - coverage = False -import os -from os.path import basename, dirname -import sys -from functools import wraps -import unittest - -import json - -from fabric.api import * -from fabric.colors import red -from fabric.decorators import task - -from armstrong.dev.virtualdjango.test_runner import run_tests as run_django_tests -from armstrong.dev.virtualdjango.base import VirtualDjango -from django.core.exceptions import ImproperlyConfigured - -if not "fabfile" in sys.modules: - sys.stderr.write("This expects to have a 'fabfile' module\n") - sys.stderr.write(-1) -fabfile = sys.modules["fabfile"] - - -FABRIC_TASK_MODULE = True - - -__all__ = ["clean", "command", "create_migration", "docs", "pep8", "test", - "reinstall", "runserver", "shell", "spec", "syncdb", ] - -def pip_install(func): - @wraps(func) - def inner(*args, **kwargs): - if getattr(fabfile, "pip_install_first", True): - with settings(warn_only=True): - if not os.environ.get("SKIP_INSTALL", False): - local("pip uninstall -y %s" % get_full_name(), capture=False) - local("pip install .", capture=False) - func(*args, **kwargs) - return inner - -@contextmanager -def html_coverage_report(directory="./coverage"): - # This relies on this being run from within a directory named the same as - # the repository on GitHub. It's fragile, but for our purposes, it works. - run_coverage = coverage - if run_coverage and os.environ.get("SKIP_COVERAGE", False): - run_coverage = False - - if run_coverage: - local('rm -rf ' + directory) - package = __import__('site') - base_path = dirname(package.__file__) + '/site-packages/' + get_full_name().replace('.', '/') - print "Coverage is covering: " + base_path - cov = coverage.coverage(branch=True, - source=(base_path,), - omit=('*/migrations/*',)) - cov.start() - yield - - if run_coverage: - cov.stop() - cov.html_report(directory=directory) - else: - print "Install coverage.py to measure test coverage" - - -@task -def clean(): - """Find and remove all .pyc and .pyo files""" - local('find . -name "*.py[co]" -exec rm {} \;') - - -@task -def create_migration(name, initial=False, auto=True): - """Create a South migration for app""" - command((("schemamigration", fabfile.main_app, name), { - "initial": bool(int(initial)), - "auto": bool(int(auto)), - })) - - -@task -def command(*cmds): - """Run and arbitrary set of Django commands""" - runner = VirtualDjango() - runner.run(fabfile.settings) - for cmd in cmds: - if type(cmd) is tuple: - args, kwargs = cmd - else: - args = (cmd, ) - kwargs = {} - runner.call_command(*args, **kwargs) - - -@task -def pep8(): - """Run pep8 on all .py files in ./armstrong""" - local('find ./armstrong -name "*.py" | xargs pep8 --repeat', capture=False) - - -@task -@pip_install -def test(): - """Run tests against `tested_apps`""" - from types import FunctionType - if hasattr(fabfile, 'settings') and type(fabfile.settings) is not FunctionType: - with html_coverage_report(): - run_django_tests(fabfile.settings, *fabfile.tested_apps) - return - else: - test_module = "%s.tests" % get_full_name() - try: - __import__(test_module) - tests = sys.modules[test_module] - except ImportError: - tests = False - pass - - if tests: - test_suite = getattr(tests, "suite", False) - if test_suite: - with html_coverage_report(): - unittest.TextTestRunner().run(test_suite) - return - - raise ImproperlyConfigured( - "Unable to find tests to run. Please see armstrong.dev README." - ) - - -@task -def runserver(): - """Create a Django development server""" - command("runserver") - - -@task -def shell(): - """Launch shell with same settings as test and runserver""" - command("shell") - - -@task -def syncdb(): - """Call syncdb and migrate on project""" - command("syncdb", "migrate") - - -@task -def docs(): - """Generate the Sphinx docs for this project""" - local("cd docs && make html") - - -@task -@pip_install -def spec(verbosity=4): - """Run harvest to run all of the Lettuce specs""" - defaults = {"DATABASES": { - "default": { - "ENGINE": "django.db.backends.sqlite3", - "NAME": ":memory:", - }, - }} - - get_full_name() - defaults.update(fabfile.settings) - v = VirtualDjango() - v.run(defaults) - v.call_command("syncdb", interactive=False) - v.call_command("migrate") - v.call_command("harvest", apps=fabfile.full_name, - verbosity=verbosity) - -def get_full_name(): - if not hasattr(fabfile, "full_name"): - try: - package_string = local("cat ./package.json", capture=True) - package_obj = json.loads(package_string) - fabfile.full_name = package_obj['name'] - return fabfile.full_name - except: - sys.stderr.write("\n".join([ - red("No `full_name` variable detected in your fabfile!"), - red("Please set `full_name` to the app's full module"), - red("Additionally, we couldn't read name from package.json"), - "" - ])) - sys.stderr.flush() - sys.exit(1) - return fabfile.full_name - - -@task -def reinstall(): - """Install the current component""" - local("pip uninstall -y `basename \\`pwd\\``; pip install .") diff --git a/armstrong/dev/tests/__init__.py b/armstrong/dev/tests/__init__.py index 3ad9513..e69de29 100644 --- a/armstrong/dev/tests/__init__.py +++ b/armstrong/dev/tests/__init__.py @@ -1,2 +0,0 @@ -from pkgutil import extend_path -__path__ = extend_path(__path__, __name__) diff --git a/armstrong/dev/tests/runner.py b/armstrong/dev/tests/runner.py new file mode 100644 index 0000000..5580a56 --- /dev/null +++ b/armstrong/dev/tests/runner.py @@ -0,0 +1,12 @@ +try: + from django.test.runner import DiscoverRunner +except ImportError: # < Django 1.6 + from .utils.runner import DiscoverRunner + + +class ArmstrongDiscoverRunner(DiscoverRunner): + def __init__(self, *args, **kwargs): + """Find our "tests" package, not just "test*.py" files""" + + super(ArmstrongDiscoverRunner, self).__init__(*args, **kwargs) + self.pattern = "test*" diff --git a/armstrong/dev/tests/utils/__init__.py b/armstrong/dev/tests/utils/__init__.py index f7ea4a4..aebf5bc 100644 --- a/armstrong/dev/tests/utils/__init__.py +++ b/armstrong/dev/tests/utils/__init__.py @@ -1,4 +1 @@ -from pkgutil import extend_path -__path__ = extend_path(__path__, __name__) - -from armstrong.dev.tests.utils.base import ArmstrongTestCase, override_settings \ No newline at end of file +from .base import ArmstrongTestCase, override_settings diff --git a/armstrong/dev/tests/utils/backports.py b/armstrong/dev/tests/utils/backports.py index 6f401b8..47a064e 100644 --- a/armstrong/dev/tests/utils/backports.py +++ b/armstrong/dev/tests/utils/backports.py @@ -1,8 +1,8 @@ - from django.conf import settings, UserSettingsHolder from django.utils.functional import wraps +# DEPRECATED remove when we drop Django 1.3 support class override_settings(object): """ Acts as either a decorator, or a context manager. If it's a decorator it @@ -45,5 +45,3 @@ def enable(self): def disable(self): settings._wrapped = self.wrapped - - diff --git a/armstrong/dev/tests/utils/base.py b/armstrong/dev/tests/utils/base.py index f0dde57..235fb85 100644 --- a/armstrong/dev/tests/utils/base.py +++ b/armstrong/dev/tests/utils/base.py @@ -1,19 +1,26 @@ from django.test import TestCase as DjangoTestCase -import fudge from django.db import models -# Backport override_settings from Django 1.4 +# DEPRECATED remove when we drop Django 1.3 support try: from django.test.utils import override_settings except ImportError: from .backports import override_settings +try: # If the component uses fudge, provide useful shared behavior + import fudge +except ImportError: + fudge = False + class ArmstrongTestCase(DjangoTestCase): - def setUp(self): - fudge.clear_expectations() - fudge.clear_calls() - + if fudge: + def setUp(self): + super(ArmstrongTestCase, self).setUp() + fudge.clear_expectations() + fudge.clear_calls() + + # DEPRECATED remove when we drop Django 1.3 support if not hasattr(DjangoTestCase, 'settings'): # backported from Django 1.4 def settings(self, **kwargs): @@ -24,7 +31,7 @@ def settings(self, **kwargs): .. seealso: https://github.com/django/django/blob/0d670682952fae585ce5c5ec5dc335bd61d66bb2/django/test/testcases.py#L349-354 """ return override_settings(**kwargs) - + def assertRelatedTo(self, model, field_name, related_model, many=False): if many is False: through = models.ForeignKey @@ -46,19 +53,3 @@ def assertModelHasField(self, model, field_name, field_class=None): msg = "%s.%s is not a %s" % (model.__class__.__name__, field_name, field_class.__class__.__name__) self.assertTrue(isinstance(field, field_class), msg=msg) - - def assertInContext(self, var_name, other, template_or_context): - # TODO: support passing in a straight "context" (i.e., dict) - context = template_or_context.context_data - self.assertTrue(var_name in context, - msg="`%s` not in provided context" % var_name) - self.assertEqual(context[var_name], other) - - def assertNone(self, obj, **kwargs): - self.assertTrue(obj is None, **kwargs) - - def assertIsA(self, obj, cls, **kwargs): - self.assertTrue(isinstance(obj, cls), **kwargs) - - def assertDoesNotHave(self, obj, attr, **kwargs): - self.assertFalse(hasattr(obj, attr), **kwargs) diff --git a/armstrong/dev/tests/utils/concrete.py b/armstrong/dev/tests/utils/concrete.py deleted file mode 100644 index 620606d..0000000 --- a/armstrong/dev/tests/utils/concrete.py +++ /dev/null @@ -1,55 +0,0 @@ -from django.db import connection -from django.core.management.color import no_style -import random - -def create_concrete_table(func=None, model=None): - style = no_style() - seen_models = connection.introspection.installed_models( - connection.introspection.table_names()) - - def actual_create(model): - sql, _references = connection.creation.sql_create_model(model, style, - seen_models) - cursor = connection.cursor() - for statement in sql: - cursor.execute(statement) - - if func: - def inner(self, *args, **kwargs): - func(self, *args, **kwargs) - actual_create(self.model) - return inner - elif model: - actual_create(model) - - -def destroy_concrete_table(func=None, model=None): - style = no_style() - # Assume that there are no references to destroy, these are supposed to be - # simple models - references = {} - - def actual_destroy(model): - sql = connection.creation.sql_destroy_model(model, references, style) - cursor = connection.cursor() - for statement in sql: - cursor.execute(statement) - - if func: - def inner(self, *args, **kwargs): - func(self, *args, **kwargs) - actual_destroy(self.model) - return inner - elif model: - actual_destroy(model) - - -# TODO: pull into a common dev package so all armstrong code can use it -def concrete(klass): - attrs = {'__module__': concrete.__module__, } - while True: - num = random.randint(1, 10000) - if num not in concrete.already_used: - break - return type("Concrete%s%d" % (klass.__name__, num), (klass, ), attrs) -concrete.already_used = [] diff --git a/armstrong/dev/tests/utils/runner.py b/armstrong/dev/tests/utils/runner.py new file mode 100644 index 0000000..4821e1a --- /dev/null +++ b/armstrong/dev/tests/utils/runner.py @@ -0,0 +1,293 @@ +# DEPRECATED backport for Django < 1.6 + +import os +from optparse import make_option + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.test import TestCase +from django.test.utils import setup_test_environment, teardown_test_environment +from django.utils import unittest +from django.utils.unittest import TestSuite, defaultTestLoader + + +class DiscoverRunner(object): + """ + A Django test runner that uses unittest2 test discovery. + """ + + test_loader = defaultTestLoader + reorder_by = (TestCase, ) + option_list = ( + make_option('-t', '--top-level-directory', + action='store', dest='top_level', default=None, + help='Top level of project for unittest discovery.'), + make_option('-p', '--pattern', action='store', dest='pattern', + default="test*.py", + help='The test matching pattern. Defaults to test*.py.'), + ) + + def __init__(self, pattern=None, top_level=None, + verbosity=1, interactive=True, failfast=False, + **kwargs): + + self.pattern = pattern + self.top_level = top_level + + self.verbosity = verbosity + self.interactive = interactive + self.failfast = failfast + + def setup_test_environment(self, **kwargs): + setup_test_environment() + settings.DEBUG = False + unittest.installHandler() + + def build_suite(self, test_labels=None, extra_tests=None, **kwargs): + suite = TestSuite() + test_labels = test_labels or ['.'] + extra_tests = extra_tests or [] + + discover_kwargs = {} + if self.pattern is not None: + discover_kwargs['pattern'] = self.pattern + if self.top_level is not None: + discover_kwargs['top_level_dir'] = self.top_level + + for label in test_labels: + kwargs = discover_kwargs.copy() + tests = None + + label_as_path = os.path.abspath(label) + + # if a module, or "module.ClassName[.method_name]", just run those + if not os.path.exists(label_as_path): + tests = self.test_loader.loadTestsFromName(label) + elif os.path.isdir(label_as_path) and not self.top_level: + # Try to be a bit smarter than unittest about finding the + # default top-level for a given directory path, to avoid + # breaking relative imports. (Unittest's default is to set + # top-level equal to the path, which means relative imports + # will result in "Attempted relative import in non-package."). + + # We'd be happy to skip this and require dotted module paths + # (which don't cause this problem) instead of file paths (which + # do), but in the case of a directory in the cwd, which would + # be equally valid if considered as a top-level module or as a + # directory path, unittest unfortunately prefers the latter. + + top_level = label_as_path + while True: + init_py = os.path.join(top_level, '__init__.py') + if os.path.exists(init_py): + try_next = os.path.dirname(top_level) + if try_next == top_level: + # __init__.py all the way down? give up. + break + top_level = try_next + continue + break + kwargs['top_level_dir'] = top_level + + + if not (tests and tests.countTestCases()): + # if no tests found, it's probably a package; try discovery + tests = self.test_loader.discover(start_dir=label, **kwargs) + + # make unittest forget the top-level dir it calculated from this + # run, to support running tests from two different top-levels. + self.test_loader._top_level_dir = None + + suite.addTests(tests) + + for test in extra_tests: + suite.addTest(test) + + return reorder_suite(suite, self.reorder_by) + + def setup_databases(self, **kwargs): + return setup_databases(self.verbosity, self.interactive, **kwargs) + + def run_suite(self, suite, **kwargs): + return unittest.TextTestRunner( + verbosity=self.verbosity, + failfast=self.failfast, + ).run(suite) + + def teardown_databases(self, old_config, **kwargs): + """ + Destroys all the non-mirror databases. + """ + old_names, mirrors = old_config + for connection, old_name, destroy in old_names: + if destroy: + connection.creation.destroy_test_db(old_name, self.verbosity) + + def teardown_test_environment(self, **kwargs): + unittest.removeHandler() + teardown_test_environment() + + def suite_result(self, suite, result, **kwargs): + return len(result.failures) + len(result.errors) + + def run_tests(self, test_labels, extra_tests=None, **kwargs): + """ + Run the unit tests for all the test labels in the provided list. + + Test labels should be dotted Python paths to test modules, test + classes, or test methods. + + A list of 'extra' tests may also be provided; these tests + will be added to the test suite. + + Returns the number of tests that failed. + """ + self.setup_test_environment() + suite = self.build_suite(test_labels, extra_tests) + old_config = self.setup_databases() + result = self.run_suite(suite) + self.teardown_databases(old_config) + self.teardown_test_environment() + return self.suite_result(suite, result) + + +def dependency_ordered(test_databases, dependencies): + """ + Reorder test_databases into an order that honors the dependencies + described in TEST_DEPENDENCIES. + """ + ordered_test_databases = [] + resolved_databases = set() + + # Maps db signature to dependencies of all it's aliases + dependencies_map = {} + + # sanity check - no DB can depend on it's own alias + for sig, (_, aliases) in test_databases: + all_deps = set() + for alias in aliases: + all_deps.update(dependencies.get(alias, [])) + if not all_deps.isdisjoint(aliases): + raise ImproperlyConfigured( + "Circular dependency: databases %r depend on each other, " + "but are aliases." % aliases) + dependencies_map[sig] = all_deps + + while test_databases: + changed = False + deferred = [] + + # Try to find a DB that has all it's dependencies met + for signature, (db_name, aliases) in test_databases: + if dependencies_map[signature].issubset(resolved_databases): + resolved_databases.update(aliases) + ordered_test_databases.append((signature, (db_name, aliases))) + changed = True + else: + deferred.append((signature, (db_name, aliases))) + + if not changed: + raise ImproperlyConfigured( + "Circular dependency in TEST_DEPENDENCIES") + test_databases = deferred + return ordered_test_databases + + +def reorder_suite(suite, classes): + """ + Reorders a test suite by test type. + + `classes` is a sequence of types + + All tests of type classes[0] are placed first, then tests of type + classes[1], etc. Tests with no match in classes are placed last. + """ + class_count = len(classes) + bins = [unittest.TestSuite() for i in range(class_count+1)] + partition_suite(suite, classes, bins) + for i in range(class_count): + bins[0].addTests(bins[i+1]) + return bins[0] + + +def partition_suite(suite, classes, bins): + """ + Partitions a test suite by test type. + + classes is a sequence of types + bins is a sequence of TestSuites, one more than classes + + Tests of type classes[i] are added to bins[i], + tests with no match found in classes are place in bins[-1] + """ + for test in suite: + if isinstance(test, unittest.TestSuite): + partition_suite(test, classes, bins) + else: + for i in range(len(classes)): + if isinstance(test, classes[i]): + bins[i].addTest(test) + break + else: + bins[-1].addTest(test) + + +def setup_databases(verbosity, interactive, **kwargs): + from django.db import connections, DEFAULT_DB_ALIAS + + # First pass -- work out which databases actually need to be created, + # and which ones are test mirrors or duplicate entries in DATABASES + mirrored_aliases = {} + test_databases = {} + dependencies = {} + default_sig = connections[DEFAULT_DB_ALIAS].creation.test_db_signature() + for alias in connections: + connection = connections[alias] + if connection.settings_dict['TEST_MIRROR']: + # If the database is marked as a test mirror, save + # the alias. + mirrored_aliases[alias] = ( + connection.settings_dict['TEST_MIRROR']) + else: + # Store a tuple with DB parameters that uniquely identify it. + # If we have two aliases with the same values for that tuple, + # we only need to create the test database once. + item = test_databases.setdefault( + connection.creation.test_db_signature(), + (connection.settings_dict['NAME'], set()) + ) + item[1].add(alias) + + if 'TEST_DEPENDENCIES' in connection.settings_dict: + dependencies[alias] = ( + connection.settings_dict['TEST_DEPENDENCIES']) + else: + if alias != DEFAULT_DB_ALIAS and connection.creation.test_db_signature() != default_sig: + dependencies[alias] = connection.settings_dict.get( + 'TEST_DEPENDENCIES', [DEFAULT_DB_ALIAS]) + + # Second pass -- actually create the databases. + old_names = [] + mirrors = [] + + for signature, (db_name, aliases) in dependency_ordered( + test_databases.items(), dependencies): + test_db_name = None + # Actually create the database for the first connection + for alias in aliases: + connection = connections[alias] + if test_db_name is None: + test_db_name = connection.creation.create_test_db( + verbosity, autoclobber=not interactive) + destroy = True + else: + connection.settings_dict['NAME'] = test_db_name + destroy = False + old_names.append((connection, db_name, destroy)) + + for alias, mirror_alias in mirrored_aliases.items(): + mirrors.append((alias, connections[alias].settings_dict['NAME'])) + connections[alias].settings_dict['NAME'] = ( + connections[mirror_alias].settings_dict['NAME']) + + return old_names, mirrors diff --git a/armstrong/dev/tests/utils/users.py b/armstrong/dev/tests/utils/users.py index 82cc141..1b44b7e 100644 --- a/armstrong/dev/tests/utils/users.py +++ b/armstrong/dev/tests/utils/users.py @@ -1,33 +1,22 @@ -from armstrong.dev.tests.utils.base import ArmstrongTestCase -from django.contrib.auth.models import User import random - -def generate_random_user(): - r = random.randint(10000, 20000) - return User.objects.create(username="random-user-%d" % r, - first_name="Some", last_name="Random User %d" % r) - - -def generate_random_staff_users(n=2): - orig_users = generate_random_users(n) - users = User.objects.filter(pk__in=[a.id for a in orig_users]) - users.update(is_staff=True) - return [a for a in users] - - -class generate_random_staff_usersTestCase(ArmstrongTestCase): - def test_returns_2_users_by_default(self): - self.assertEqual(len(generate_random_staff_users()), 2) - - def test_returns_n_users(self): - r = random.randint(1, 5) - self.assertEqual(len(generate_random_staff_users(r)), r) - - def test_all_users_are_staff(self): - users = generate_random_staff_users() - for user in users: - self.assertTrue(user.is_staff) - - -def generate_random_users(n=2): - return [generate_random_user() for i in range(n)] \ No newline at end of file +try: + from django.contrib.auth import get_user_model +except ImportError: # Django < 1.5 + from django.contrib.auth.models import User +else: + User = get_user_model() + + +def generate_random_users(count, **extra_fields): + """Generator to create ``count`` number of unique random users""" + + num = random.randint(1000, 2000) + while count > 0: + fields = dict( + username="random-user-%d" % num, + first_name="Some", + last_name="Random User %d" % num, + **extra_fields) + yield User.objects.create(**fields) + num += random.randint(2, 20) + count -= 1 diff --git a/armstrong/dev/virtualdjango/__init__.py b/armstrong/dev/virtualdjango/__init__.py deleted file mode 100644 index 3ad9513..0000000 --- a/armstrong/dev/virtualdjango/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -from pkgutil import extend_path -__path__ = extend_path(__path__, __name__) diff --git a/armstrong/dev/virtualdjango/base.py b/armstrong/dev/virtualdjango/base.py deleted file mode 100644 index b155a76..0000000 --- a/armstrong/dev/virtualdjango/base.py +++ /dev/null @@ -1,57 +0,0 @@ -import django -import os, sys - -DEFAULT_SETTINGS = { - 'DATABASE_ENGINE': 'sqlite3', - 'DATABASES': { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': 'mydatabase' - } - }, -} - -class VirtualDjango(object): - def __init__(self, - caller=sys.modules['__main__'], - default_settings=DEFAULT_SETTINGS): - self.caller = caller - self.default_settings = default_settings - - - def configure_settings(self, customizations, reset=True): - # Django expects a `DATABASE_ENGINE` value - custom_settings = self.default_settings - custom_settings.update(customizations) - - settings = self.settings - if reset: - self.reset_settings(settings) - settings.configure(**custom_settings) - - def reset_settings(self, settings): - if django.VERSION[:2] == (1, 3): - settings._wrapped = None - return - - # This is the way to reset settings going forward - from django.utils.functional import empty - settings._wrapped = empty - - @property - def settings(self): - from django.conf import settings - return settings - - @property - def call_command(self): - from django.core.management import call_command - return call_command - - def run(self, my_settings): - if hasattr(self.caller, 'setUp'): - self.caller.setUp() - - self.configure_settings(my_settings) - return self.call_command - diff --git a/armstrong/dev/virtualdjango/test_runner.py b/armstrong/dev/virtualdjango/test_runner.py deleted file mode 100644 index 1b18604..0000000 --- a/armstrong/dev/virtualdjango/test_runner.py +++ /dev/null @@ -1,12 +0,0 @@ -from armstrong.dev.virtualdjango.base import VirtualDjango - -class VirtualDjangoTestRunner(VirtualDjango): - def run(self, my_settings, *apps_to_test): - super(VirtualDjangoTestRunner, self).run(my_settings) - self.call_command('test', *apps_to_test) - - def __call__(self, *args, **kwargs): - self.run(*args, **kwargs) - -run_tests = VirtualDjangoTestRunner() - diff --git a/package.json b/package.json index c39c8e0..40e3974 100644 --- a/package.json +++ b/package.json @@ -1,12 +1,8 @@ { "name": "armstrong.dev", - "version": "1.14.0alpha.0", + "version": "2.0.0alpha.0", "description": "Tools needed for development and testing of Armstrong", "install_requires": [ - "South==0.7.3", - "Fabric==1.3.3", - "pep8 == 0.6.1", - "coverage == 3.5.1", - "fudge == 1.0.3" + "decorator<4.0" ] } diff --git a/setup.py b/setup.py index e34dfcb..8cc11d2 100644 --- a/setup.py +++ b/setup.py @@ -1,98 +1,40 @@ -""" -setup.py file for building armstrong components. - -Nothing in this file should need to be edited, please see accompanying -package.json file if you need to adjust metadata about this package. -""" - -from distutils.core import setup -import json +# Nothing in this file should need to be edited. +# Use package.json to adjust metadata about this package. +# Use MANIFEST.in to include package-specific data files. import os +import json +from setuptools import setup, find_packages -info = json.load(open("./package.json")) - - -def convert_to_str(d): - """ - Recursively convert all values in a dictionary to strings - - This is required because setup() does not like unicode in - the values it is supplied. - """ - d2 = {} - for k, v in d.items(): - k = str(k) - if type(v) in [list, tuple]: - d2[k] = [str(a) for a in v] - elif type(v) is dict: - d2[k] = convert_to_str(v) - else: - d2[k] = str(v) - return d2 -info = convert_to_str(info) -NAMESPACE_PACKAGES = [] +info = json.load(open("./package.json")) -# TODO: simplify this process def generate_namespaces(package): - new_package = ".".join(package.split(".")[0:-1]) - if new_package.count(".") > 0: - generate_namespaces(new_package) - NAMESPACE_PACKAGES.append(new_package) -generate_namespaces(info["name"]) - + i = package.count(".") + while i: + yield package.rsplit(".", i)[0] + i -= 1 +NAMESPACE_PACKAGES = list(generate_namespaces(info['name'])) if os.path.exists("MANIFEST"): os.unlink("MANIFEST") -# Borrowed and modified from django-registration -# Compile the list of packages available, because distutils doesn't have -# an easy way to do this. -packages, data_files = [], [] -root_dir = os.path.dirname(__file__) -if root_dir: - os.chdir(root_dir) - - -def build_package(dirpath, dirnames, filenames): - # Ignore dirnames that start with '.' - for i, dirname in enumerate(dirnames): - if dirname.startswith('.'): - del dirnames[i] - if '__init__.py' in filenames and 'steps.py' not in filenames: - pkg = dirpath.replace(os.path.sep, '.') - if os.path.altsep: - pkg = pkg.replace(os.path.altsep, '.') - packages.append(pkg) - elif filenames: - # Strip off the length of the package name plus the trailing slash - prefix = dirpath[len(info["name"]) + 1:] - for f in filenames: - # Ignore all dot files and any compiled - if f.startswith(".") or f.endswith(".pyc"): - continue - data_files.append(os.path.join(prefix, f)) - - -[build_package(dirpath, dirnames, filenames) for dirpath, dirnames, filenames - in os.walk(info["name"].replace(".", "/"))] - setup_kwargs = { - "author": "Bay Citizen & Texas Tribune", + "author": "Texas Tribune & The Center for Investigative Reporting", "author_email": "dev@armstrongcms.org", "url": "http://github.com/armstrong/%s/" % info["name"], - "packages": packages, - "package_data": {info["name"]: data_files, }, + "packages": find_packages(), "namespace_packages": NAMESPACE_PACKAGES, + "include_package_data": True, "classifiers": [ - 'Development Status :: 3 - Alpha', - 'Environment :: Web Environment', + 'Development Status :: 5 - Production/Stable', + 'Environment :: Console', 'Framework :: Django', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', + 'Topic :: Software Development :: Testing', ], }