From 3399a8f06a16be0033773aa2bcf54c292c541ebf Mon Sep 17 00:00:00 2001 From: Anuj Kumar Date: Tue, 4 Dec 2018 13:33:59 -0800 Subject: [PATCH 1/3] changing manifest --- MANIFEST.in | 10 +++++----- stor/MANIFEST.in | 6 ++++++ stor_dx/MANIFEST.in | 2 ++ stor_s3/MANIFEST.in | 2 ++ stor_swift/MANIFEST.in | 2 ++ 5 files changed, 17 insertions(+), 5 deletions(-) create mode 100644 stor/MANIFEST.in create mode 100644 stor_dx/MANIFEST.in create mode 100644 stor_s3/MANIFEST.in create mode 100644 stor_swift/MANIFEST.in diff --git a/MANIFEST.in b/MANIFEST.in index ac7a28b0..70391a0f 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,9 +1,9 @@ include AUTHORS include LICENSE include README.md -include requirements*.txt -include stor/default.cfg -include stor/default.env -include stor/stor-completion.bash -prune stor/tests +include stor*/requirements*.txt +include stor/stor/default.cfg +include stor/stor/default.env +include stor/stor/stor-completion.bash +prune stor*/stor*/tests exclude Makefile diff --git a/stor/MANIFEST.in b/stor/MANIFEST.in new file mode 100644 index 00000000..bac86c63 --- /dev/null +++ b/stor/MANIFEST.in @@ -0,0 +1,6 @@ +include README.md +include requirements*.txt +include stor/default.cfg +include stor/default.env +include stor/stor-completion.bash +prune stor/tests diff --git a/stor_dx/MANIFEST.in b/stor_dx/MANIFEST.in new file mode 100644 index 00000000..11a47684 --- /dev/null +++ b/stor_dx/MANIFEST.in @@ -0,0 +1,2 @@ +include requirements*.txt +prune stor_dx/tests diff --git a/stor_s3/MANIFEST.in b/stor_s3/MANIFEST.in new file mode 100644 index 00000000..cea9cdcc --- /dev/null +++ b/stor_s3/MANIFEST.in @@ -0,0 +1,2 @@ +include requirements*.txt +prune stor_s3/tests \ No newline at end of file diff --git a/stor_swift/MANIFEST.in b/stor_swift/MANIFEST.in new file mode 100644 index 00000000..74bd5b8d --- /dev/null +++ b/stor_swift/MANIFEST.in @@ -0,0 +1,2 @@ +include requirements*.txt +prune stor_swift/tests \ No newline at end of file From 3580aeba57535684dfe758f2603fb9d29d34e3e1 Mon Sep 17 00:00:00 2001 From: Anuj Kumar Date: Wed, 5 Dec 2018 10:13:50 -0800 Subject: [PATCH 2/3] updating docs and removing drive from top level --- docs/package_split.rst | 45 ++++++++++++++++++++----------- stor_dx/stor_dx/__init__.py | 5 +--- stor_s3/stor_s3/__init__.py | 5 +--- stor_swift/stor_swift/__init__.py | 6 +---- 4 files changed, 33 insertions(+), 28 deletions(-) diff --git a/docs/package_split.rst b/docs/package_split.rst index bf9483fa..b7816dec 100644 --- a/docs/package_split.rst +++ b/docs/package_split.rst @@ -1,34 +1,49 @@ Stor Modularization =================== -The legacy version of stor has been split into 4 packages: ``stor``, ``stor_dx``, ``stor_swift``, and -``stor_s3``. ``stor_swift``, ``stor_dx``, ``stor_s3`` are modular packages implemented to be used with -the core package ``stor`` directly. The core package ``stor`` now only supports Posix and Windows -filesystems, apart from supporting extra plugins. +Stor is implemented as modular packages: the core ``stor`` package which is required to use stor, and +which handles posix and windows (local) paths, and the plugins ``stor_dx``, ``stor_swift``, and +``stor_s3``. The plugins are meant to be used with the core package ``stor``, and are not meant to +function as stand-alone packages. This may be changed in the future. + +The legacy version of stor was implemented as one monolith which supported dx, swift, s3, etc, but +was changed due to extraneous dependencies and modular requirements. This page describes these changes. Implementation -------------- -Each of the modular packages work by registering themselves onto pkg_resources with an entry_point -``stor.providers``. This entry point should be a function which takes in a prefix and a path, and -returns the cls that the path should be instantiated to, as well as the path that should be initialized. +Each of the modular packages work by registering themselves onto ``pkg_resources`` with an entry_point +``stor.providers`` using setuptools. This configuration is placed in ``setup.cfg`` of each package. +This entry point is a function which takes in a prefix and a path, and returns the class that the path +should be instantiated to, as well as the path (with any possible changes) that should be initialized. Typically, this function is called ``class_for_path`` in ``stor_dx``, ``stor_s3``, and ``stor_swift``. -Each plugin module currently raise an error if the prefix passed is not the prefix it supports. For -example, ``stor_dx.class_for_path`` errors if the prefix is not ``dx``. ``get_class_for_path`` in each -plugin module may assume that the prefix is the true prefix to the path argument as this is guaranteed -by the core ``stor`` package. + +.. code:: python + + [entry_points] + stor.providers = + dx = stor_dx:class_for_path + + +Each plugin currently raises an error if the prefix provided is not the prefix it supports. For +example, ``stor_dx.class_for_path`` errors if the prefix is not ``dx``. In addition, ``class_for_path`` +in each plugin module can assume that the prefix argument is truly the prefix to the path argument as +this is guaranteed by the core ``stor`` package. Future plugins will be implemented in this fashion. Code Changes ------------ +The following code blocks and functions were affected when shifting legacy stor to a modular version. + `stor.copy`, `stor.copytree` and ``stor.open`` which were earlier present in the core ``stor.utils`` and -``stor.obs`` have been split according to their individual functionalities into the three packages. +``stor.obs`` are now split according to their individual functions into the three plugin packages. These functions in the core package now only deal with posix/windows paths while the three plugins -implement the finer aspects of the logic individual to each platform. The only external effect of -these changes is that `stor.copy` and `stor.copytree` now don't support a ``source`` kwarg, instead -expect the first argument to be a ``Path | str``, which is then taken to be the source to be copied from. +implement the finer aspects of the logic individual to each platform. The external effect of +these changes is that `stor.copy` and `stor.copytree` which used to support ``source`` kwarg argument in +legacy stor, instead now expect an initial argument to be a ``Path | str``, which is then taken to be the +source to be copied from. The destination is decided by the ``dest`` kwarg as in legacy stor. ``is_swift_path`` has been removed from the core `stor` package. Thus, using ``stor.is_swift_path`` will fail. This is because the plugins determine the prefix they support and the core package cannot know in diff --git a/stor_dx/stor_dx/__init__.py b/stor_dx/stor_dx/__init__.py index 97167e39..7b378950 100644 --- a/stor_dx/stor_dx/__init__.py +++ b/stor_dx/stor_dx/__init__.py @@ -3,11 +3,8 @@ from stor_dx import utils -drive = DXPath.drive - - def class_for_path(prefix, path): - if prefix+'://' != drive: + if prefix+'://' != DXPath.drive: raise ValueError('Invalid prefix to initialize DXPaths: {}'.format(prefix)) cls = utils.find_dx_class(path) return cls, path diff --git a/stor_s3/stor_s3/__init__.py b/stor_s3/stor_s3/__init__.py index fb041981..2d476489 100644 --- a/stor_s3/stor_s3/__init__.py +++ b/stor_s3/stor_s3/__init__.py @@ -2,11 +2,8 @@ from stor_s3.s3 import S3Path -drive = S3Path.drive - - def class_for_path(prefix, path): - if prefix+'://' != drive: + if prefix+'://' != S3Path.drive: raise ValueError('Invalid prefix to initialize S3Paths: {}'.format(prefix)) cls = S3Path return cls, path diff --git a/stor_swift/stor_swift/__init__.py b/stor_swift/stor_swift/__init__.py index 7f3446d8..0fdbab02 100644 --- a/stor_swift/stor_swift/__init__.py +++ b/stor_swift/stor_swift/__init__.py @@ -2,12 +2,8 @@ from stor_swift.swift import SwiftPath - -drive = SwiftPath.drive - - def class_for_path(prefix, path): - if prefix+'://' != drive: + if prefix+'://' != SwiftPath.drive: raise ValueError('Invalid prefix to initialize SwiftPaths: {}'.format(prefix)) cls = SwiftPath return cls, path From 1a9a4f6fe188b58bedbd2025e1526fd2e2e6f3df Mon Sep 17 00:00:00 2001 From: Anuj Kumar Date: Wed, 5 Dec 2018 14:00:40 -0800 Subject: [PATCH 3/3] addressed doc changes review comments --- docs/package_split.rst | 27 +++++---------------------- docs/release_notes.rst | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 22 deletions(-) diff --git a/docs/package_split.rst b/docs/package_split.rst index b7816dec..4aa45295 100644 --- a/docs/package_split.rst +++ b/docs/package_split.rst @@ -7,7 +7,8 @@ which handles posix and windows (local) paths, and the plugins ``stor_dx``, ``st function as stand-alone packages. This may be changed in the future. The legacy version of stor was implemented as one monolith which supported dx, swift, s3, etc, but -was changed due to extraneous dependencies and modular requirements. This page describes these changes. +was changed due to extraneous dependencies and modular requirements. Stor now supports multiple backends +flexibly, so that only required dependencies need to be installed. This page describes these changes. Implementation @@ -26,32 +27,14 @@ Typically, this function is called ``class_for_path`` in ``stor_dx``, ``stor_s3` dx = stor_dx:class_for_path -Each plugin currently raises an error if the prefix provided is not the prefix it supports. For +A plugin must implement a single function that takes in A(prefix) and B(str) and returns a tuple of +(C(class), D(str)), then it can be registered with ``stor.providers``. The prefix determines the path +prefix that the plugin in question would support, and the paths stor would forward to the plugin. For example, ``stor_dx.class_for_path`` errors if the prefix is not ``dx``. In addition, ``class_for_path`` in each plugin module can assume that the prefix argument is truly the prefix to the path argument as this is guaranteed by the core ``stor`` package. Future plugins will be implemented in this fashion. -Code Changes ------------- - -The following code blocks and functions were affected when shifting legacy stor to a modular version. - -`stor.copy`, `stor.copytree` and ``stor.open`` which were earlier present in the core ``stor.utils`` and -``stor.obs`` are now split according to their individual functions into the three plugin packages. -These functions in the core package now only deal with posix/windows paths while the three plugins -implement the finer aspects of the logic individual to each platform. The external effect of -these changes is that `stor.copy` and `stor.copytree` which used to support ``source`` kwarg argument in -legacy stor, instead now expect an initial argument to be a ``Path | str``, which is then taken to be the -source to be copied from. The destination is decided by the ``dest`` kwarg as in legacy stor. - -``is_swift_path`` has been removed from the core `stor` package. Thus, using ``stor.is_swift_path`` will -fail. This is because the plugins determine the prefix they support and the core package cannot know in -advance if ``swift://`` is a supported path. In cases where ``stor.is_swift_path`` was being used, -``stor.is_obs_path`` is a possible substitution. Thus, each individual plugin ``stor_dx``, ``stor_s3`` and -``stor_swift`` is now responsible for supporting ``is_dx_path``, ``is_s3_path`` and ``is_swift_path`` resp. - - Versioning ---------- Since the core stor package and the plugins live on the same github repo, and PBR is used for semantic diff --git a/docs/release_notes.rst b/docs/release_notes.rst index 125dd050..5f9cfa0e 100644 --- a/docs/release_notes.rst +++ b/docs/release_notes.rst @@ -22,6 +22,22 @@ API Breaks * ``is_swift_path`` has been removed from the core ``stor`` package. Hence, ``stor.is_swift_path`` is no longer supported. Can use `stor.is_obs_path` in such cases after consideration. +Code Changes +------------ + +The following code blocks and functions were affected when shifting legacy stor to a modular version. + +* `stor.copy`, `stor.copytree` and ``stor.open`` which were earlier present in the core ``stor.utils`` and + ``stor.obs`` are now split according to their individual functions into the three plugin packages. + These functions in the core package now only deal with posix/windows paths while the three plugins + implement the finer aspects of the logic individual to each platform. +* ``is_swift_path`` has been removed because the plugins determine the prefix they support and the core + package cannot know in advance if ``swift://`` is a supported path. In cases where ``stor.is_swift_path`` + was being used, ``stor.is_obs_path`` is a possible substitution, given the initial intent was to check + against cloud paths. Thus, each individual plugin ``stor_dx``, ``stor_s3`` and ``stor_swift`` is now + responsible for supporting ``is_dx_path``, ``is_s3_path`` and ``is_swift_path`` resp. + + v3.0.0 ------