Docs and manifest update after splitting packages

MANIFEST.in

@@ -1,9 +1,9 @@
 include AUTHORS
 include LICENSE
 include README.md
-include requirements*.txt
-include stor/default.cfg
-include stor/default.env
-include stor/stor-completion.bash
-prune stor/tests
+include stor*/requirements*.txt
+include stor/stor/default.cfg
+include stor/stor/default.env
+include stor/stor/stor-completion.bash
+prune stor*/stor*/tests
 exclude Makefile

docs/package_split.rst

@@ -1,40 +1,38 @@
 Stor Modularization
 ===================
 
-The legacy version of stor has been split into 4 packages: ``stor``, ``stor_dx``, ``stor_swift``, and
-``stor_s3``. ``stor_swift``, ``stor_dx``, ``stor_s3`` are modular packages implemented to be used with
-the core package ``stor`` directly. The core package ``stor`` now only supports Posix and Windows
-filesystems, apart from supporting extra plugins.
+Stor is implemented as modular packages: the core ``stor`` package which is required to use stor, and
+which handles posix and windows (local) paths, and the plugins ``stor_dx``, ``stor_swift``, and
+``stor_s3``. The plugins are meant to be used with the core package ``stor``, and are not meant to
+function as stand-alone packages. This may be changed in the future.
+
+The legacy version of stor was implemented as one monolith which supported dx, swift, s3, etc, but
+was changed due to extraneous dependencies and modular requirements. Stor now supports multiple backends
+flexibly, so that only required dependencies need to be installed. This page describes these changes.
 
 
 Implementation
 --------------
 
-Each of the modular packages work by registering themselves onto pkg_resources with an entry_point
-``stor.providers``. This entry point should be a function which takes in a prefix and a path, and
-returns the cls that the path should be instantiated to, as well as the path that should be initialized.
+Each of the modular packages work by registering themselves onto ``pkg_resources`` with an entry_point
+``stor.providers`` using setuptools. This configuration is placed in ``setup.cfg`` of each package.
+This entry point is a function which takes in a prefix and a path, and returns the class that the path
+should be instantiated to, as well as the path (with any possible changes) that should be initialized.
 Typically, this function is called ``class_for_path`` in ``stor_dx``, ``stor_s3``, and ``stor_swift``.
-Each plugin module currently raise an error if the prefix passed is not the prefix it supports. For
-example, ``stor_dx.class_for_path`` errors if the prefix is not ``dx``. ``get_class_for_path`` in each
-plugin module may assume that the prefix is the true prefix to the path argument as this is guaranteed
-by the core ``stor`` package.
-
-
-Code Changes
-------------
-
-`stor.copy`, `stor.copytree` and ``stor.open`` which were earlier present in the core ``stor.utils`` and
-``stor.obs`` have been split according to their individual functionalities into the three packages.
-These functions in the core package now only deal with posix/windows paths while the three plugins
-implement the finer aspects of the logic individual to each platform. The only external effect of
-these changes is that `stor.copy` and `stor.copytree` now don't support a ``source`` kwarg, instead
-expect the first argument to be a ``Path | str``, which is then taken to be the source to be copied from.
-
-``is_swift_path`` has been removed from the core `stor` package. Thus, using ``stor.is_swift_path`` will
-fail. This is because the plugins determine the prefix they support and the core package cannot know in
-advance if ``swift://`` is a supported path. In cases where ``stor.is_swift_path`` was being used,
-``stor.is_obs_path`` is a possible substitution. Thus, each individual plugin ``stor_dx``, ``stor_s3`` and
-``stor_swift`` is now responsible for supporting ``is_dx_path``, ``is_s3_path`` and ``is_swift_path`` resp.
+
+.. code:: python
+
+    [entry_points]
+    stor.providers =
+        dx = stor_dx:class_for_path
+
+
+A plugin must implement a single function that takes in A(prefix) and B(str) and returns a tuple of
+(C(class), D(str)),  then it can be registered with ``stor.providers``. The prefix determines the path
+prefix that the plugin in question would support, and the paths stor would forward to the plugin. For
+example, ``stor_dx.class_for_path`` errors if the prefix is not ``dx``. In addition, ``class_for_path``
+in each plugin module can assume that the prefix argument is truly the prefix to the path argument as
+this is guaranteed by the core ``stor`` package. Future plugins will be implemented in this fashion.
 
 
 Versioning

docs/release_notes.rst

@@ -22,6 +22,22 @@ API Breaks
 * ``is_swift_path`` has been removed from the core ``stor`` package. Hence, ``stor.is_swift_path`` is no
   longer supported. Can use `stor.is_obs_path` in such cases after consideration.
 
+Code Changes
+------------
+
+The following code blocks and functions were affected when shifting legacy stor to a modular version.
+
+* `stor.copy`, `stor.copytree` and ``stor.open`` which were earlier present in the core ``stor.utils`` and
+  ``stor.obs`` are now split according to their individual functions into the three plugin packages.
+  These functions in the core package now only deal with posix/windows paths while the three plugins
+  implement the finer aspects of the logic individual to each platform.
+* ``is_swift_path`` has been removed because the plugins determine the prefix they support and the core
+  package cannot know in advance if ``swift://`` is a supported path. In cases where ``stor.is_swift_path``
+  was being used, ``stor.is_obs_path`` is a possible substitution, given the initial intent was to check
+  against cloud paths. Thus, each individual plugin ``stor_dx``, ``stor_s3`` and ``stor_swift`` is now
+  responsible for supporting ``is_dx_path``, ``is_s3_path`` and ``is_swift_path`` resp.
+
+
 v3.0.0
 ------
 

stor/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include requirements*.txt
+include stor/default.cfg
+include stor/default.env
+include stor/stor-completion.bash
+prune stor/tests

stor_dx/MANIFEST.in

@@ -0,0 +1,2 @@
+include requirements*.txt
+prune stor_dx/tests

stor_dx/stor_dx/__init__.py

@@ -3,11 +3,8 @@
 from stor_dx import utils
 
 
-drive = DXPath.drive
-
-
 def class_for_path(prefix, path):
-    if prefix+'://' != drive:
+    if prefix+'://' != DXPath.drive:
         raise ValueError('Invalid prefix to initialize DXPaths: {}'.format(prefix))
     cls = utils.find_dx_class(path)
     return cls, path

stor_s3/MANIFEST.in

@@ -0,0 +1,2 @@
+include requirements*.txt
+prune stor_s3/tests

stor_s3/stor_s3/__init__.py

@@ -2,11 +2,8 @@
 from stor_s3.s3 import S3Path
 
 
-drive = S3Path.drive
-
-
 def class_for_path(prefix, path):
-    if prefix+'://' != drive:
+    if prefix+'://' != S3Path.drive:
         raise ValueError('Invalid prefix to initialize S3Paths: {}'.format(prefix))
     cls = S3Path
     return cls, path

stor_swift/MANIFEST.in

@@ -0,0 +1,2 @@
+include requirements*.txt
+prune stor_swift/tests

stor_swift/stor_swift/__init__.py

@@ -2,12 +2,8 @@
 from stor_swift.swift import SwiftPath
 
 
-
-drive = SwiftPath.drive
-
-
 def class_for_path(prefix, path):
-    if prefix+'://' != drive:
+    if prefix+'://' != SwiftPath.drive:
         raise ValueError('Invalid prefix to initialize SwiftPaths: {}'.format(prefix))
     cls = SwiftPath
     return cls, path