Code comments extras

[thorwhalen]

This is a place to dump extra commentary about code, commits, architecture, that we may want to have a link for, which we can then use in code comments to direct the reader here, or in various documenting resources.

[thorwhalen]

current_pypi_version: Safer way to analyze errors (and determine if the package exists or other HTTPError)

From this version
to the new version.

To address the TODO in the given code, it's important to implement a more robust error handling mechanism that can differentiate between various types of HTTP errors (such as 404 for "package not found" or other issues). Here’s a proposed solution:

1. Add specific exception handling: Utilize the requests library's exceptions to handle different HTTP errors explicitly.
2. Check if the package exists: Handle cases where the package does not exist separately.
3. Improve version comparison: Ensure the sorting of version strings is robust by using packaging.version.

Here’s the revised code:

import requests
from requests.exceptions import HTTPError, ConnectionError, Timeout
from pathlib import Path
from typing import Union
from packaging.version import Version, InvalidVersion

# Default PyPI URL template
DLFT_PYPI_PACKAGE_JSON_URL_TEMPLATE = "https://pypi.org/pypi/{package}/json"

def current_pypi_version(
    pkg_dir: Path,
    *,
    name: Union[None, str] = None,
    url_template=DLFT_PYPI_PACKAGE_JSON_URL_TEMPLATE,
    use_requests=requests_is_installed,
) -> Union[str, None]:
    """
    Return version of package on pypi.python.org using json.

    ::

        current_pypi_version('py2store')
        '0.0.7'

    :param pkg_dir: Directory of the package
    :param name: Name of the package (optional)
    :param url_template: URL template for PyPI JSON endpoint
    :param use_requests: Boolean flag to use requests library
    :return: A version (string) or None if there was an exception
    """
    pkg_dir, pkg_dirname = validate_pkg_dir(pkg_dir)
    name = name or get_name_from_configs(pkg_dir)
    assert (
        pkg_dirname == name
    ), f'pkg_dirname ({pkg_dirname}) and name ({name}) were not the same'
    url = url_template.format(package=name)
    
    try:
        response = requests.get(url) if use_requests else http_get_json(url, use_requests=use_requests)
        response.raise_for_status()
        data = response.json()
    except HTTPError as http_err:
        if http_err.response.status_code == 404:
            # Package not found on PyPI
            return None
        else:
            raise
    except (ConnectionError, Timeout) as conn_err:
        # Handle network related errors
        print(f"Network error occurred: {conn_err}")
        return None
    except Exception as err:
        # Catch-all for other exceptions
        print(f"An error occurred: {err}")
        return None
    
    releases = data.get('releases', [])
    if releases:
        try:
            sorted_versions = sorted(releases, key=lambda r: Version(r))
            return sorted_versions[-1]
        except InvalidVersion:
            print("Invalid version string encountered.")
            return None
    
    return None

# Helper functions (assuming they are defined elsewhere in the code)
def validate_pkg_dir(pkg_dir: Path) -> tuple:
    # Placeholder for actual validation logic
    pkg_dirname = pkg_dir.name
    return pkg_dir, pkg_dirname

def get_name_from_configs(pkg_dir: Path) -> str:
    # Placeholder for actual logic to get name from configs
    return pkg_dir.name

def http_get_json(url: str, use_requests: bool = True) -> dict:
    # Placeholder for actual HTTP GET request logic
    if use_requests:
        response = requests.get(url)
        response.raise_for_status()
        return response.json()
    else:
        # Handle alternative HTTP GET logic if not using requests
        pass

requests_is_installed = True  # Assume the `requests` library is installed for this example

Explanation:

1. Exception Handling: Added try-except blocks to catch and handle different types of exceptions (HTTPError, ConnectionError, Timeout, and a generic Exception).
2. HTTP 404: Specifically handles the HTTPError for a 404 status code, which indicates that the package does not exist on PyPI, returning None in this case.
3. Version Sorting: Used the packaging.version.Version class to handle version strings properly and ensure correct sorting.
4. Fallback for HTTP Requests: Added a check to decide whether to use requests or another method for HTTP GET requests, depending on the use_requests flag.

This solution improves the robustness and clarity of the error handling in the current_pypi_version function.

[thorwhalen]

Finding the path of a module.

Is the following the best way to get the path to the local directory that has the code for a given module? What problems might this solution have? What other solutions might you propose? What problems might they have?

    # if pkg_dir is a module object, get the path to the module
    if isinstance(pkg_dir, ModuleType):
        pkg_dir = os.path.dirname(pkg_dir.__file__)

Using os.path.dirname(pkg_dir.__file__) to get the path to the local directory of a given module is a common approach, but it has its limitations and potential issues. Here are some considerations and alternative solutions:

Current Approach

if isinstance(pkg_dir, ModuleType):
    pkg_dir = os.path.dirname(pkg_dir.__file__)

Pros:

• Simple and straightforward.
• Works for most typical use cases where the module is loaded from a standard filesystem.

Cons:

• Namespace Packages: This approach does not work well with namespace packages, which may not have a __file__ attribute.
• Frozen Modules: Modules that are frozen (embedded in a binary) might not have a __file__ attribute.
• Missing __file__ Attribute: Some dynamically created modules or certain environments might have modules without a __file__ attribute.

Alternative Solutions

1. Using importlib.util.find_spec

The importlib.util.find_spec function can be used to locate the module spec, which provides more robust information about the module.

import importlib.util
import os

def get_module_path(module):
    if isinstance(module, ModuleType):
        spec = importlib.util.find_spec(module.__name__)
        if spec is None or spec.origin is None:
            raise ValueError(f"Cannot find the origin for module {module.__name__}")
        return os.path.dirname(spec.origin)
    return module

Pros:

• Handles namespace packages better.
• More informative and robust for finding the module’s location.

Cons:

• Slightly more complex.
• Still may not handle all edge cases, such as certain dynamically generated modules or unusual packaging methods.

2. Using pkgutil Module

The pkgutil module can be used to get the loader for a module, which in turn can provide the filename.

import pkgutil
import os

def get_module_path(module):
    if isinstance(module, ModuleType):
        loader = pkgutil.get_loader(module)
        if loader is None or loader.get_filename() is None:
            raise ValueError(f"Cannot find the filename for module {module.__name__}")
        return os.path.dirname(loader.get_filename())
    return module

Pros:

• Works well with traditional and namespace packages.
• Provides a clear error message if the module cannot be found.

Cons:

• Similar complexity to importlib.util.find_spec.
• May not handle all edge cases perfectly, such as modules loaded in non-standard ways.

Potential Problems with All Solutions

• Edge Cases: Modules loaded in non-standard ways, such as from a zip file, a database, or dynamically created in memory, may not have a traditional filesystem path.
• Environment Specific Issues: Different environments (e.g., virtual environments, Docker containers, cloud functions) may have specific quirks that affect module loading and file paths.
• Performance: For large packages or complex environments, locating the module's path might introduce some performance overhead, especially if done repeatedly.

Conclusion

The initial approach using os.path.dirname(pkg_dir.__file__) is often sufficient but can be improved for robustness by using importlib.util.find_spec or pkgutil.get_loader. These alternatives handle a broader range of scenarios but still may not cover all possible edge cases. The choice of method depends on the specific requirements and constraints of the environment in which the code is running.

[thorwhalen]

Getting versions

Here's a function that gets four opinions of versions.

from wads.pack import versions_from_different_sources

versions_from_different_sources('dol')

{'tag': '0.2.49',
 'current_pypi': '0.2.49',
 'highest_not_yanked_pypi': '0.2.49',
 'setup_cfg': '0.2.49'}

Implementing these rules for validation in validate_versions