[Proposal]_Implement_LakeCtlWrapper_based_on_lake-sdk_py

[Lypsolon]

Summary

Currently, the LakeCtlWrapper is implemented by running a subprocess and accessing the lakeCtl binary.
This was a good approach, especially for testing, as it was easy and fast, but it is not Pythonic and is a bit OS-specific.

This proposal describes what needs to be done to implement the wrapper functionality using the lake-sdk Python setup.

Description

Outile.

• the current LakeCtlWrapper has been built in a way that we can replace the binary with lake-sdk_py function calls without having to rewrite any external code (ayon_usd_addon, ci_tools)
• there are 2 options to get the lake-sdk_py into the repo. Vendors and as a dependency
  • vendors are easier regarding our dependency package creation tool.
  • as a dependency is a cleaner setup, we can't change things in lake-sdk_py if we wish.
    We could also "vendor" the lake-sdk_py source code and write the wrapper as an extension, but this would mean we are handling a complete package instead of "just" a wrapper.

In the end, the lake-sdk_py has all the features we need. We need to wrap them into a single class that acts a bit more like the CTL.
We should also implement a single version of the same wrapper class. This would be great for some add-ons. It would allow a developer to be sure there is no second instance and no risk when changing or accessing data from it.

Approval

Automated

• Running the current Automated tests
• Extending tests to prove behaviours to be correct
• Extending test to prove singleton behaviour.

Packages

• Ayon_Usd_Addon needs to agree to the changes
• A test implementation into Ayon_Automator should be present (to test against package upload)
• A test implementation of Ayon_Automator using the new setup should be done (to prove that a package can be developed end to end; it could be Ayon_Usd_Bin_Package or Ffmpack)

Devs

• antirotor (as head of pipeline)
• martastain or 64qam (should agree to the behaviour as this tool would upload to a cloud system that is partially handed by them)

I don't think there is a point in having Client Success or QA sign off on this, as it is purely in the background and should only be touched Programmatically.

To Reproduce (if issue/bug/etc)

No response

User Stories

As a developer, I would like to run the wrapper on any operating system without carrying around the binrays for the Lake CTL.

As a developer, I would like full Pythonic access to the lakeCtl
As a developer, I would like a clean and controllable implementation with full call stacks, dock strings and data as class behaviour instead of data streams from a subprocess

Impact

Low

Assumed Complexity

Days

Impact Scale

an entire tool group

Other tools that get touched

• Ayon_Automator
• Ayon_Usd
• Ayon_Resolver
• Every package that needs to be uploaded to lakeFs

How it Solves the Problem

A full Python implementation allows us to

• remove the binaries
• pass data in a clean and Pythonic way
• have better call enforcement
• have everything in functions and thereby have better tracebacks and hints.

Implementation Idea/Details

The lakeCtl._run() func would be replaced by the associated function calls in the lake-sdk_py

Implementing a convenient upload function would probably be a good idea / necessary as the default system is a bit involved, and the LakeCtl wrapper is a class that already stores most of the needed info for it.

‎LakeCtl.construct_repo_info (NEW)

• takes the path to a lakeFs repo and constructs all the info needed for uploading, downloading, creating a commit, pulling updates

‎LakeCtl.upload_objects (NEW)

• uploads a single or a list of objects to the lakeFs server.
• lake-sdk_py ref
  • Commit commit(repository, branch, commit_creation, source_metarange=source_metarange)
  • ObjectStats upload_object(repository, branch, path, if_none_match=if_none_match, storage_class=storage_class, force=force, content=content)

‎LakeCtl.list_repo_objects (PRESENT)

• lake-sdk_py ref
  • RefList list_branches(repository, prefix=prefix, after=after, amount=amount)

‎LakeCtl.clone_local (PRESENT)

• this could be implemented as a parallel download from the lake fs server via list objects and get objects. PS: as far as I'm aware, there is no clone-repo function (probably because it is not a low-level function but more a collection of functions)
• lake-sdk_py ref
  • list_objects
  • get_object I'm not fully sure if this is the right function for the job.

LakeCtl.clone_element (PRESENT)

• this could be a simple get_object; the nice thing is the get object would get all its config from the class instance, so we only need the lake_path and dst path to clone
• lake-sdk_py ref
  • get_object I'm not fully sure if this is the right function for the job.

‎LakeCtl.commit_local (PRESENT)

• This would be a collection of upload_objects, creating a commit on the server.
• lake-sdk_py ref
  • commit

LakeCtl.init (PRESENT)

• this would stay mainly the same; it would just need to configure the lake-sdk_py objects instead of storing the data for the lake binary.

lakeCtl._run (REMOVED)

• could be removed as it was only used to access the lakeCtl bin as a subprocess

Technical Approach

No response

Architecture

The general architecture could change in the following ways.

• The Python calls would end up being serial and blocked. So, we must consider using the work_hanlder to allow for parallel processing. But this would also mean we must implement fences so other functions can wait until completion.

• Class construction and function class signatures should not change as this would cause a refactor of independent projects.

• import scale/dependencies could be created if we don't vendor the lake-sdk_py code. over all the amount off imports would increase and startup for the package would increas

Unresolved Questions

• It is currently unclear what the performance impact of this implementation will be.
• It is unclear how dispatching should be handled for parallel up and downloads. The parent function calls might already be dispatched; this should not create a problem, but it should be tested just to be sure.
• It is unknown if we need to change the function calls signatures, thereby causing a need to change tools that depend on this tool. this should be avoided at all costs but should be taken into consideration; we will know more when creating an example implementation.

milestones

• Decide if dependencies should be vendors or used as a dependent package (PyPI, etc.)
• create a scratch example implementation; assume this will be rewritten
• test the performance impact against the current tooling
• extend the tests to cover all new and or missing features
• create an in-depth implementation
• create an example implementation into Ayon_Automator
• create an example implementation off the updated Ayon_Automator into (ayon_usd, Ffpack, ayon_usd_resovler)

Potential Problems

• performance issues
• missing or weakly implemented functions
• function call signature changes and there by the need to refactor dependent tools

Long-term Issues

• As this library is very thin in operation, I will assume that many of the problems will appear very directly.

• But I will mention that changes in lake fs or their Python wrappers could cause major changes to this library in general, and I'm not sure how much that could be the case with the go/binary lakeCtl implementation

Transient Problems

No response

impacting Branches

ayon-usd

• for data download

ayon-automator

• for data upload

Dependencies

lakeFs-Server
ayon-bin-bridge-server

outgoing Dependencies

ayon-automator
ayon-usd

dependency Issues

No response

Dependant Branches

No response

Common Dependencies

lakefs-sdk

non-persistent test (e.g. development time tests)

Short-term tests for example, implementation could use a locally hosted lakeFs server and the ayon_usd addon for download and a custom bin package created from the ayon_usd_bin_package (the package that ayon_usd addon is downloading parts of)

Testing Plan (president tests)

• create a docker-compose setup that starts a lakeFs server for testing and a second docker that runs the tests.
  • the sample data on the lakeFs server should be generated and not stored so that we can have a large number of objects without storing them all in the repo
• the baseline tests should be
  • create a package and upload it
  • download the package and commit something new to it
  • get data from the package repo and compare it with the expected data (we know that from before we upload)
  • clone an entire repo
  • make changes to the cloned repo and upload + commit it again. Then check if the data is correct

Documentation

Documentation generated from code would be enough as this will never be used manually.

• add docstrings
• add type hints
• auto-generate documentation output via sphinx (integrated via Ayon_Automator)

but we could add the following handwritten examples

• explain how to upload, download, commit, get data, configure (this would just be the tests in written form)

Maintenance

The system will be low in maintenance if the tests are extensive enough. Most of the maintenance would be reactive to changes in lake fs.
the maintenance could be of two parts

• run tests periodically ( to ensure that changes in lake fs or other dependencies will be found early on )
• manual code changes if a test fails.

It would also be advised to freeze and couple the ayon_bin_bridge_client and the ayon_bin_bridge_server so that none of them can be out of sync. This could be done via an action that simultaneously forces you to create a release for both of them.

Risk Management

There are a few directly visible risks

• Changes in function call signatures would cause refactors if they are not avoidable. I'm afraid there is no way around this. This can only be prevented by creating a good example implementation and then deciding if it is worth the risk.
• performance issues: this will be avoided via extensive testing
• lake fs changes killing our implementation: this will be avoided by freezing the server and client into a single package so that they cant be out of sync, as an addition it would also be advised to always stay a version behind latest if no security reason forces the update

Appendices

No response

Is there an existing issue with this?

• I have searched the existing issues

Are there any labels you wish to add?

• I have added the relevant labels to the bug report.