Content Downloading rework. #364
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…oader and DepotDownloaderException. Add kDoc. Change DepotDownloader to use a fifo based system to download content in order of add.
…rialization, and update sample.
…t code. This may have unforseen issues, but stops some games from failing because depot 1523211 doesnt have a code.
…mContent. Update ProtoParser to support webui protobufs. All in hopes to being p2p downloads to depot downloader.
…ove network utilization
…g other new thread" This reverts commit 110abe9.
Refactors depot downloading to use separate coroutines for network chunk processing and file writing, improving concurrency and overall download speed. This change introduces dedicated coroutine workers for network chunk downloads and file writing, managing these tasks in parallel. It also ensures proper cleanup and channel closing, preventing resource leaks and ensuring all workers complete their tasks before the download is considered finished. Additionally, it improves the handling of last depot scenarios.
Migrates from Channels to Flows for chunk processing to improve concurrency and simplify data flow management. This change introduces dedicated flows for decompression and file writing, enabling better control over resource utilization and error handling. It also decouples network I/O from decompression and writing operations by processing in parallel, preventing potential bottlenecks. Increases chunk processing throughput and reduces latency by better utilizing available CPU cores.
Refines exception handling for chunk downloads in the CDN client. Adjusts the logic to validate destination buffer size based on whether a depot key is provided.
Imports the necessary Kotlin flow operators for the 'DepotDownloader' class to resolve compilation errors and ensure proper data stream processing.
Changes the dispatcher used for the download task to IO. This ensures that download tasks, which are I/O bound, run on a dispatcher that is optimized for I/O operations, preventing potential blocking of the default dispatcher.
Ensures that the downloader is cancelled when its parent job is cancelled, allowing for better control and resource management in complex workflows.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR completely redoes the ContentDownloader feature for JavaSteam.
Notables changes:
javasteam-depotdownloader)Drafting this for now to let some Android related projects try it out and report any issues before finalizing.Checklist