Distribute-Compressor

Distribute-Compressor is a project suite that enables recursive directory traversal and file compression into ZIP archives. It utilizes a task queue system to distribute compression tasks across DC-Handler instances, supporting multiple queue implementations including RabbitMQ and a planned DC-Queue (custom solution).

Components

DC-Interface

The control program that:

1. Recursively scans directories
2. Converts file metadata to JSON
3. Manages queue communication
4. Receives processed results

Using

Before you send files to queue - you should set parameters for queue. For instance, if you use RabbitMQ - set host, username and password. There are different ways to do this.

You can set parameters through arguments of command line, like:

distributed_compressor_interface --queue-type RabbitMQ --host example.com --user admin --password admin

or the same

distributed_compressor_interface --queue-type RabbitMQ --host example.com --user admin --password admin

DC-Interface saves last configuration in config-file. It allows you don't repeat queue-parameters in the next program launch.

All flags for DC-Interface:

Flag	Full Named Flag	Meaning
-QT	--queue-type	set queue-type [RabbitMQ, DC-Queue]
-H	--host	set hostname of Queue
-U	--user	set username of Queue profile
-P	--password	set password of Queue profile
-RC	--ResetConfig	reset config-file

Next, you can launch distributed_compressor_interface:

distributed_compressor_interface ~/Desktop/exampleDirectory

If you don't send parameters or set not all parameters, you should set the absent parameters in interactive poll. Next this parameters will store in config-file(in ~/.config/... directory on Linux and in AppData on Windows).

DC-Handler

DC-Interface if program that uses for processing tasks, that received from Queue. It perform compressing and send result to result-queue.

Using

To start a DC-Handler:

distributed_compressor_handler

The program will prompt for required parameters in interactive mode.

Alternatively, you can provide all parameters directly:

distributed_compressor_handler --queue-type RabbitMQ --host example.com --user admin --password admin

Flags:

Flag	Full Named Flag	Meaning
-QT	--queue-type	set queue-type [RabbitMQ, DC-Queue]
-H	--host	set hostname of Queue
-U	--user	set username of Queue profile
-P	--password	set password of Queue profile

Note: These parameters are not stored in a config file to allow running multiple handler instances on the same machine.

DC-Queue

Custom implementation of Task-Queue.(Not implemented yet)

Building

This project requires some additional libraries:

1. Boost
2. LibArchive
3. AMQP-CPP

Before building, you should install these dependencies. For example, use apt install libarchive-dev libboost-all-dev. It is recommended to build like dynamic library AMQP-CPP from source (https://github.com/CopernicaMarketingSoftware/AMQP-CPP).

Next, go to the root directory of the project and build the project using CMake.

Project Structure

tree-structure:

.
├── CMakeLists.txt
├── DC-Handler
│   ├── CMakeLists.txt
│   ├── include
│   │   ├── CompressHandler.hxx
│   │   ├── DC_Handler_ConfigLoader.hxx
│   │   └── TaskExecutor.hxx
│   └── src
│       ├── CompressHandler.cxx
│       ├── DC_Handler_ConfigLoader.cxx
│       └── main.cxx
├── DC-Interface
│   ├── CMakeLists.txt
│   ├── include
│   │   ├── DC_Interface_ConfigLoader.hxx
│   │   └── FileSeeker.hxx
│   └── src
│       ├── DC_Interface_ConfigLoader.cxx
│       ├── FileSeeker.cxx
│       └── main.cxx
├── DC-Queue
│   ├── include
│   └── src
│       └── main.cxx
├── images
│   └── UML_diagram.png
├── include
│   ├── CMakeLists.txt
│   ├── ConfigLoader.hxx
│   ├── JSON_FilePacker.hxx
│   ├── nlohmann
│   │   └── json.hpp
│   ├── queue_input_strategy
│   │   ├── DC_Queue_QueueInputStrategy.hxx
│   │   ├── QueueInputStrategy.hxx
│   │   └── RabbitMQ_QueueInputStrategy.hxx
│   ├── QueueParams.hxx
│   ├── RabbitMQ_TaskQueueClient.hxx
│   ├── TaskPool.hxx
│   └── TaskQueueClient.hxx
├── README.md
└── src
    ├── CMakeLists.txt
    ├── ConfigLoader.cxx
    ├── RabbitMQ_TaskQueueClient.cxx
    ├── TaskPool.cxx
    └── TaskQueueClient.cxx