Repository navigation
Architecture
FCS has been developed as a Software as a Service (SaaS) run in the Service Oriented Architecture. The platform may be accessed as a web application or via REST API.
Architecture of FCS may be divided into few fundamental parts: Management module, Task Servers, Crawling Units and Autoscaling module.
Management module was designed as a web application responsible for basic user accounts management and handling requests from clients. Users are also able to control their crawling tasks and modify tasks' parameters. Management module enables clients to download their crawled data and send feedback with evaluation of crawling results. Tasks management via REST API is also possible. All users are authorized and their requests are authenticated with OAuth2 Token.
Each of Task Servers is responsible for handling just one crawling task at the same time. Division into Task Servers is purely logical - many of them may be run on one physical machine. Each Task Server owns databases for storing both links to be processed and crawled data. Management module communicates with Servers via REST API. Server API provides similar operations as Client API but users are not allowed to use it directly.
Every client is responsible for cyclical collecting data acquired by service. When user receives its data, Task Server dedicated to a given task can free its storage.
Crawling Units execute tasks requested by various clients. A single Crawling Unit is able to perform simultaneously many tasks. Each Crawling Unit receives from a Task Server a package of links and returns retrieved data. According to the idea of autoscaling, each Crawling Unit is run if and only if it is required (in order to ensure both guaranteed efficiency and minimal costs).
FCS was designed to be an autoscalable platform in order to provide guaranteed efficiency regardless of the number of clients. Besides, the autoscalability ensures the reduction of costs. Autoscaling module analyses Crawling Units efficiency and exploited bandwidth and on the basis of these data, manages the amount of Crawling Units depending on the current clients' needs. In this way, guaranteed efficiency is kept at the stable level. To reduce running costs, Autoscaling module frees unused resources and shut unused Crawling Units down.
The presented approach justifies an application of cloud computing power on-demand technique. Each Crawling Unit may work in a cloud on a separate machine and exploit paid resources if and only if they are needed. The idea of using cloud computing power on-demand in crawling tasks was a very important factor that motivated us to build Focused Crawling Service platform. Nevertheless, this feature has not been applied yet and is the part of the future work (see: Future works)
Thus FCS guarantees the SLA mechanism what means that each client is provided with a guaranteed amount of resources and the minimal, but also the maximal (defined in each user's quota), efficiency of crawling process.