Great expansion of statistics capabilities

[ArthurKoba]

Is your feature request related to a problem? Please describe.
The functionality of statistics is currently very limited, and little data is provided for analytics.

I suggest adding sections to the statistics menu to make interacting with it more convenient. In the "General" section, you can place the statistics that are currently displayed without specifying a specific node in the selector. Remove the selector itself and move it to "Nodes", which will display node statistics + what I will describe next for improvements. And add the "Traffic by users" section, which is sorely lacking, I will also describe it further.

Describe the solution you'd like
Nodes. We need to expand the use of nodes (node_usages in the database) both in terms of the duration of data storage (the parameter must be configured via the panel) and in terms of the data being collected (CPU max and avg usage, RAM max and avg usage), only uplink and downlink are currently being collected. This will allow you to better understand what is happening with the nodes and see where things are overloaded at the basic level. Of course, grafana + prometeus + node exporter is better suited for advanced analytics. I would make the graphs for specific nodes a little more configurable so that it would be possible to specify some specific samples on them, for example CPU max + CPU avg, or CPU avg + RAM avg, or CPU. It would also be nice to add a comparison graph for non-slip nodes, where you can also point to the displayed parameters, as well as select the nodes to be added.

Traffic by users. A very necessary section in which it will be possible to analyze precisely which users use which nodes, and how much traffic they consume. So far, there is no exact idea of what this section should look like, but what exactly the section should include:

1. Choosing one user for analytics, if we want to test only him specifically. For example, User1 uses Node1, Node2, Node3 and outputs a configurable consumption graph.
2. Choosing a specific server for analytics, if we want to check which users used how much traffic. By the way, it is worth adding here that I would like to see a division into download and upload, now this parameter is summed up as far as can be seen from the database, but I will write about this again later.
3. Indication of time intervals: for all time, for periods, as well as a point, from a specific time to a specific one. The data collected in the database may not even require modifications to display this data, unless, of course, you add a division into download and upload, but if you do, then this parameter should also be added (all, download and upload).
   the last day should not be included in the fall or rise, it is not informative.
4. Specify user filtering options (active, disabled, pending).
5. Detailed 1 and 2, which initially shows all users, and if you click on some, a nested list of traffic usage by servers is revealed.

In principle, this is probably sufficient functionality, but it is potentially possible to expand this list, again, I suggest we discuss this.

Here are the comments I would definitely make:

1. I would remove some users from the statistics, for example, bridge users. Such users are used in bridged (chained) configurations. For example, a regular user connects to the input node, and then his traffic is transmitted via a bridge to the output node. The traffic of such bridge users is usually very high and is rarely useful for statistics. To configure this parameter, I suggest adding two parameters to the user: display in the "Nodes" statistics section and display in the "Traffic by users" statistics section. And add an overload parameter to the "Nodes" and "Traffic by users" sections, so that even for users whose display is turned off, it can be turned on. This would be convenient in rare cases when you want to see the full picture, but it is not convenient to search for all such users and change their parameters.
2. I would remove some nodes from the statistics in the Nodes and General section, because it often becomes necessary to use transit, bridged nodes that can be used for chain security. For example, in the Russian Federation, supervisory authorities now monitor the use of VPN services through users' applications, and if the user's IP address is foreign when connecting to government services, they begin to monitor it. If the suspicion of using a VPN is confirmed, government services block the address of this VPN service and it is no longer possible to connect directly to it. But if you have a foreign bridge server and run traffic through it, then when the output is blocked, the connection remains. Such servers, as well as bridge users, transmit a large amount of traffic and break the overall statistics. For example, there may be 2-3 countries with output nodes and you want to estimate usage distributions, but bridge nodes usually overlap the display and make the graph look lovely. To solve this problem, you need to add one parameter to the nodes - display it in statistics. And also add an overloading parameter in the statistics itself, so that in rare cases, without setting the parameter, all nodes can be temporarily enabled in each node.
3. The division of traffic in the stored rows into download and upload, in general, I think it is not difficult to implement, however, disk memory consumption will increase by about 20%.
4. For the node, we would like to be able to adjust the colors for display on the graphs, when they are calculated randomly, we have to compare the color and the server each time, which causes additional cognitive load. To do this, add the "use color in statistics" parameter to the node, which is random by default. If it is random, when drawing the graph, you need to select the color so that it does not overlap with the existing ones.

Devices. While writing and reviewing the dashboard, I realized that HWID and UserAgent statistics would also be useful. I suggest adding a separate section for these parameters, where you can see the distribution by UserAgent, HWID devices (system, versions, languages, etc.), and the time of the last update (subscription usage).

I've probably covered everything in general, but another thing that would definitely be interesting to add is inbound statistics, since when using bridges, statistics are only displayed for the input of a specific node, but it's unclear where the user's traffic goes next, or to which node specifically... But I think this should definitely be put on hold for now.

Additional context
I've probably covered everything in general, but something else that would definitely be potentially interesting to add is inbound statistics, since when using bridges, statistics are only displayed for the input of a specific node, but it's unclear where the user's traffic goes next, or to which node specifically... But I think this should definitely be put off for now.

Связанные issue:

1. HWIDs monitoring #516
2. Feature: Per-node traffic limits for users #212 - Immediately after the implementation of this functionality, it is possible to limit user traffic by domain relatively quickly, a related problem.

[mustfqyl]

I completely agree, the statistics page is currently very dysfunctional and needs improvement. The "Traffic by users" section is particularly useful because it makes it easier to track users who are using excessive bandwidth.

Also, it would be great to add a feature to the panel, replacing the "notifications" section, that allows users to specify a limit for excessive node usage and receive notifications when that limit is exceeded.