REMP Tracker and Segments - Log errors to the right channel

[Andre-Amaro-Matias]

Description

The REMP Segment and Tracker modules are logging both error and info messages to the same file (using the same facility), which makes debugging and error checking almost impossible, due to the verbosity of the info messages.

What is happening

1. Using /var/log/messages (syslog dafault location) for log lines from segments and tracker;
2. Both INFO and ERROR severities are logged to this log file.

Impact of this issue

Right now, we've changed the start-up script to log STDOUT to a file and STDERR to another. Problem is, the application logs are being written to STDERR only, even when it's an info message.
Because of this, we have the following logfiles:

[PROD] [root@REMP ~] $ ls -alkhtr /var/log/tracker/
total 849M
-rw-r--r--   1 nginx nginx    0 Oct 25 17:40 tracker.out
-rw-r--r--   1 nginx nginx 116M Nov 21 00:01 tracker.err-20191121.gz
-rw-r--r--   1 nginx nginx 132M Nov 22 00:01 tracker.err-20191122.gz
-rw-r--r--   1 nginx nginx  91M Nov 23 00:01 tracker.err-20191123.gz
-rw-r--r--   1 nginx nginx  57M Nov 24 00:01 tracker.err-20191124.gz
-rw-r--r--   1 nginx nginx 298M Nov 25 00:01 tracker.err-20191125
drwxr-xr-x   2 nginx nginx  196 Nov 25 00:01 .
drwxr-xr-x. 20 root  root  4.0K Nov 25 01:01 ..
-rw-r--r--   1 nginx nginx 112M Nov 25 11:47 tracker.err

What should be happening

There should be an application-wide configuration to output the messages (info) and the errors (warning,error,etc.) to different locations. Additionally, logging should follow the user's preferences when it comes to the minimum severity to which log to file.

1. There could be a conf file to specify a file path to the desired location.
2. There could be an option in the conf file to select the desired loglevel of the messages to write to file.

Version used
https://github.com/remp2020/remp/tree/0.9.1

[rootpd]

Hi Andre,

thanks for the extensive report, we'll take a look on why the info messages are being passed to stdout.

There's however already a possibility to configure verbosity of output that could help you. Looking at the file sizes you linked, you probably kept your applications configured on the debug level.

In your .env file (for both Tracker and Segments) or in your environment, please set following environment variable values:

• Segments API:
  • SEGMENTS_DEBUG=false
• Tracker API:
  • TRACKER_DEBUG=false

Then, only the startup information and possible future errors will be printed.

[rootpd]

When you do the change, please restart the applications - we haven't implemented live-reload of configs yet.

[Andre-Amaro-Matias]

Hey Peter,

Thanks for the quick reply. Right now, we're redirecting the output of the process itself (and the severity it uses) using the following method on the unit file:

• For SEGMENTS:
  • ExecStart=/usr/bin/bash -c 'exec /var/www/remp/lib/tracker/segments >>/var/log/segments/segments.out 2>>/var/log/tracker/segments.err'
• For TRACKER:
  • ExecStart=/usr/bin/bash -c 'exec /var/www/remp/lib/tracker/tracker >>/var/log/tracker/tracker.out 2>>/var/log/tracker/tracker.err'

I'll try and change the values on the .env file and report back to you.

Thanks again!