Docker Logs Retention

Legends

Log Rotation Using JSON file format

To setup rotating logs, add the following lines in /etc/docker/daemon.json file (create the file if it doesn’t exist already) and restart the docker daemon using systemctl restart docker on all instances. In the following configuration, the maximum file size is kept to 20 MB for a maximum of 3 files.

{
"log-driver": "json-file",
"log-opts": {
   "max-size": "20m",  # Max size of the log files.  
   "max-file": "3"     # The maximum number of log files that can be present.
   }
}

Please do not remove already existing ( if any ) lines in the /etc/docker/daemon.json file, as it may corrupt the docker deamon configuration. Consult Expertflow personnel for help on this.

Verifying Logs

Logs for each container are available in files as well as within the docker daemon. To see the logs for any container, execute docker ps, and get the id of the container. Use that id to see the logs using docker logs container_id. At any given time, the active machine would be the one with the highest keep alive priority, so the logs would be stored there.

Logs aggregation on the host

Container logs are flushed on the container restart. This section covers guidelines for aggregating logs of Expertflow components on the host.  All logs are forwarded to the host system's rsyslogd process. Rsyslogd aggregates logs based on container names and write them to respective files. These log files will be rotated by another application called logrotated to maintain a minimum of the last 30 files.

Requires Host Reboot

A system reboot is required after these configurations. 

1. Create a root log directory on the host. 

   BASH

   mkdir /var/log/dockerlfs

2. make a copy of the currently running rsyslogd.conf using 

   BASH

   cp /etc/rsyslog.conf{,-backup}

3. in /etc/rsyslog.conf 

   1. add/update contents to look like below ( remove # at the start of these lines as they are already present in the file)  

      CODE

      $ModLoad imtcp   
      $InputTCPServerRun 514

   2. Add/enable these parameters  in /etc/rsyslog.conf

      /etc/rsyslogd.conf

      BASH

      $ActionQueueFileName fwdRule1
      $ActionQueueSaveOnShutdown on
      $ActionQueueType LinkedList
      $ActionResumeRetryCount -1
      $imjournalRatelimitInterval 0
      $imjournalRatelimitBurst 0

4. Create a new file /etc/rsyslog.d/docker_daemon.conf and add below the given contents. 

   /etc/rsyslog.d/docker_daemon.conf 

   BASH

   $template DockerLogs, "/var/log/dockerlfs/daemon.log"
   if $programname startswith 'dockerd' then -?DockerLogs
   & stop

5. Create a new file /etc/rsyslog.d/docker_container.conf add below-given contents.

   /etc/rsyslog.d/docker_container.conf

   BASH

   $template DockerContainerLogs,"/var/log/dockerlfs/%hostname%_%syslogtag:R,ERE,1,ZERO:.*container_name/([^\[]+)--end%.log"
   if $syslogtag contains 'container_name'  then -?DockerContainerLogs
   & stop

6. Reconfigure journald - edit /etc/systemd/journald.conf and add/enable.

   CODE

   RateLimitInterval=0

7.  For log rotation, reconfigure. create a new file /etc/logrotate.d/dockerlogs and add the following.

   /etc/logrotate.d/dockerlogs

   BASH

   /var/log/dockerlfs/*.log {
     copytruncate
     compress
     dateext
     size 20M
     daily
     dateformat -%Y%m%d%H%s
     missingok
     rotate 30
   }

8. Restart system services.

   BASH

   systemctl restart rsyslog
   systemctl restart systemd-journald

9. Check the status of configurations by 

   BASH

   systemctl status rsyslog
   
   ● rsyslog.service - System Logging Service
      Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2020-06-16 15:41:41 PKT; 6h ago
        Docs: man:rsyslogd(8)
              http://www.rsyslog.com/doc/
    Main PID: 4161 (rsyslogd)
       Tasks: 8
      Memory: 12.8M
      CGroup: /system.slice/rsyslog.service
              └─4161 /usr/sbin/rsyslogd -n
   
   Jun 16 15:41:41 port1 systemd[1]: Starting System Logging Service...
   Jun 16 15:41:41 port1 rsyslogd[4161]:  [origin software="rsyslogd" swVersion="8.24.0-52.el7" x-pid="4161" x-info="http://www.rsyslog.com"] start
   Jun 16 15:41:41 port1 systemd[1]: Started System Logging Service.

10. Run ncat 

    ncat

    BASH

    nc -zvw 3000ms  localhost 514
    Ncat: Version 7.50 ( https://nmap.org/ncat )
    Ncat: Connected to ::1:514.
    Ncat: 0 bytes sent, 0 bytes received in 0.05 seconds.

    or use lsof the command for the same purpose 

    lsof

    BASH

    # lsof -i:514

11. Edit Docker daemon /etc/docker/daemon.json and add the following code snippet to enable forwarding container logs to the host. 

    /etc/docker/daemon.json

    CODE

    {
    	"log-driver": "syslog",
    	"log-opts": {
    	"syslog-address": "tcp://127.0.0.1:514",
    	"tag": "container_name/{{.Name}}",
    	"labels": "production",
    	"syslog-facility": "daemon"
    	}
    }
    

Please do not remove already existing ( if any ) lines in the /etc/docker/daemon.json file, as it may corrupt the docker deamon configuration. Consult Expertflow personnel for help on this.

1. Restart the Docker services 

   BASH

   // shutdown the solution 
   efutils all down
   // bring the solution up again 
   systemctl restart docker
   efutils all up 
   
   

Validate Configurations

File names may differ based upon the context and project name used, but the last part of file names will identify the component name.

1. Restart the application stack to start sending their corresponding logs to the Docker daemon, which should now forward all container logs to the host. 

   BASH

   cd /var/log/dockerlfs/
   
   ls -ltr

   CODE

   total 11768
   -rw-------. 1 root root    6091 Jun 16 20:15 localhost_minio_1.log
   -rw-------. 1 root root    5281 Jun 16 20:15 localhost_bot-connector_1.log
   -rw-------. 1 root root   55335 Jun 16 20:15 localhost_ecm-services_1.log
   -rw-------. 1 root root   46237 Jun 16 20:16 localhost_mysql_1.log
   -rw-------. 1 root root   58450 Jun 16 20:16 localhost_database-connector_1.log
   -rw-------. 1 root root   85133 Jun 16 20:16 localhost_mre-microservice_1.log
   -rw-------. 1 root root   31446 Jun 16 20:16 localhost_activemq_1.log
   -rw-------. 1 root root   31465 Jun 16 20:17 localhost_ccm_1.log
   -rw-------. 1 root root   40939 Jun 16 20:19 daemon.log
   -rw-------. 1 root root  218661 Jun 16 20:19 localhost_eabc_1.log
   -rw-------. 1 root root   82426 Jun 16 20:19 localhost_ecm_1.log
   -rw-------. 1 root root  141140 Jun 16 22:05 localhost_mre-frontend_1.log
   -rw-------. 1 root root   86612 Jun 16 22:05 localhost_customer-gadget_1.log
   -rw-------. 1 root root   78237 Jun 16 22:05 localhost_ecm-frontend_1.log
   -rw-------. 1 root root   89137 Jun 16 22:05 localhost_agent-gadget_1.log
   -rw-------. 1 root root   37694 Jun 16 22:05 localhost_file-engine_1.log
   -rw-------. 1 root root   76818 Jun 16 22:05 localhost_mongo_1.log
   -rw-------. 1 root root 3318065 Jun 16 22:06 localhost_umm_1.log
   -rw-------. 1 root root 1473795 Jun 16 22:06 localhost_comm-server_1.log
   -rw-------. 1 root root 2558706 Jun 16 22:06 localhost_mre_1.log
   -rw-------. 1 root root 1538594 Jun 16 22:06 localhost_chat_1.log
   -rw-------. 1 root root 1538594 Jun 16 22:06 localhost_app-gateway_1.log

2. Tail one or more files to see logs output 

   BASH

   tail -f /var/log/dockerlfs/{localhost_file-engine_1.log,localhost_comm-server_1.log,localhost_umm_1.log}
   tail -f /var/log/dockerlfs/* for all logs to be tailed on your screen/terminal