@shykes - what is the recommended way to manage logs? In this case, postgres would generate logs that need to be rotated (with a sighup to parent process). I'm not sure if I can use the host's logrotate to do that.
Currently I build containers with supervisord as pid 1 and logrotate running inside the container,but with logs being saved to a bind mounted volume.
Is this correct?
P.S. there are no docs, blog posts or articles on this topic. I'm a little puzzled if people are living with ephemeral logs.
There are actually a lot of blog posts and docs on this subject. Previous posters already explained some of the solutions, but I was just searching on this topic this morning so I thought I would share what I found:
It seems that the recently recommended setup is to create a container specifically to host volumes for both logs and other persistent data (ie database files). You then connect each container that needs to write to those volumes using the volumes-from directive. This is explained in blog posts and included in the documentation.
You can run your docker container supervised with tools like systemd, runit etc - and have stderr/stdout (and signals) forwarded on from the container process to the supervisor - from then on you do what you would normally do if it wasn't running in docker.
Other approaches are to have the app(s) in container log to a VOLUME that is bind-mounted in from the underlying host (where you can access them directly) - yet another approach is to bind mount in syslog or other tools into the container and allow the process(s) inside the container to log to it. All work well.
Docker captures stderr; you can configure Postgres to simply use stderr logging, and then capture it in the host system.
Another option is syslog, which Postgres also supports, with which you would log to the host's syslog daemon. (You can mount the host's /dev/log if you don't want to deal with setting up the networking.)
If you're sending logs out to the host's syslog, then the log creator is your syslog daemon.
Also, processes in containers exist as processes on the host (they have different PIDs inside and outside of the container due to PID namespacing); so logrotate should be able to send signals to containerized processes.
hey thanks for that comment - could you talk about it in a slight more detail.
I have rsyslog on the host and I'm sending logs to host syslog through bind mounting of /dev/log . Is this what you meant by sending out logs to host ? I'm having a lot of trouble figuring out how to do it any other way.
Could you also confirm that rsyslog on host will be able to send appropriate signals to the container's syslog ... which forwards it to the containerized process ? I'm unable to find documentation in rsyslog (or syslog-ng) that talks about this behavior so I'm not sure.
Sighup is needed when processes (such as Postgres) write directly to log files. These files are open, and when you want to rotate the file, you have to tell the process to close the files so that it will start writing to a new file.
If you tell Postgres to log to rsyslog (or any other syslog daemon), the log data will be sent to rsyslog via UDP, TCP or a Unix socket. Postgres itself will not have any files open, so there is no need to sighup it.
You will have to sighup rsyslog in the host system, though.
If you bind mount /dev/log of the host into your container, you only run rsyslogd on the host. There's no need to send any SIGHUP to anything in the container.
Currently I build containers with supervisord as pid 1 and logrotate running inside the container,but with logs being saved to a bind mounted volume.
Is this correct?
P.S. there are no docs, blog posts or articles on this topic. I'm a little puzzled if people are living with ephemeral logs.