필사 모드: The complete guide to Linux log operations: tying journald, rsyslog, and logrotate into one system

• Introduction
• 1. The path a log travels
• 2. journalctl — learn how to query first
• 3. Journal retention policy — the settings that protect the disk
• 4. rsyslog — exporting to files and to remote destinations
• 5. logrotate — keeping files from growing forever
• 6. What to keep and how much of it
• 7. Five traps you will hit often
• Quiz: check your understanding
• Closing
• References
• Further reading

Introduction

Failures caused by logs wear two faces. Either the log is missing when you need it, or it fills the disk when you do not. Both are the result of running the default plumbing without ever designing it.

This blog has designing structured logging, and that post deals with what an application should record and in which fields. This post is the layer below it. The subject is where the logs an application emits flow on the host, how much is kept, and when it gets deleted. Not the content of the log, but its plumbing.

The scope covers three layers. systemd-journald collects, rsyslog sends to files and to remote destinations, and logrotate rotates the files. The baseline is systemd 250 or newer, rsyslog 8.x, and logrotate 3.18 or newer; differences in paths between distributions are noted as they come up.

1. The path a log travels

On a modern Linux system the starting point for logs is journald in most cases. There are four inputs.

• The kernel ring buffer (/dev/kmsg)
• The syslog socket (/dev/log)
• The native journal API
• The standard output and standard error of services

That last item matters. Everything a process running as a systemd unit prints to the screen goes into the journal automatically. It means logs are recorded even when the application never writes to a file itself, which also lines up with standard practice in the container era.

There are two paths out of the journal to the next stage.

systemctl status systemd-journald.service
systemctl status rsyslog.service
ls -l /run/systemd/journal/

rsyslog normally either reads the journal through the imjournal module or takes the socket directly with imuxsock. Turning both on at the same time records the same message twice, so you must use only one. Start by checking what your distribution defaults to.

grep -rn 'imjournal\|imuxsock' /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null

The first thing to settle when designing the plumbing is which side is the single source of truth. There are three options. Treat the journal as the original and files as secondary; treat files as the original and keep the journal to a minimum; or keep both and ship everything to a central collector. The third is the most common and also carries the most duplication.

The deciding factor is tooling. If the investigation tools your team uses are centred on journalctl, the journal is the natural original; if you have many file-reading agents or existing scripts, files are the natural original. What matters is writing down which one you picked. If that is not settled, logs pile up in two places in two different formats, and every investigation adds time spent working out which side is correct.

2. journalctl — learn how to query first

Before you change any settings, you should be fluent at querying. Fast queries mean shorter outages.

journalctl -u nginx.service -n 200 --no-pager
journalctl -u nginx.service -f
journalctl -p err -b
journalctl -k -b -1
journalctl --since '2026-08-15 14:00' --until '2026-08-15 14:30'
journalctl -g 'timeout|refused' --since today
journalctl -o json-pretty -n 1
journalctl --list-boots
journalctl --user-unit myapp.service

The precise meaning of each option is as follows.

• -u filters by unit or by pattern. --user-unit is for user session units.
• -p is the priority filter and accepts both names and numbers. The order is emerg(0), alert(1), crit(2), err(3), warning(4), notice(5), info(6), debug(7).
• -b queries by boot. -b -1 is the previous boot.
• -k shows kernel messages only.
• -g applies a regular expression to the MESSAGE field.
• -o is the output format and supports short, short-iso, verbose, json, json-pretty, cat, and others.
• -x appends the explanation from the message catalog. Useful when you hit an unfamiliar error.

The journal holds structured fields, so you can query field by field. That is the decisive difference from a plain text log.

journalctl _PID=1234
journalctl _UID=1000 --since today
journalctl _SYSTEMD_UNIT=nginx.service _TRANSPORT=stdout
journalctl -F _SYSTEMD_UNIT | head -30

-F lists every value present in a given field. You use it to see at a glance which units are producing logs.

3. Journal retention policy — the settings that protect the disk

The default behaviour of journald differs between distributions. The biggest difference is whether the journal is stored persistently on disk.

journalctl --disk-usage
ls -ld /var/log/journal /run/log/journal 2>/dev/null

If the /var/log/journal directory exists the storage is persistent; if it does not and only /run/log/journal is there, storage is volatile. When it is volatile, every earlier log disappears on reboot. That is the common reason nobody can find the cause after a server reboots unexpectedly.

The configuration lives in /etc/systemd/journald.conf or in fragment files under /etc/systemd/journald.conf.d/.

[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=2G
SystemKeepFree=1G
SystemMaxFileSize=128M
MaxRetentionSec=30day
MaxFileSec=1day
RateLimitIntervalSec=30s
RateLimitBurst=10000
ForwardToSyslog=yes

What each entry means.

• Storage=persistent creates /var/log/journal and stores persistently.
• SystemMaxUse is the maximum space the journal as a whole may consume; SystemKeepFree is the free space to leave behind. Whichever of the two conditions is stricter is the one that applies.
• MaxRetentionSec is the retention period. Setting it alongside the size condition is the safer option.
• RateLimitIntervalSec and RateLimitBurst stop a runaway service from filling the disk. This is the key setting that prevents one service flooding the log and pushing out the logs of every other service.

Apply it with a restart.

sudo systemctl restart systemd-journald
journalctl --disk-usage

To shrink a journal that has already grown large, use the vacuum family of commands.

sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-files=5

Destructive command warning: the vacuum commands delete archived journal files. If the records of the incident you are investigating fall inside that range, they go too, so even when the disk is urgent, export the range you need to a file first.

journalctl --since '2026-08-15 00:00' -o export > /backup/journal-20260815.export

You can also verify journal integrity.

journalctl --verify

4. rsyslog — exporting to files and to remote destinations

There are environments where the journal alone is enough, but if you need to integrate with file-based tooling or ship to a remote collector, you need rsyslog.

The configuration structure has three parts: loading modules, rules, and templates. The modern syntax (RainerScript) reads better.

module(load="imuxsock")
module(load="imklog")

template(name="DetailedFormat" type="string"
  string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")

if $programname == 'myapp' then {
  action(type="omfile" file="/var/log/myapp/app.log" template="DetailedFormat")
  stop
}

*.info;mail.none;authpriv.none;cron.none  /var/log/messages
authpriv.*                                /var/log/secure

The default log file paths differ between distributions. RHEL-family systems use /var/log/messages and /var/log/secure; Debian/Ubuntu-family systems use /var/log/syslog and /var/log/auth.log. This is the part that breaks most often when documents or scripts are moved between systems.

Remote forwarding should use TCP and should have a queue in front of it.

action(type="omfwd"
  target="logs.example.com" port="514" protocol="tcp"
  queue.type="linkedlist"
  queue.filename="fwdRule1"
  queue.maxdiskspace="1g"
  queue.saveonshutdown="on"
  action.resumeRetryCount="-1")

Specifying queue.filename lets the queue spill to disk when the memory queue fills. action.resumeRetryCount="-1" means retry indefinitely. This is the minimum configuration that keeps you from losing logs when the collection server dies briefly, and in exchange you have to keep an eye on free disk space. A queue file that grows without limit becomes an incident in its own right.

Validating and applying the configuration goes like this.

sudo rsyslogd -N1
sudo systemctl reload rsyslog
logger -p local0.info 'test message from operator'
sudo tail -5 /var/log/messages

rsyslogd -N1 checks configuration syntax only. Run it before you apply anything, every time. If there is a syntax error, rsyslog does not come up, and every log produced in the meantime is gone.

5. logrotate — keeping files from growing forever

logrotate runs periodically and rotates the files that match its conditions. The configuration lives in /etc/logrotate.conf and under /etc/logrotate.d/.

/var/log/myapp/*.log {
    daily
    rotate 14
    maxsize 100M
    missingok
    notifempty
    compress
    delaycompress
    create 0640 myapp myapp
    su myapp myapp
    sharedscripts
    postrotate
        /bin/kill -USR1 $(cat /run/myapp/myapp.pid 2>/dev/null) 2>/dev/null || true
    endscript
}

The documented meaning of each directive.

• rotate 14: keep files through 14 rotations, then delete.
• daily: rotate once a day. weekly and monthly also exist.
• size: rotate only when the file has grown past the given size. It ignores the time condition.
• maxsize: rotate when the size is exceeded regardless of the time condition.
• minsize: the size must be exceeded, but the time condition must also be met.
• compress: compress with gzip by default.
• delaycompress: defer compression by one cycle. Needed when a process is still holding the file.
• missingok: do not raise an error if the file is absent, just move on.
• notifempty: do not rotate if the file is empty.
• create mode owner group: right after rotation, create a new file with the same name and the given permissions.
• su user group: perform the rotation with the permissions of the given user and group.
• sharedscripts: run the scripts only once even when a wildcard matches several files.
• dateext: append a date to the rotated file name.
• olddir: move rotated files into a different directory.

The most important choice is whether you use create or copytruncate.

With the create mode you have to tell the process to reopen through a postrotate hook. nginx conventionally uses USR1 and many daemons use HUP. If you do not send that signal, the process keeps writing to the renamed old file, and that file becomes phantom capacity that never disappears from df.

copytruncate is the fallback for a program you cannot make reopen its files. Document the fact that it is a choice that accepts the possibility of losing logs.

Always validate with a dry run.

sudo logrotate --debug /etc/logrotate.d/myapp
sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.d/myapp
cat /var/lib/logrotate.status

According to the documentation, --debug (-d) makes no changes at all and does not update the state file either. You can simulate safely with it. -f forces rotation regardless of the conditions and does change files, so use it carefully.

The default location of the state file is /var/lib/logrotate.status. The path can differ by distribution, so check the actual path in the unit file or the cron script.

6. What to keep and how much of it

Policy is harder than the technical settings. Here are the criteria, split along three axes.

Axis 1 — investigability. How many days at minimum do you need to find the cause of a failure. Allowing for a problem that happens over the weekend and is noticed on Monday, seven days is the floor and fourteen is the practical minimum.

Axis 2 — regulation and audit. Authentication logs and privilege change logs frequently have a retention period fixed by law or by internal policy. Separate those logs from the rest and apply their own policy.

Axis 3 — cost. Disk is finite. Rather than keeping the original for a long time, the usual arrangement is short on the original, long in central collection.

A tiered arrangement that reflects those three axes is close to the industry standard.

Sizing capacity is a matter of measurement. Run it for a few days and then calculate.

journalctl --disk-usage
du -sh /var/log
du -x -h --max-depth=1 /var/log | sort -h | tail -10

The arithmetic is simple. Multiply the daily log volume by the number of retention days, allow for the compression ratio, and add headroom on top. Compression generally does a lot for text logs, but the benefit drops once already-compressed data or binary payloads are mixed in, so trust measurements over estimates. And size it for the log volume during an incident, not during normal operation. When something breaks, error logs grow by tens of times, and if the disk fills at that moment you lose the very logs the investigation needs.

An incident where logs fill the disk and take the service down is still one of the most common failure types. Put two lines of defence in place. The first is SystemMaxUse in journald and maxsize in logrotate; the second is separating /var/log onto its own filesystem so it cannot fill the root filesystem. For the diagnostic order on disk capacity problems, see the Linux incident response command guide.

7. Five traps you will hit often

Trap 1 — you deleted the logs and the space did not come back. A process is still holding the file.

sudo lsof -nP +L1 | head

The fix is not deletion but a reopen signal or a restart. The underlying mechanism is covered in the guide to file descriptors and inodes.

Trap 2 — the same message is recorded twice. Either rsyslog is reading the journal and the socket at the same time, or ForwardToSyslog is on while rsyslog is also using imjournal. Consolidate down to one input path.

Trap 3 — the timestamps do not line up. The journal shows local time by default, while file logs may be in a different format. When investigating, standardising on UTC causes less confusion.

journalctl --utc --since '2026-08-15 05:00' --until '2026-08-15 06:00'
timedatectl status

Trap 4 — container logs fill the node. Container standard output accumulates in files on the node. Rotation has to be configured separately on the container runtime side, and that configuration works independently of logrotate. If node disk warnings keep repeating, start here.

Trap 5 — a runaway service pushes out other logs. The journald rate limit can be adjusted per service.

[Service]
LogRateLimitIntervalSec=10s
LogRateLimitBurst=500

Whether these directives are available depends on the systemd version, so if they do not work, check the systemd.exec documentation for the version you have installed.

Quiz: check your understanding

ls -ld /var/log/journal /run/log/journal
journalctl --list-boots

[Journal]
Storage=persistent
SystemMaxUse=2G

sudo logrotate --debug /etc/logrotate.d/myapp

postrotate
    /bin/kill -USR1 $(cat /run/myapp/myapp.pid 2>/dev/null) 2>/dev/null || true
endscript

journalctl --since '2026-08-15 00:00' --until '2026-08-15 12:00' -o export > /backup/journal-incident.export
sudo journalctl --vacuum-size=500M
journalctl --disk-usage

[Journal]
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=14day

action(type="omfwd"
  target="logs.example.com" port="514" protocol="tcp"
  queue.type="linkedlist"
  queue.filename="fwdRule1"
  queue.maxdiskspace="1g"
  queue.saveonshutdown="on"
  action.resumeRetryCount="-1")

journalctl _SYSTEMD_UNIT=nginx.service _TRANSPORT=stdout
journalctl -u nginx.service -o json-pretty -n 1
journalctl -F _TRANSPORT

Closing

Log plumbing, designed once, lasts for years. And if you never design it, it quietly accumulates problems for years. Checking just these five things when you build a new server prevents most incidents.

Is the journal persistent. Does the journal have a size ceiling. Do the log files have a rotation policy. Does a reopen signal go out after rotation. Is /var/log able to fill the root filesystem.

Put those five lines in your server build checklist. Once they are in, you never have to think about them again. The time you get back is far more valuable spent on making the content of the logs good.

References

• journalctl(1) — man7.org (verified 2026-08-15)
• logrotate(8) — man7.org (verified 2026-08-15)
• rsyslog official documentation (verified 2026-08-15)
• systemd official documentation — freedesktop.org (verified 2026-08-15)

Further reading

• Previous: The complete guide to undoing things in Git
• Next: The complete guide to backup and restore
• Designing structured logging — what an application should record
• The complete guide to systemd service management — how units and the journal connect
• Linux terminal — practise the query commands
• crontab parser — check rotation schedules