Better log rotation in pysystemtrade

The default pysystemtrade setup includes a Python script that renames old echo files with the date and an .arch extension, and deletes them after 30 days:

• sysproduction/clean_truncate_echo_files.py

This script is executed by the scheduled job run_cleaners, usually just after daily orders are created. This is fine. The script works. But there are a few issues:

• the behaviour is not configurable
• the old logs are not compressed
• there’s a lot of Python code for doing not much really

But the bigger problem is that there is already a standard unix tool that does this stuff. Unsurprisingly, the act of cleaning up old logs is a common task. logrotate is your friend here; it does its one job very well, is highly configurable, and has been battle tested on countless systems for decades. You can read the man pages here, and a decent howto article here.

setup with pysystemtrade

#

To replicate the existing pysystemtrade behaviour, you’ll want a config, something like

/home/user/echos/*.txt
{
    daily
    rotate 30
    maxage 30
    missingok
    dateext
    notifempty
}

You could put the config in a file named logrotate.conf, perhaps alongside the private config files, so /home/user/private_config/logrotate.conf. And then run it daily from cron with a command like

$ logrotate /home/user/private_config/logrotate.conf -v --state /home/user/.logrotate.state

The default echo cleaning script would need to be turned off too. Edit private_control_config.yaml so it looks like:

process_configuration_methods:
  run_cleaners:
    clean_backtest_states:
      max_executions: 1
    clean_echo_files:
      max_executions: 0

However, for my own system I want different behaviour:

• my log files are named *.log not *.txt
• I only want to keep logs for a couple of weeks
• I want to compress old files
• I want to wait one day before compressing, as usually when I’m looking at logs, it’s from the day before
• I don’t want the date part at the end of the filename, I want it before the extension

To do this with the existing Python scripts would be fiddly and painful. But with logrotate, I can adjust the config like:

/home/user/pysystemtrade/*.log
{
    daily
    rotate 10
    maxage 14
    missingok
    dateext
    compress
    notifempty
    delaycompress
    extension .log
}

My logs are rotated exactly as I want. And I have not had to change any Python files