shournal

A (file-) journal for your shell

Note: for a more formal introduction please readBashing irreproducibility withshournalon bioRxiv.

What is it (good for)?

TL;DR:

• Integrated tool to increase the reproducibility of your workon the shell: what did you do when and where and what files were modified/read.Auto-snapshot scripts and config-files as used.
• Stand-alone tool to monitor file events of a command/process (tree),similar to
  strace -f -e close $cmdbut without the limitations imposed by ptrace-based solutionsand a lot faster (Overhead).

More details please:

Using your shell's history is nice. But sometimes you want more:

• What files were modified or read by a command? Or reverse: What shell-command(s)were used to create/modify or read from a certain file?
• You executed a script. What was the script-content by the time it was called?
• The command read a config-file - which one, and what was in it?
• What other commands were executed during the same shell-session?
• What about working directory, command start- and end-time or theexit status ($?) ?

shournal provides the answer to these questions whilecausing only a negligible overhead.
Besides its ability to monitor a whole process tree for fileevents it can be integrated really tight into the shell -you won't even notice it (;
See also: shell-integration

Besides output on the command-line in a human-readable format (or JSON)you can export (parts of) your command-history intoa standalone html-file where it is displayed in an interactivetime-line-plot. Further miscellaneous statistics are displayed inbar-plots, e.g. the commands with most file-modifications (see also above gif).

Using the external softwareshournal-to-snakemakean observed shell-command-series can be directly transformed into rules forthe Snakemake workflow engine,a tool to create reproducible and scalable data analyses.

shournal runs only on GNU/Linux.

Examples

Please note: below examples make use of theshell-integration.
Otherwise shournal --exec $cmd andother boilerplate-code would have been necessary.
Instead of printing the --query-results to terminal, you can also createfancy html-plots, by appending --output-format html -o out.html.Use an ordinary web-browser for display.

• Create a file and ask shournal, how it came to be:

  $ SHOURNAL_ENABLE # monitor all commands using the shell-integration
  $ echo hi > foo
  $ shournal --query --wfile foo
  Command id 1 returned 0 - 14.05.19 10:19 : echo hi > foo
    Written file(s):
       /home/user/foo (3 bytes) Hash: 15349503233279147316
  
  

• shournal can be configured, to store specific read files, like shell-scripts,within it's database. An unmodified file occupies space only once.

  $ ./test.sh
  $ shournal -q --history 1
  Command id 2 returned 0 - 14.05.19 14:01 : ./test.sh
    Read file(s):
       /home/user/test.sh (213 bytes) id 1
           #!/usr/bin/env bash
           echo 'Hello world'
  
  

• What commands were executed at the current working directory?

  shournal --query --command-working-dir "$PWD"
  

• What commands were executed within a specific shell-session? Theuuid can be taken from the command output of a previous query.

  shournal --query --shell-session-id $uuid
  

• For the full list of query-options, please enter

  shournal --query --help
  

Installation

Binary releases

For Debian/Ubuntu-based distributions .deb-packages are available on therelease-page.Three different editions are provided for different use-cases: most users willwant to install shournal on a real host (or virtual machine) andshournal-docker inside Docker(or another container platform).shournal-fanotify does not contain the kernel backend andis targeted at institutions where the usage of out-of-tree kernel-modulesis discouraged.
Only LTS-releases are officially supported, the packages are known to work onDebian 9 (Stretch), Debian 10 (Buster), Ubuntu 18.04 (Bionic) and Ubuntu 20.04 (Focal).Install deb-packages as usual, e.g.
sudo apt install ./shournal_2.2_amd64.deb
An update of shournal should be performed after all users havelogged out, because the shell integrations need to be resourced.Further in case of the kernel module backend unloading the oldversion stops all running observations.

After installation:Depending on your distribution, additional steps might be necessary toenable the (recommended) uuidd-daemon. If systemd is in use, onemay need to:

systemctl enable uuidd
systemctl start uuidd

Add yourself or other users to the group shournalk:
sudo adduser $USER shournalk (relogin to take affect).
You may override this group:

mkdir -p /etc/shournal.d/
echo GROUPNAME > /etc/shournal.d/kgroup

replacing GROUPNAME with the value of your choice. This rule takesinto effect the next time shournal's kernel module is loaded ( socall e.g. modprobe -r shournalk; modprobe shournalk or reboot).

To enable the shell-integration:

• for bash: put the following to the end of your ~/.bashrc
  source /usr/share/shournal/SOURCE_ME.bash
  
• for zsh: put the following to the end of your ~/.zshrc
  source /usr/share/shournal/SOURCE_ME.zsh
  
• for other shells: please open an issue, if you want your favoriteshell to be integrated as well (contributions welcome).

and run SHOURNAL_ENABLE afterwards.

More details and advanced options (logging commands executed via ssh)can be found here.

Compile and install from source

Please refer to the instructions found within thecompile-README.

FAQ

• Does shournal track file rename/move operations?
  No, but most often it should not be a problem. Using the--wfile commandline-query-option, shournal finds the stored commandby content (size, hash) and mtime, not by its name.For the name, --wname can be used.More concrete:

  shournal --exec sh -c 'echo foo > bar; mv bar bar_old'
  

  Querying for bar_old by content (--wfile-option) yields exactlythe given command, however, --wname bar_old does not work(--wname bar of course works). To use the bar_old file name(and not content) as basis for a successful query, in this case--command-text -like '%bar_old%' can be used.
• What happens to an appended file?
  How to get a "modification history"?Please read above rename/move-text first.Appending to a file is currently handled as if a new one was created -only the last command, which modified a given file can be found withgood certainty (by file content).However, querying by path/filename works.If the file was appended and renamed, things get more complicated.
• To track files, they can be hashed. Is that slow for big files?
  No, because per default only certain small parts of the file are hashed.
• What does the following message mean and how to get rid of it?:
  fanotify_mark: failed to add path /foobar ... Permission denied.This message might be printed on executing a command with shournal.Most probably the administrator mounted a filesystem object for which you don't havepermissions, thus you cannot monitor file events.In this case you cannot perform file operations at this pathanyway, so it should be safe to silence this warning by adding thepath within the config-file in section [mounts]. If you want to ignore allfanotify_mark permission errors, you can set the flag in section[mounts]:

  [mounts]
  ignore_no_permission = true
  

Configuration

shournal stores a self-documenting config-file typically at~/.config/shournalwhich is created on first run. It can be edited either directly witha plain text editor or via --edit-cfg.For completeness, the most important points are listed here as well.

• Write- and read events can be configured, so only events occurring atspecific (include-)paths are stored. Put each path into a separateline, all paths being enclosedby triple quotes:

  include_paths = '''
    /home/me
    /media
  '''
  

  Each exclude_path should be a sub-path of an include path.

• Note that by default, thereis a limit on the number of logged events per command (max_event_count).Read files (e.g. scripts) can further be configuredto be stored within shournal's database.Files are only stored, if the configured max. file-size, file extension(e.g. sh) and mimetype (e.g. application/x-shellscript) matches.To find a mimetype for a given file you should use
  shournal --print-mime test.sh.The correspondence of mimetype and file extensionis explained in more detail within the config-file.Further, at your wish, read files are only stored if you have write permission for them(not only read) - often system-provided scripts (owned by root) are not of particularinterest.

  shournal will not store more read files per command, than max_count_of_files.Matching files coming first have precedence.

Running inside Docker

To use shournal within Docker (or another container platform),depending on the backend the following steps are necessary:
kernel module backend
Install shournal on the host and shournal-docker inside the container.For unprivileged containers sysfs is mounted readonly. In this casecreate a bindmount from /sys/kernel/shournalk_root to/tmp/shournalk-sysfs, e.g.
docker run ... -v /sys/kernel/shournalk_root:/tmp/shournalk-sysfs.

fanotify backend
Install shournal-docker (or shournal-fanotify) inside docker.For unprivileged containers the capabilities SYS_ADMIN, SYS_PTRACE andSYS_NICE are required, e.g.
docker run ... --cap-add SYS_ADMIN --cap-add SYS_PTRACE --cap-add SYS_NICE.
You may need to configure the backend.

Backend configuration

shournal provides two backends, a custom kernel module and fanotify.The kernel module is used by default, except the shournal-fanotifyedition is installed, where only the fanotify backend isavailable. In general it is recommended to stick with the kernel moduleas it is faster and has less interference with the process environment -for example no new mount namespaces have to be created and no filedescriptor inheritance is necessary to wait for the end of a processtree. See also:shell-integration.
If both backends are installed you may configure the default one globallyby creating the file /etc/shournal.d/backend or for each user by creating~/.config/shournal/backend with content ko or fanotify.

Disk-space - get rid of obsolete file-events

Depending on the file-activity of the observed commands, shournal'sdatabase will sooner or later grow. When you feel that enough timehas passed and want to get rid of old events, this can be done by e.g.shournal --delete --older-than 1ywhich deletes all commands (and file-events) older than one year.More options are available, see alsoshournal --delete --help

Remote file-systems

• shournal is able to monitor file events of specific processes (PID's).Therefore, remote filesystems such as NFS or sshfs can be observed aslong as shournal runs on the same (virtual) machine as the observedprocess. Consequently file events another kernel performs are lost.
• For sshfs in case of the fanotify backend it is necessary,to add -o allow_root to the sshfs-options,otherwise permission errors during fanotify_mark are raised.See also: https://serverfault.com/a/188896

Security

kernel-module backend

In the kernel module it is ensured that each user is only allowed tomonitor his/her own processes. Further, the kernel thread, which processesfile events, runs with effective caller credentials and checksallowed accesses on a per-file basis. Memory allocations are cgroup-aware,even for reading (in case of hashing) and writing (in case of logging)files.

fanotify backend

shournal-run-fanotify is a so called "setuid"-program: whenever a regular user calls it, it runswith root-permissions in the first place. As soon as possible, it runs effectively with userpermissions though.It must be setuid for two reaons:

• fanotify requires root for initializing, because it is inprinciple able, to forbid a process to access a file. shournal does not make useof this feature so this is not a real security concern.
• unsharing the mount namespace requires root, because setuid-programs could still referto seemingly obsolete mounts. This means that under awkward circumstances an unmount-event,which has security-relevant consequences (e.g. mounting a new file to /etc/shadow) might notpropagate to processes which run in other mount namespaces.To make sure mount-propagation applies, all mounts, which carry setuid-binariesor files they refer to, should be mounted shared, or no (security-relevant)mount/unmount events should occur, after the first shournal-process started.Shared mounts are the default in all recent distributions I know of.See alsoman 7 mount_namespaces andshared subtrees.

Limitations

Processes can communicate via IPC (inter-process-communication).If the observed process A instructs the not observed process Bvia IPC to modify a file, the filesystem-event is not registered byshournal.

For performance reasons, all files opened with write-permissionsare reported as written by shournal, irrespective of whetherthe process actually wrote to it. By using file size and content (hash)you should be able to cover those cases.

The provided timestamp is determined shortly after a file wasclosed. Note that it is possible that some other process haswritten to it in between. This however is only aproblem, if that other process was itself not observed.

Whether memory mapped (see mmap(2) ) file-events are reported correctlydepends on when the underlying file-descriptor is closed. It is thusapplication dependent and does not work in general.

Additional limitations of the fanotify backend

The file observation only works, if the process does not unshare themount-namespace itself, e.g. monitoring a program startedvia flatpak fails.For further limitations please visit the fanotify manpage.

Known Issues

• on NFS-storages: file events are lost, if the user does not haveread-permissions while a file is closed.Steps to reproduce:
  • open a file readable for you on a NFS storage
  • chmod it 000
  • close it --> the event is lost

How does it work?

shournal attempts to deterministically associate files and shell-commands without changing the users workflow. Under Linux file operations areperformed by the kernel, tracing these operations thus requires OS-level support.During the execution of a shell-command, shournal instruments the kernel totrace files used by the shell-process and any of it’s descendant processes. Moreparticular, to keep the tracing-overhead low, only the closing of files is tracedand (meta-)data collection starts afterwards in an asynchronous manner.

shournalk as a kernel module runs directly in kernel space and is based ontracepointsand theftrace-frameworkwhich basically allow for custom code to be run at certain kernelexecution paths without recompilation of the kernel itself. Only threeevents are traced: closing of files, fork and exit. (Meta-)data collectionalso takes place entirely in kernel space.

The fanotify backend employs the kernel-nativefanotify filesystem APIto register for close-events of wholemount-points which are isolated against unrelatedprocesses using unsharedmount namespaces.shournal thereby ensures that all file-operations during the execution of a shell-command refer to the same, unique mount namespace. While the process-filteringtakes place in kernel space — so only file-events of observed processesare copied to user-space — the (meta-)data collection happens in userspace.

Overhead

File tracing imposes a runtime overhead.A detailed performance evaluation may follow soon. For now:We measured the following command executions with shournal v2.5:

• compile elfutils-0.176
• git checkout — checkout the Linux kernel’s source code from v4.19 to v3.10.
• kernel copy — cp of the 4.19 Linux source.

The benchmark involves tracing, (meta-)data collection and saving toa binary temporary file. As this file can be kept indefinitely, thefinal storing into the SQL-database is not part of the measurement.

The relative runtime-overheads are shown in below table,strace listed for comparison with ptrace-based solutions:

For the cp benchmark, where ~120.000 file-events occurredin ~4 seconds, the runtime overhead of the fanotify backend becomessignificant. Note that many file-events in short time constitute aworst-case. Where performance is critical, the kernel module backendshould be used.

The storage overhead largely depends on configuration, e.g. the numberof stored scripts and file-metadata is limited by default, to avoid e.g.a backup-script from flooding the database. For the cp-testthe average disk-usage per file-event is approx. 174 bytes which alreadyincludes indexes to speed up queries. So one GiB of disk-space issufficient for approx. 6 million events. Based on the experience of real-worldusers the database is typically not larger than a few hundred megabytesafter months of usage.

Credits

shournal makes use of great tools and libraries, most importantly the Qt-framework,xxhash, tsl::ordered_map and cmake and also the Linux-Kernel's fanotify.For the html-plot d3js, jquery, popper.js, bootstrap, webpackand others are used.

Thanks to the developers!

The project arose in the Hoffmann ResearchGroup: Computational Biology of Agingat the Fritz Lipmann Institute in Jena (Germany).Special thanks toSteve Hoffmannand Konstantin Riege - without you this projectcouldn't have been accomplished.

License

The whole project is licensed under the GPL, v3 or later(see LICENSE file for details)
except

• The kernel module within kernel/ which is licensed underthe GNU General Public License version 2 only.
• The libraries withinextern/ → Please refer to the licenses within theirrespective directories.
• The javascript-libraries in the auto-generatedhtml-export/dist/main.js → the licenses arestored in html-export/dist/main.licenses.txt.

Copyleft (C) 2021, Tycho Kirchner