noclobber and the >| redirection operator
I recently encountered an issue in a project I started working on where it wouldn't build correctly. This wasn't too unexpected or concerning, since I had just set up the build environment, and as I've found throughout my undergraduate career, setup instructions are generally outdated. Moreover, CI was passing, and after asking around, others were able to build the project just fine, so it wasn't anything to raise alarms about.
So I dug into what the build system was doing, and found a step invoked by the build system that essentially performed this operation.
# Do stuff foo > output.txt # Do more stuff
The problem suddenly became clear to me.
Now, for background, I have the following setting enabled in my
~/.bashrc to prevent me from accidentally overwriting files.
set -o noclobber
This is not a shell default. In fact, in
noclobber is disabled, which means that it is possible to clobber (or overwrite via the shell redirection operators) the contents of the file. You can check for yourself by running the following:
set -o | grep noclobber
If it's enabled, you should see the following
The original author intended for that command to create
output.txt unconditionally. In other words:
output.txtdoesn't exist, create the file and then write to it
- Otherwise, truncate
output.txtand then write to it
However, what ended up happening was that the script would check the exit code, and recognize that the redirection failed, and then terminate. As such, the build system would report a failure, since it expected that to always succeed.
The POSIX-compatible way of doing this sort of clobbering is to use the
>| operator. According to the POSIX specification:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of word to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.
Apparently this is one of those things that not many people know of.
I had to explain to a few others (the people I first checked with to see if they were running into build problems) why this was failing on my machine but not theirs, and how assuming that
> would always clobber the file is a bad assumption.
But a quick patch later, and I was on my way.