Written by Karl Ding
on 

noclobber and the >| redirection operator

I recently encountered an issue in a project I started working on where it wouldn't build correctly. This wasn't too unexpected or concerning, since I had just set up the build environment, and as I've found throughout my undergraduate career, setup instructions are generally outdated. Moreover, CI was passing, and after asking around, others were able to build the project just fine, so it wasn't anything to raise alarms about.

So I dug into what the build system was doing, and found a step invoked by the build system that essentially performed this operation.

# Do stuff
foo > output.txt
# Do more stuff

The problem suddenly became clear to me.

Now, for background, I have the following setting enabled in my ~/.bashrc to prevent me from accidentally overwriting files.

set -o noclobber

This is not a shell default. In fact, in bash and zsh, noclobber is disabled, which means that it is possible to clobber (or overwrite via the shell redirection operators) the contents of the file. You can check for yourself by running the following:

set -o | grep noclobber

If it's enabled, you should see the following

noclobber       on

The original author intended for that command to create output.txt unconditionally. In other words:

  1. If output.txt doesn't exist, create the file and then write to it
  2. Otherwise, truncate output.txt and then write to it

However, what ended up happening was that the script would check the exit code, and recognize that the redirection failed, and then terminate. As such, the build system would report a failure, since it expected that to always succeed.

The POSIX-compatible way of doing this sort of clobbering is to use the >| operator. According to the POSIX specification:

Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of word to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.

Apparently this is one of those things that not many people know of.

I had to explain to a few others (the people I first checked with to see if they were running into build problems) why this was failing on my machine but not theirs, and how assuming that > would always clobber the file is a bad assumption.

But a quick patch later, and I was on my way.