Daily Source Reading: cat
Work In Progress Article
Work in progress. Look over there, a 3-headed monkey!
I don't believe in keeping drafts offline. Read it unpolished while you can! Once it's out there, it forces me to actually follow through.
In the meantime, if you wanna refresh this page from time to time, follow along for live updates of me reading the code and writing this post.
cat
Our second day of reading source code from BSD systems! So far we
are keeping our timeline (which soon will probably crumble into dust).
Yesterday we had our first contender, echo. Today we keep it in a
similar spirit: cat. Both are mostly focussed on getting some input
and yeeting it somewhere else.
Although still on a manageable size (~500 lines of code), it's too
much to look at how both, FreebSD and OpenBSD do it. At least for
a single blog post. So we are going to pick one implementation and
maybe, if time permits, look at some notable differences if there are
any.
We will also just plainly ignore some of the more mundate stuff like option parsing and just focus on the nitty-gritty of it all.
By giving both versions a quick glance, we will again pick OpenBSD
for an easier reading. It's almost only half the size of the
FreeBSD version. We will later delve into why that is. But for now
we just want to understand the core mechanics.
A new thing that I will introduce here is a list of potential action
items. Say OpenBSD applying pledge, while the FreeBSD version
doesn't, might allow for some sneaky patches that add capsicum where
it isn't applied yet. One of these days I will add a static page to
keep track of these items. But that day is not today as I can't be
arsed right now.
So, lets' jump in.
OpenBSD
We will start with our entry point, right at the top:
int main(int argc, char *argv[]) { int ch; if (pledge("stdio rpath", NULL) == -1) err(1, "pledge"); while ((ch = getopt(argc, argv, "benstuv")) != -1) {
So far this makes sense. Right from the start we want to limit any
capabilities to standard I/O and looking up paths, hence the
additional rpath request.
The argument parsing with getopt that follows, we will just
gleefully jump past that. It just goes through the options, sets some
flags and done. Nothing to see here, move along.
The only interesting part here is that, if there are no arguments, so
just plain cat, then we don't need to look up any files, so we
pledge again to remove the rpath permissions.
if (argc == 0) { if (pledge("stdio", NULL) == -1) err(1, "pledge"); cat_file(NULL);
Otherwise we just loop over the files and cat them:
} else { for (; *argv != NULL; argv++) cat_file(*argv); }
Now, our first real entry to the code seems to be cat_file.
cat_file has two different main paths.
if (bflag || eflag || nflag || sflag || tflag || vflag) { if (path == NULL || strcmp(path, "-") == 0) { cook_buf(stdin, "stdin"); clearerr(stdin); } else { if ((fp = fopen(path, "r")) == NULL) { warn("%s", path); rval = 1; return; } cook_buf(fp, path); fclose(fp); }
and anything else not matching that:
} else { if (path == NULL || strcmp(path, "-") == 0) { raw_cat(STDIN_FILENO, "stdin"); } else { if ((fd = open(path, O_RDONLY)) == -1) { warn("%s", path); rval = 1; return; } raw_cat(fd, path); close(fd); } } }
What these flags have in common, is that they all require knowledge about the current line number and count. Therefore it requires reading the input character by character.
This is a hell of a lot slower than just dumping the raw input straight to output.
The latter one seems to be the easier to read, so let's get that one out of the way. For brevity, I'll skip the variable declarations.
void raw_cat(int rfd, const char *filename) { // (variable declarations) wfd = fileno(stdout); if (buf == NULL) { if (fstat(wfd, &sbuf) == -1) err(1, "stdout"); bsize = MAXIMUM(sbuf.st_blksize, BUFSIZ); if ((buf = malloc(bsize)) == NULL) err(1, NULL); } while ((nr = read(rfd, buf, bsize)) != -1 && nr != 0) { for (off = 0; nr; nr -= nw, off += nw) { if ((nw = write(wfd, buf + off, nr)) == -1 || nw == 0) err(1, "stdout"); } } if (nr == -1) { warn("%s", filename); rval = 1; } }
We start of by getting the file descriptor for the output (standard
out, a pipe, whereever we are gonna be writing to). If we haven't
done it yet, we check with fstat if there is an optimal I/O
blocksize for the destination and allocate a write buffer of the
appropriate size or use BUFSIZ by default if there is no optimal
size. A quick and easy optimization to make sure we run at best
settings.
The remainder is easy. Just read bsize chunks from our input and
write them as whole blocks to the output. The inner =for- loop
ensures that non-blocking writes get handled properly.
Now for the other function, cook_buf. We are not gonna look at
every single line and every single flag. I'll leave that up to the
reader as an exercise if boredom strikes.
void cook_buf(FILE *fp, const char *filename) { for (prev = '\n'; (ch = getc(fp)) != EOF; prev = ch) { if (prev == '\n') { if (sflag) { // ... } } if (ch == '\n') { // ... } else if (ch == '\t') { // ... } else if (vflag) { // ... } if (putchar(ch) == EOF) break; } if (ferror(fp)) { warn("%s", filename); rval = 1; clearerr(fp); } if (ferror(stdout)) err(1, "stdout"); }
The function is a bit long due to the handling of all the different
flags, but otherwise in it's core it is literally just reading
character by character and emitting it via putchar.
Good to keep that in mind when using flags on cat that there is a
huge perfomance difference with some flags (well…all flags except
for -u).
FreeBSD
Pretty much same appraoch, but twice the size of the code. Instead of pledge, it uses
casper to set more fine-grained permissions:
fa = fileargs_cinit(casper, argc, argv, O_RDONLY, 0,
cap_rights_init(&rights, CAP_READ, CAP_FSTAT, CAP_FCNTL, CAP_SEEK),
FA_OPEN | FA_REALPATH);
Then there are a few differences in regards to unix domain sockets.
FreeBSD does some special handling here that seems mostly like
optimization to me. I still have to dive deeper into the socket
implementation before I can fully explain it.
There is also some optional #define, in case FreeBSD is being
bootstrapped on a different system (i.e. Linux or MaxOS) and turns
off most features except -l and -u flags.
The version for raw_cat is almost the same. But it does also query
sysconf to check for _SC_PHYS_PAGES size if inode protection
mode is on, or checks the maximum of page size or optimal buffer size,
depending on which is larger.
Same idea, just a bit more fine-grained in how it optimizes the buffer size.
cook_cat is almost the same as in OpenBSD, barring some style
differences.
Conclusion
Again, OpenBSD "wins" here on simplicity, but cat has use-cases of,
for example, merging large files or going through large files to pipe
somewhere else, that these additional performance optimizations done
by FreeBSD make a lot of sense.
It's interesting how even a seemingly "simple" tool like cat can
involve so many optimizations.
So that's it, another day, another source file done.
Next up we are picking pwd, as I won't have a lot of time tomorrow
and it's a manageable size to get through.