31: cat

We're now going to implement many of the tools found in the GNU Coreutils collect. These are implementations of utilities found primarily on Linux, but you'll have similar commands on macOS, FreeBSD, and even Windows has something like them.

I chose the Coreutils because they're deceptively simple tools that almost anyone can create, but that have secret little tricks you only find when you really sit down to implement them. They're also well documented and you should be able to access them from any system. This makes them very good targets for doing replicas.

I also like these tools because a big use case for Go is systems programming. Systems programming is any kind of code you write that controls systems like automation tools, process tools, and network servers. By implementing your own versions of the Coreutils you'll also learn how to do things like automating your system and working with the network.

Escaping Tutorial Hell

When I talk to beginners they frequently complain about "tutorial hell." From what I can gather this is where you watch Youtube videos and read blog posts, but never actually make anything yourself.

I feel that this is mostly caused by people not learning how to go from "idea" to "working software," which is why I added the Remake It challenges throughout the course. By having you recreate my code from your own descriptions you take the first step into creating your own code from your own descriptions.

The next part in your escape from tutorial hell is to have a series of small projects you can create on your own to get better at the process. In everything you create there's always a beginning, a middle, and an end. The way you get better at this is to create many small projects that begin and end quickly. Eventually you'll learn that every large project is nothing more than many small projects chained together.

The Rules of the Game

Let's establish some rules for this little game we're going to play:

1. You need to read the challenge and study the target utility to copy.
2. You are then required to attempt to implement the tool with absolutely no further help from me in this exercise.
3. You can search for documentation and search for clues, but you have to try to not get any help during this phase.
4. Once you've attempted your own solution for 1 day you can then look at the Requirements section where I lay out the various Go modules I used to complete my first version of the project. With this information, continue your attempt.
5. After one more day you can then look at the Spoilers section and only take notes on how I did it. Don't copy my code. Instead, you have to do what you did in the Remake It sections and write down notes based on my code, then use those notes to finish your version.
6. After this you are allowed to completely copy my code line-by-line if you need to figure out how it works, BUT if you had to do that then you have to repeat step 5 again until you can actually Remake It.
7. Finally, you then have to work on this until you get as faithful a copy of the original, or you get bored and want to move on.

The purpose of this challenge process is to get you to attempt your own projects, and then gradually give you more and more clues until you solve it. The purpose is NOT to make you feel stupid. You're still learning so it will take you a while to learn how to turn your ideas (or other people's requirements) into working code. Keep practicing and trying until you can, but if you give up because you think you "failed" then you'll never learn how to do it.

Pro-Tips

Here's some important tips to help you with this game:

1. Get started fast! You want to have a starter project ready to go and get your project up and running without thinking about it. Don't waste tons of time finding the perfect libraries, get that thing up and running first.
2. Once you have it started then do some quick hacking and research "live" on your project. I find that beginners think every line they type has to be perfect, but programming is way more fun if you approach it interactively and get dirty. I think most things are more fun that way.
3. After you dot his initial exploration (which we call a "spike") trash that garbage, but first take notes. Usually your first version isn't very good, but you learn a lot. Starting over with notes on how to do it usually results in a far better version, and also teaches you even more.
4. Establish an end goal early. You want these projects to be quick, but if you don't have an end goal in mind you'll risk beating that code until it's glue. At the beginning decide how far you want to take it, and stop when you get there.
5. Don't set impossible goals. I know, some of you out there think it'll make everyone think you're so smart if you take the ls tool and also add your own MMO video game with bitcoin to it. What's smart is picking a reasonable goal that's only a little bit beyond your skill, but not one that's too easy or too hard.
6. Give yourself a chance. You're just starting out, so comparing your ability to mine is unfair. Keep reminding yourself that you're a beginner and you need time to grow and learn about who you are as a programmer first.

The Challenge

The first tool you'll be required to implement is cat. You can find the documentation here:

• https://www.gnu.org/software/coreutils/manual/html_node/cat-invocation.html

Your goal is to create an initial cat tool, in Go, that does at least 2 things cat does. Once you have that working you should attempt to implement all of the command line options you can implement in one week.

Requirements

You'll find that theres a list of core packages you'll use in most of these tools. You'll need something to work with files, something to work with strings, and something to work with command line arguments.

You should try to use Go's Documentation Site to find the resources you need on your own. Being able to find what you need to accomplish a task is an important skill every programmer needs to learn. You can also run the pkgsite tool to get a locally hosted version of the offical docs.

The packages I used in my version of cat are:

Spoilers

My version of cat doesn't implement everything, and most of my versions of the tool won't be complete solutions. The purpose of this little "spoiler" isn't to give you a full solution, but instead to help you get off the ground for your own solution.

The best way to use these spoilers is to do this:

1. Attempt your own version of the tool.
2. After your attempt, compare it to mine to see if I hve any better ideas.
3. If you have no idea how to even start then take a quick look at mine and take notes.
4. Attempt it based on your notes to see if you can figure it out now.
5. If you're finally unable to make the tool then you should do a full copy of it.
6. If you do a full copy, then follow the Remake It process: take notes on your copy, delete your copy, then recreate it from your notes.

package main

import (
  "fmt"
  "flag"
  "os"
  "log"
  "strings"
)

type Opts struct {
  Number bool
  Squeeze bool
  Filenames []string
}

func ParseOpts() (Opts) {
  var opts Opts

  flag.BoolVar(&opts.Number, "n", false, "Number all nonempty output lines, starting with 1")
  flag.BoolVar(&opts.Squeeze, "s", false, "Suppress repeated adjacent blank lines")
  flag.Parse()

  if flag.NArg() < 1 {
    log.Fatal("USAGE: cat [-n] [-s] file0 [fileN]")
    os.Exit(1)
  }

  opts.Filenames = flag.Args()

  return opts
}

func main() {
  opts := ParseOpts()

  for _, filename := range opts.Filenames {
    in_file, err := os.ReadFile(filename)

    if err != nil { log.Fatalf("cannot open %s: %v", filename, err) }

    if(opts.Number) {
      count := 1
      for line := range strings.Lines(string(in_file)) {
        if opts.Squeeze && len(line) <= 1 {
          continue
        }

        fmt.Printf("%0.4d: %s", count, line)
        count++
      }
    } else {
      fmt.Print(string(in_file))
    }
  }
}

Testing It

You will now write a simple automated test for your cat program. Learning how to test my code automatically is probably my biggest super power as a solo developer. When you code on your own you have to automate everything possible, and automated tests help me make sure my code keeps working while I make changes. They also make it possible for me to make big changes without worrying too much that I'm breaking other things.

Testing these command line tools is also a great way to learn about automation in general. In this exercise we'll write a small incorrect test and discuss different strategies for testing. Then in the remaining exercises we'll use Go testing package to try some of these strategies.

How to Test cat

You have many options for testing cat, but only a few of them are accessible to you at this point:

1. You can write another Go program that runs your cat with different options, confirming its output is correct.
2. You can use Go's testing package to create a test that calls your ParseOpts() function to feed it options.
3. You can rewrite your cat to take any input and output, then craft fake inputs and outputs.
4. You could hack up some bash scripts that just run it and look at what cat does.
5. You could "firewall" your option parsing into a subpackage, then inside this package you'd load a configuration that changes how cat runs for testing.

For this exercise you'll be doing #2, which is the most direct and common way to do your testing.

My (Wrong) Test

To write a test you create a file with test.go as the ending. In this case I create a cat_test.go and put a test in it:

View Source file go-coreutils/tested/cat/cat_test.go Only

package main

import (
  "testing"
  "os"
)

func WithArgs(args []string, test func ()) {
  old_args := os.Args
  defer (func() { os.Args = old_args })()
  os.Args = args
  test()
}

func TestParseOpts(t *testing.T) {
  args := []string{"cat", "-n", "main.go"}

  WithArgs(args, func () {
    opts := ParseOpts()

    if opts.Number != true {
        t.Errorf("opts.Number should be true, not: %v", opts.Number)
    }
  })

  /*
  // This fails because flag can't be run more than once
  args = []string{"cat.exe", "-s", "main.go"}
  WithArgs(args, func () {
    opts := ParseOpts()
    if opts.Number != true {
        t.Errorf("opts.Number should be true, not: %v", opts.Number)
    }
  })
  */
}

NOTE: For my book/course publishing software I have to create a tested/cat directory to put the tested versions of the commands. You should just modify your cat directory and ignore my tested/ path.

Running Tests

You can run tests using the go test command:

$ go test
PASS
ok      lcthw.dev/go/go-coreutils/cat   0.003s

If it works this is what you'll see. If you see a failure it'll be like this:

$ go test
--- FAIL: TestParseOpts (0.00s)
    cat_test.go:22: opts.Number should be true, not: true
FAIL
exit status 1
FAIL    lcthw.dev/go/go-coreutils/cat   0.002s

I truncated this output so you can read it.

Analysis

The first thing you can notice is I have a WithArgs() function that does a few fancy things:

1. I make a copy of the os.Args variable. This variable is the "raw" command line arguments that flag processes when you call flag.BoolVar() and friends.
2. I then create a defer that ensures os.Args is set back to its original values when this function exits.
3. I then alter the arguments to cat through this "backdoor" before calling the given test function.

You should stop and study this eldritch horror. I would say this is a moderate failure because it doesn't let you test cat multiple times (see below). Also, hacking the os.Args is probably going to cause problems no matter how careful you are.

Once I have WithArgs() I can then use it in the TestParseOpts function. Go will load the files you name test.go and it will run any functions that have specific names. In this case, TestParseOpts() but can be any name after Test and must take a *testing.T variable.

Inside TestParseOpts() I use WithArgs to temporarily change the arguments to cat -n main.go and then confirm that opts.Number is true.

Why It's Wrong

If you scan further down you'll see a commented out block that shows why this isn't a great approach. First, you can't run flag.Parse() more than once, so the test fails with an error. That's because flag.Parse() modifies os.Args which is interesting because I just told you that's a bad thing. Not sure why flag gets to do that, but it's why you can't really use this.

How to Fix It

There's a few ways to fix it, but in the next exercise we'll alter od to make testing easier. If you want to try fixing cat_test.go yourself, then think about how you might change ParseOpts() to use flag.FlagSet.