Goem - The Missing Go Extension Manager

Golang is a young language that tries to make many things better than the “ancient” languages. It almost reaches the speed of C without the need of your own memory management. Also it respects modern patterns and makes writing concurrent programs easier than most other languages. But the strict model of Go has disadvantages when it comes to resolving third party dependencies and managing different branches of your Go project. Goem tries to fix this issues.

tl;dr

If you have ever tried to manage a bigger project in Go, you probably ran into the limitations of the basic go get and go build commands. For example go get only catches the master branch, while go build expects to find all your sources in your $GOPATH. These limits can make things difficult, if you want to maintain different branches. Things get even more complicated, if you have several Go projects on your development machine and all of them depend on different version of the same third party packages. This is where goem tries to help. It is a wrapper around the go command and extends its functionality. Go is a young language, which has its weaknesses. One of the biggest disadvantages of Go is the deploying of Go apps. This is where goem makes things easier and extends the capabilities of the go tool.

Installation

Goem is written in pure Go. So all you need is a working Go installation. If you are on Gentoo you can use the ebuild provided with the source. It simplifies the installation process to:

                
                      emerge -av goem

For all the others, there is a really simple build script, that compiles goem and tries to install it to /usr/local/bin.

                
                      bash build

Gofile

First of all you need a Gofile. A Gofile is a goem configuration file, which is encoded in json format. Goem expects the Gofile in the projects root. A basic example is provided with the goem distribution and can be found in ./example.

Let’s start with an empty Gofile and fill it one by one to show how it works.

{
}

This configuration does nothing meaningful. But it enables you to use goem. Basically it tells goem that there are no environments and no third party packages.

                
                      {
    "testdir" : "./test"
}

We just added the testdir option. This specifies where the testfiles are relative to the root of your project. So far we added nothing you could not accomplish with the basic go format. Nevertheless this option is needed, as goem provides its own copy of each dependency of your project.

                
                      {
    "env" : [
    ],
    "testdir" : "./test"
}

Now we added the empty env array. Each entry in env represents an environment goem can handle. For this purpose goem reads the environmental variable $GO_ENV. You can imagine the $GO_ENV as a description of the environment, where your Go app is supposed to live. Thinkable values for $GO_ENV are for example development, staging or production. If this variable is empty, goem sets it to development as fallback. This means, if you don’t need more than one environment you should always fill at least the development environment. We are going to do so in the next step.

                
                      {
    "env" : [
        {
            "name" : "development",
            "packages" : [
            ]
        }
    ],
    "testdir" : "./test"
}

As you can see, we added an json object, which represents the GO_ENV development. This environment is used, when you set GO_ENV to development or if you unset GO_ENV. As you can see, each env has a name and a packages array. We will add our first dependency in the next step.

                
                  
                    
                    
                      {
    "env" : [
        {
            "name" : "development",
            "packages" : [
                {
                    "name" : "github.com/adeven/goenv",
                    "branch" : "master"
                }
            ]
        }
    ],
    "testdir" : "./test"
}

                    
                  

              

Here it is. We defined our first project specific third party dependency. Each entry off the packages array is a json object which has a name and a branch. The name has the same syntax as Go imports. So if you would write something like:

                
                      import(
    "github.com/adeven/goenv"
)

you will write just the same for a package name. Let’s play around with the branch.

                
                  
                    
                    
                      {
    "env" : [
        {
            "name" : "development",
            "packages" : [
                {
                    "name" : "github.com/adeven/goenv",
                    "branch" : "dea6bae7dea12e5aaa480c3b9352f23b66c5fa32"
                }
            ]
        }
    ],
    "testdir" : "./test"
}

                    
                  

              

At the time of this writing, goem only supports git based third party packages. So a branch can be everything which can represent a git branch. For example you can use an explicit branch, like we did in the previous example, or you can use a git commit hash like in the current example. This way you can make sure, that you always use the same version. This ensures that your project still runs, even if the package maintainer decides to break the interface of his library.

But there is more magic in the branch option. Of course the package you define as an dependency also can have a Gofile. Goem checks all dependencies recursively of each package, tries to arrange them and writes the result to the Gofile.lock. So it is possible that two packages have incompatible dependencies to the same package. You can try to resolve this, by defining a branch with the > and < operators.

Imagine you want to define, that your package depends on github.com/adeven/goenv and it needs to be at least commit dea6bae7dea12e5aaa480c3b9352f23b66c5fa32 you could define it in your Gofile like this:

                
                  
                    
                    
                      {
    "env" : [
        {
            "name" : "development",
            "packages" : [
                {
                    "name" : "github.com/adeven/goenv",
                    "branch" : ">dea6bae7dea12e5aaa480c3b9352f23b66c5fa32"
                }
            ]
        }
    ],
    "testdir" : "./test"
}

                    
                  

              

On the other hand you can use <dea6bae7dea12e5aaa480c3b9352f23b66c5fa32 to define that your package depends on an older version than dea6bae7dea12e5aaa480c3b9352f23b66c5fa32.

Of course it is also possible to use a package, that only exists on your local machine. To use such package set the branch option to a path relative to your project root or via an absolute path. It is important that your path starts either with a . or a /. Goem will use softlinks to this package, so every change is instant. Such a Gofile could look like this:

                
                  
                    
                    
                      {
    "env" : [
        {
            "name" : "development",
            "packages" : [
                {
                    "name" : "github.com/adeven/goenv",
                    "branch" : "/home/user/programming/goenv/"
                }
            ]
        }
    ],
    "testdir" : "./test"
}

                    
                  

              

Adding more environments or packages works like described above.

Using goem

After creating your brandnew Gofile with all your dependencies you can start using goem. Remember that goem commands are always executed from the projects root.

goem bundle

After creating a Gofile you can run goem bundle. First it will create all necessary directories. All data downloaded and maintained by goem is located in the .go directory which is located in the root of your project. Then it will try to resolve the dependencies of your project and all of the sub-dependencies.

If this suceeds, goem will write a Gofile.lock. The Gofile.lock is basically a Gofile, but it holds the resolved dependencies. If your are interested which versions goem is actually installing, you can find it here. Finally it will download all dependencies or set the necessary symlinks and organize it in the .go directory.

If you run goem build several times, it will just check whether the dependencies have changed and update them rather than downloading them.

goem list

You can use goem list to see which packages are already installed.

                
                      #list the installed packages
goem list

github.com/adeven/goenv

goem build

Once you ran goem bundle you can use goem build to finally build your binary. Under the hood goem build does several steps. First of all it reads the $GO_ENV environmental variable. As described above goem assumes you are using the development environment, if $GO_ENV is an empty string. After evaluating $GO_ENV goem will load the Gofile.lock and calls the bundler to set all packages to the according branches. As the last step it will set the $GOPATH to your .go directory and build the binary. You can specify the name and the destination of your binary with with the -o flag.

                
                      #this will build the binary in the root of the project and call it a.out
goem build

                
                      #this will build the binary in the sub directory bin and call it "creative_name"
goem build -o bin/creative_name

goem test

Goem test works similar to go test. The only difference is, that it defaults to testdir instead of ..

goem help

This command can be used to display a short help. The syntax is similar to the other command. To get help for the topic bundle, just type goem help bundle. That was easy, wasn’t it?

Practical goem

After all this, you might still be wondering why you should use goem. So let’s take a look at some real world examples.

Deploying Go Apps

As you may have noticed we are deploying our go apps with capistrano. In the past we just used go get. This has a serious drawback: You can only deploy the master. Using goem offers you the chance to use several environments (development, production, staging) and deploy different branches to all of them. In fact goem is most powerful in combination with a tool like goenv. This dreamteam enables you to select from several configurations, environments and branches with just one environmental variable, which is GO_ENV.

Working with different versions

Go is a young language and things can change very quick. If you used a third party package in your project and the maintainer decided to change the interface you have two choices: stick with the old version and hope you can maintain it by yourself or adapt your software to the new version. If you decide to stick with the old version, you can set it easy in your Gofile. On the other hand, if you decide to port your software, you are still able to switch branches easily.

Now imagine your project is the dependency of another software you use. When using go get instead of goem bundle you will have to merge your staging changes to master, everytime you want to this software, as go get can only fetch the master. With goem testing several branches against the same software is a breeze.

Goem - the Missing Go Extension Manager