How I Structure Go Packages

Making reusable packages that don't suck

Packages should solve one problem domain, be reusable by default, and individually testable.

Two Types of Packages

In-App packages are created as part of a greater application. These typically serve a specific purpose. They are often created to isolate functionality rather than address reuse.

Stand alone packages are created to share functionality across many applications. They should be independent of application implementation. Allowing users to call the package in many different ways.

StanD alone

In-App Packages

Directory Structure

There are many proposed structures for In-App packages. But there is no standard everyone follows. 

In-App

$ find . -type d
./internal/app/
./internal/pkg/
./internal/pkg/nonreusablepkg2/
./internal/pkg/nonreusablepkg1/
./pkg/reusablepkg1/
./pkg/reusablepkg2/

There are recommendations such as creating a /pkg directory for public sharable packages. While creating an /internal/pkg directory for internal only packages.

$ find . -type d
./app/
./reusablepkg1/
./nonreusablepkg1/
./reusablepkg2/

I put all packages in the repository top level under their own package names. I like this method as it keeps all packages in the same place, a sane easy to find location.

Directory Structure

For stand alone packages, there is a common pattern followed by most.

Stand Alone

Stand alone packages should be in their own repository.

The code files should be located in the top level directory. Any sub-packages should be directories in the top level.

$ find . -type f
./go.mod
./drivers/redis/redis_test.go
./drivers/redis/redis.go
./drivers/cassandra/cassandra.go
./drivers/cassandra/cassandra_test.go
./LICENSE
./go.sum
./README.md
./CONTRIBUTING.md
./hord_test.go
./hord.go

Naming and Organization

Naming files within a package is a key part of keeping a package organized. The guidelines below apply to both in-app and stand-alone packages.

 

  • The primary file, the one that should have the package documentation should have the same name as the package. For example, a package by the name of config should start with a config/config.go file.
  • It's good to break out functionality into multiple files. Make sure that similar functionality is grouped together. I.E. yaml.go, env.go, json.go, or consul.go.
  • When breaking out functionality, keep types, constants or errors where they are referenced or where they logically sit. Many times, this may be the base file i.e. config.go. Avoid creating a constants.go or types.go.
  • Apply the same logic to utility functions, put them where they are most used, or default to the base file. Avoid creating a util.go.

Structuring the Package Interface

As important as naming and directory structure. How we structure the interface is how users work with our package. The functions, methods, and types we export from our package dictates how users use it.

 

Some packages export simple functions that hold no long term state. Other packages will export structures to hold data and provide methods of access. Many packages, will do both.

Just Functions

Many packages provide users with simple helper functions. These functions take data as input, and provide something new or modified as output.

Package Interface

  • These types of packages should never be used to keep data. As this prevents users from creating multiple instances to work with different sets of data.
  • If you find yourself using init() to create a sync.Map or initialize other things. Ask yourself, if I run multiple tests in parallel, will it break?
  • Watch out from turning these packages into a generic utils package. It's easy to group helper functions into one package, but a package should solve one problem. Utilities is not one problem.
package config

// Marshal will take in an interface and format it
// into config data.
func Marshal(in interface{}) ([]byte, error) {
	// do stuff
}

// Unmarshal will take in raw config data and parse
// it into the provided interface.
func Unmarshal(in []byte, out interface{}) error {
	// do stuff
}

Structs with Methods

Some packages store context, data, or control services. These packages usually export a pointer to a custom type with methods. Each pointer, is an individual instance a user can interact with.

Package Interface

  • By using the custom types with our getter and setter methods. Users can create many instances of Config each with different data.

  • When writing tests, each test should have its own instances of Config. Then, they can run in parallel.

  • Stick to using New() or Dial() to create new instances. Users would be calling config.New().

package config

import "sync"

// Config is a custom type for holding and changing configuration
// data.
type Config struct {
	sync.RWMutex
	item string
}

// New will create a new instance of configuration.
// Each instance is unique and can hold different information.
func New() *Config {
	// setup the config object
	return &Config{}
}

// Item is used to access Config.item
func (c *Config) Item() string {
	c.RLock()
	defer c.RUnlock()
	return c.item
}

// SetItem is used to change Config.item
func (c *Config) SetItem(s string) {
	c.Lock()
	defer c.Unlock()
	c.item = s
}

Push Dependencies, Don't initialize them again

Sometimes, it is necessary for packages to load outside dependencies. This is common with Config, Logging, etc. It's tempting to re-initialize these on package load, but don't do that. Let the calling application push those dependencies to the package.

package database

import "github.com/some/logger"

// don't do this... just don't
func init() {
	log = logger.New()
	cfg = config.New()
}

func MyFunc() {
	// do work
	log.Printf("Tell someone about it")
}
package database

// imagine some imports here

type DB struct {
	cfg *config.Config
	log *logger.Logger
}

func Dial(log *logger.Logger, cfg *config.Config) (*DB, error) {
	// create a new DB control object
}

func (db *DB) MyFunc() {
	// do work
	db.log.Printf("Tell someone about it")
}

Do not even...

Better, but not perfect

You don't need those dependencies, not like that atleast

In the example before, we pushed a config object into our database package. But is that what we want? Now our package is locked into using this specific config package. What did it buy us? Dynamically update our DB config? It, sounds nice, but doesn't work like you think it will.

 

In this case, the implementing app should create a new DB instance and shut the old one down. That's the whole point of this package structure.

package database

type DB struct {
	server   string
	password string
}

func Dial(server, password string) (*DB, error) {
	// create a new DB control object
}

NOw We are Talking

When pushing dependencies, ask yourself. Is there a better way of doing this? A way that doesn't kill reusability?

But What about that logger? Just... Don't

Logging has no place in stand alone or in-app packages. It should only be executed within the implementing application.

 

While logging within packages is a pattern that is popular in other languages. That doesn't mean it is right for Go.

  • Is it the right log level? An error to you, might not be an error to me.
  • It removes the users ability to action on error. Maybe I need to do more than log.

Handle Errors by Passing them up the stack

But if you can't log, how do you get errors to the user? That's where multi-value returns, and custom error types come into the picture.

 

As a package, it's preferable to pass any errors to the calling application. This allows the calling application to check the error and perform a specific action.

myVal, err := db.MyFunc()
if err == db.ErrMyCustomError {
	// do something fancy
}
if err != db.ErrMyCustomError && err != nil {
	// ok maybe log and do something less fancy
}

Handling Errors Asynchronously

Sometimes a package must perform tasks in the background. When writing these types of packages it is still better to provide the user with an error. But rather than returning the error on a function, call back to the user with an error.

 

I like two ways of accomplishing this. Either through an error channel, or an error function. Error functions are user defined functions that the package can callback with errors.

type Setup struct {
	// user defined error function
	errFunc func(error)
}

func main() {
	// on setup
	err := New(Setup{
		errFunc: func(err error) {
			if err == db.ErrMyCustomError {
				// do something fancy
			}
			if err != db.ErrMyCustomError && err != nil {
				// ok maybe log and do something less fancy
			}
		},
	})
}

Packages should be made asynchronous only when necessary. It is often easier to implement asynchronous execution within the implementing application itself. This includes error handling.

If you can't convert an In-App package to STandAlone it's too tightly coupled

Earlier we discussed the differences between in-app and stand alone packages. While there is a often a difference in the locations of these packages. Functionally, and structurally, there should be no difference between the two.

 

A well structured in-app package, should be able to be moved to a stand alone package with minimal changes.

Summary

  • Naming and location of packages are important. But there is no official standard, try to keep it sane.
  • In app and stand alone packages should functionally and structurally work the same.
  • When storing data, use structs with pointers and methods. Not only is this better for users, its also easier to keep things goroutine safe.
  • When not storing data, keep it simple.
  • Don't initialize dependencies, push them from the implementing application. Or better yet, structure the package to not need them.
  • Always, pass errors to users for error handling. Don't assume a package knows better than the implementing application. 

EOF

Benjamin Cane

Twitter: @madflojo 
LinkedIn: Benjamin Cane
Blog: BenCane.com

Like this? Check out my some of my other stuff:

You can also share this on Twitter, LinkedIn, or Reddit!

Principal Engineer - American Express