use private git repository as go mod dependency

The go modules is inbuilt dependency management tool for golang and from go 1.14 it is in GA. In this article, we will discuss how we can use a private git repository as a go module. Go uses 3 sites at the time of downloading new dependency. This works well with public go modules for private go modules we need to make some changes that we will go through in this article.

The Go team is providing the following services run by Google: a module mirror for accelerating Go module downloads, an index for discovering new modules, and a global go.sum database for authenticating module content.

• https://proxy.golang.org
• https://index.golang.org
• https://sum.golang.org

To make it working with private repo we need to make 2 changes.

• Bypass go proxy

Go provides an environment variable called GOPRIVATE which allows us to provide a comma-separated list of glob patterns of private modules.

• Provide credentials to download go modules

We need to create a personal access token or app token for your user in github or gitlab or bitbucket.

Local environment

We can add a make target like this and by running make vendor we will be able to download private modules.

.PHONY: vendor
vendor:
 @GO111MODULE=on \
 GOPRIVATE=bitbucket.org/shovanmaity/go-pvt-module,github.com/shovanmaity/go-pvt-module,gitlab.com/shovanmaity/go-pvt-module \
 GIT_TERMINAL_PROMPT=1 \
 go mod tidy
 @GO111MODULE=on \
 GOPRIVATE=bitbucket.org/shovanmaity/go-pvt-module,github.com/shovanmaity/go-pvt-module,gitlab.com/shovanmaity/go-pvt-module \
 GIT_TERMINAL_PROMPT=1 \
 go mod download
 @GO111MODULE=on \
 GOPRIVATE=bitbucket.org/shovanmaity/go-pvt-module,github.com/shovanmaity/go-pvt-module,gitlab.com/shovanmaity/go-pvt-module \
 GIT_TERMINAL_PROMPT=1 \
 go mod vendor

If we have ssh key setup in our local environment then we can run these commands

git config — global \
 url.”git@github.com”.insteadOf \
 “https://github.com"
git config — global \
 url.”git@gitlab.com”.insteadOf \
 “https://gitlab.com"
git config — global \
 url.”git@bitbucket.org”.insteadOf \
 “https://bitbucket.org"

We can export the access key in local gitconfig(~/.gitconfig). Here X_TOKEN will works with both password and personal access token. But it is not recommended.

git config — global \
 url.”https://${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com".insteadOf \
 “https://github.com"
git config — global \
 url.”https://${GITLAB_USERNAME}:${GITLAB_TOKEN}@gitlab.com".insteadOf \
 “https://gitlab.com"
git config — global \
 url.”https://${BITBUCKET_USERNAME}:${BITBUCKET_TOKEN}@bitbucket.org".insteadOf \
 “https://bitbucket.org"

Docker environment
In the docker environment we need to define some environment variables and pass it build time. Here is one example docker file.

ARG GITHUB_USERNAME
ARG GITHUB_TOKEN
ARG GITLAB_USERNAME
ARG GITLAB_TOKEN
ARG BITBUCKET_USERNAME
ARG BITBUCKET_TOKEN
RUN git config — global \
 url.”https://${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com".insteadOf \
 “https://github.com"
RUN git config — global \
 url.”https://${GITLAB_USERNAME}:${GITLAB_TOKEN}@gitlab.com".insteadOf \
 “https://gitlab.com"
RUN git config — global \
 url.”https://${BITBUCKET_USERNAME}:${BITBUCKET_TOKEN}@bitbucket.org".insteadOf \
 “https://bitbucket.org"

Here is one make target to call docker build.

.PHONY: image
.DEFAULT_GOAL := image
image:
 docker build -t test-private-go-mod:ci \
 — build-arg GITHUB_USERNAME=$(GITHUB_USERNAME) \
 — build-arg GITHUB_TOKEN=$(GITHUB_TOKEN) \
 — build-arg GITLAB_USERNAME=$(GITLAB_USERNAME) \
 — build-arg GITLAB_TOKEN=$(GITLAB_TOKEN) \
 — build-arg BITBUCKET_USERNAME=$(BITBUCKET_USERNAME) \
 — build-arg BITBUCKET_TOKEN=$(BITBUCKET_TOKEN) \
 -f Dockerfile .
 docker image prune -f — filter label=type=build-container

We can export these secret values in ci vm. In local build we can call make target like -

shovan@probot:~$ GITHUB_USERNAME=shovanmaity GITHUB_TOKEN=*** GITLAB_USERNAME=shovanmaity GITLAB_TOKEN=*** BITBUCKET_USERNAME=shovanmaity BITBUCKET_TOKEN=*** make

This source code is available in this github repo and build log is available here.

Step 14/15 : RUN make vendor
 — -> Running in 7ed5b19a7be1
go: downloading github.com/***/go-pvt-module v0.0.1
go: downloading gitlab.com/***/go-pvt-module v0.0.1
go: downloading bitbucket.org/***/go-pvt-module v0.0.1
Removing intermediate container 7ed5b19a7be1
 — -> 0883b1325195
Step 15/15 : RUN go run main.go
 — -> Running in 6d993d5f35f3
 — — — — — — — — — — — — — — -
I am bitbucket private repo.
I am github private repo.
I am gitlab private repo.
 — — — — — — — — — — — — — — -