I use gitlab and gitlab-ci and love it. It is the best self hosted CI / Version control system out there in my opinion.
I noticed a strange behavior where running
knife upload from my workstation would return within seconds, where as the same command as a gitlab-ci job would take over 2 minutes.
Grr, 6 minutes for a build? Thats way too long.
If you have seen the same problem, jump to the bottom for the tl;dr version.
Through trial and error, I found that there are two work arounds:
# Takes 2+ minutes sudo -i -u gitlab-runner knife upload --chef-repo-path . roles -V -n
# Takes seconds to run sudo -u gitlab-runner knife upload --chef-repo-path . roles -V -n
# Takes seconds to run sudo -i -u gitlab-runner /usr/bin/knife upload --chef-repo-path . roles -V -n
This is progress. It looks like using sudo with
-i (simulate initial login) is causing problems. Also giving the full path fixes the issue. Thats when I stumbled on this SO question:
Through more trial and error, it definitely looks to be a ruby problem. My gitlab build runner is ubuntu 14.04 using the default ruby 1.9 from apt (no RVM or rbenv). Sure enough, ruby commands take a long time
# Returns in milliseconds time /usr/bin/env which
# Returns in minutes time /user/bin/env ruby
Hmmm… Sounds like a problem with the path.
Looking at the shadow file shows some interesting information:
getent shadow gitlab-runner gitlab-runner:!:16643:0:99999:7:::
Looking at the first character after
gitlab-runner, we see
!. That indicates that the user account has logins disabled. That gets me thinking, the user probably has a different PATH.
# My user path env | grep PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# gitlab-runner path sudo -i -u gitlab-runner env | grep PATH PATH=/home/gitlab-runner/.rvm/gems/ruby-2.2.2/bin:/home/gitlab-runner/.rvm/gems/ruby-2.2.2@global/bin:/home/gitlab-runner/.rvm/rubies/ruby-2.2.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/gitlab-runner/.rvm/bin
No wonder. The gitlab-runner has a super weird path to a non existent RVM installation. I’m guessing that there is a 1 minute timeout looking at the first 2 weird paths, then when it gets to /usr/local it returns immediately.
Success, I decided to fix the problem with the work around of adding /usr/bin/ to the knife command, instead of fixing ruby on the vm, since it will be replaced with docker containers and gitlab multi runner soon.
Success. 25 seconds for a build is more like it
tl;dr make sure you use the full path to knife in your gitlab-ci (/usr/bin/knife)
devops:lab:environments: stage: deploy script: - /usr/bin/knife upload --chef-repo-path . environments