Buildinfo with Bitbucket

Greetings-

I am using Bitbucket pipelines to push builds and related stuff into Octopus. Everything is working fine except for the build-information step. Currently I have copied the .sh script referenced here: Bitbucket Pipelines: Pipes and integrating with Octopus Deploy - Octopus Deploy as part of the script to build the JSON file. However, I am getting an error in that script

Comparing between null and 6297acf
fatal: not a git repository (or any parent up to mount point /opt/atlassian/pipelines/agent)

Relevant yml

image: octopusdeploy/octo:7.4.3552-alpine
script:
  - export VERSION=$PROJECT_VERSION.$BITBUCKET_BUILD_NUMBER
  - /bin/sh $BITBUCKET_CLONE_DIR/setup/create-build-info.sh $BITBUCKET_REPO_OWNER $BITBUCKET_REPO_SLUG $BITBUCKET_BUILD_NUMBER $BITBUCKET_COMMIT $BITBUCKET_BRANCH $BITBUCKET_GIT_HTTP_ORIGIN
  - octo build-information --package-id=$PROJECT_NAME --version $VERSION --file=octopus.buildinfo --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --overwrite-mode=OverwriteExisting

The ocotpus.buildinfo file being created is:

{
  "BuildEnvironment": "BitBucket",
  "Branch": "master",
  "BuildNumber": "178",
  "BuildUrl": "https://bitbucket.org/tersosolutions/deviceconsoleweb/addon/pipelines/home#!/results/178",
  "VcsType": "Git",
  "VcsRoot": "http://bitbucket.org/tersosolutions/deviceconsoleweb",
  "VcsCommitNumber": "6297acf",
  "Commits": []
}

As you can see, the Commits are empty due to the failure of the bash script. Any ideas why this is not working? Are there any alternatives to generating this file? I would prefer not to use octopus-cli-run since it is still in beta.

Hi @travis_phillips

Thanks for getting in touch (and welcome back)

I wrote the original blog post you referenced about BitBucket, and I’m also the author of the octopus-cli-run BitBucket pipe.

I learned the hard way that generating commits from a build server is hard. So I completely understand how frustrating it is when it’s not working.

It’s difficult to tell the exact reason without access to the repository. But the error looks like it failed to get the prevCommitHash since the message in the error showed this output:

Comparing between null and 6297acf

The highlighted null above should be the previous hash.

Basically this is the code you’d want to add debug logging into:

previousbuildNumber=$(expr $buildNumber - 1)
baseURL="https://api.bitbucket.org/2.0/repositories/$workspace/$repository/pipelines"
	
prevCommitHashURL="$baseURL/$previousbuildNumber"
prevFullHash=$(curl -s -X GET "$prevCommitHashURL" | jq  '.target.commit.hash')

One thing to consider is that if your repository is private, the code above doesn’t handle authentication. I’m fairly sure you’d need to provide some credentials, either a username and password or an access token / API key. Details of how to do this can be found in the BitBucket API documentation here:

https://developer.atlassian.com/cloud/bitbucket/rest/intro/#authentication

I hope that helps, but please let me know if you have any further questions!

Best regards

Hi Mark-

Making some progress but still running into issues with the script. Modified the cURL call to be:

prevFullHash=$(curl -u $appuser:$apppassword -s -X GET "$prevCommitHashURL" | jq  '.target.commit.hash')

With $appuser and $apppassword being passed in as variables to the script via the yml. Tested the command on a linux box and get:

curl -u {user}:{password} -s -X GET "https://api.bitbucket.org/2.0/repositories/{org}/{project}/pipelines/181" | jq '.target.commit.hash'
"2a2a752c8f6eba2d41c94eefb80272cdbdf266ec"

So the hash is coming back there now.

The error being shown is now (with added debug for the variables being set):

Previous full hash: "2a2a752c8f6eba2d41c94eefb80272cdbdf266ec"
Prev hash: 2a2a752
Commit hash: 70cc9d1
Comparing between 2a2a752 and 70cc9d1
fatal: not a git repository (or any parent up to mount point /opt/atlassian/pipelines/agent)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Building commit json

So something not quite right yet in there.

Running

git log --pretty=oneline "2a2a752".."70cc9d1" > commit.log

locally makes an empty commit.log file.

Any thoughts?

More information…

I changed the bash script to be

git log --pretty=oneline "$prevHash..$commitHash" > git-commits.log

since testing locally it did not like $prevHash and $commitHash in their own set of quotes. That allows it to pull correctly at least. Testing with the full hash and shortened version both work as well.

That leaves sorting out the error:

Prev hash: 69e210a
Commit hash: bbb42d3
Comparing between 69e210a and bbb42d3
fatal: not a git repository (or any parent up to mount point /opt/atlassian/pipelines/agent)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

It’s acting like git is not available for some reason based on the mount point it shows in the error.

Hi @travis_phillips

Without seeing the full YML pipeline, it’s difficult to know the settings you have for the BitBucket pipeline, but I suspect it’s most likely due to the setting where the setting Git clone has been disabled in the step, or entire pipeline - see this for further info.

I call this out in the limitations of the octopus-cli-run-pipe here.

If the step hasn’t got the code available (well the .git directory) doing a git log will fail.

If you did want to share the full YML pipeline, I can make this conversation private so nobody except Octopus employees and yourself can see it. Let me know if you want me to do that before you share any files :slight_smile:

Thanks, Mark

Also, another quick question

Did you add the username / password code to both the cURL call that gets the previous build number and the git commit hash?

Thanks, Mark

We can keep it public, that way if anyone else comes across this they have options :slight_smile:

I’ve commented out most of the yml file so I’m putting the relevant stuff in as nearly all my parameters come from repo/workspace variables. Your note about clone: enabled: false does not apply since I do not have it anywhere.

clone:
  depth: full

pipelines:
  default:
    - step:
        name: Deploy to Alpha
        deployment: alpha
        services:
          - docker
        image: octopusdeploy/octo:7.4.3552-alpine
        script:
          - apk update && apk upgrade
          - apk add --no-cache git curl jq
          - export COMMIT_MESSAGE=`git log --format=%B -n 1 $BITBUCKET_COMMIT`
          - export PROJECT_VERSION=`cat $BITBUCKET_CLONE_DIR/.version`
          - export VERSION=$PROJECT_VERSION.$BITBUCKET_BUILD_NUMBER
          - echo "export VERSION=$VERSION" >> set_version.sh
          - rm -f $BITBUCKET_CLONE_DIR/_rels
          - rm -f $BITBUCKET_CLONE_DIR/*.yml
          - rm -rf $BITBUCKET_CLONE_DIR/.[!.]* ..?*
          - echo "$VERSION" > $BITBUCKET_CLONE_DIR/.version
          - /bin/sh create-build-info.sh $BITBUCKET_REPO_OWNER $BITBUCKET_REPO_SLUG $BITBUCKET_BUILD_NUMBER $BITBUCKET_COMMIT $BITBUCKET_BRANCH $BITBUCKET_GIT_HTTP_ORIGIN $BITBUCKET_APP_USER $BITBUCKET_APP_PASSWORD
          - rm -rf $BITBUCKET_CLONE_DIR/*.sh
          - octo pack --id $PROJECT_NAME --version $VERSION --outFolder ./out --format nupkg --overwrite --description="$PROJECT_DESCRIPTION" --releaseNotes="$COMMIT_MESSAGE" --author="$PROJECT_AUTHOR" --title="$OCTOPUS_PROJECT"
          - octo push --package ./out/$PROJECT_NAME.$VERSION.nupkg --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --overwrite-mode=OverwriteExisting
          - octo build-information --package-id=$PROJECT_NAME --version $VERSION --file=octopus.buildinfo --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --overwrite-mode=OverwriteExisting
          - octo create-release --project "$OCTOPUS_PROJECT" --version $VERSION --packageversion $VERSION --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --ignoreexisting
          - octo deploy-release --project "$OCTOPUS_PROJECT" --version $VERSION --progress --waitfordeployment --deployTo=Alpha --tenant="Cloud" --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY
        artifacts:
          - set_version.sh

The build setup debug when the pipeline runs gives this since it’s relevant:

+ umask 000
+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="master" https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...
+ git reset --hard 67c579f901dbeed985e034bba40cb0336486ecfd
HEAD is now at 67c579f Make it work
+ git config user.name bitbucket-pipelines
+ git config user.email commits-noreply@bitbucket.org
+ git config push.default current
+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/
+ git remote set-url origin http://bitbucket.org/$BITBUCKET_REPO_FULL_NAME
+ git reflog expire --expire=all --all
+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude
+ chmod 777 $BUILD_DIR
Images used:
    build : docker.io/octopusdeploy/octo@sha256:948e2bac89dec3205b19d769478531533d14171df610c792a79b10e570bf5477

The export COMMIT_MESSAGE=`git log --format=%B -n 1 $BITBUCKET_COMMIT` in the yml file does return the commit correctly. I had been using that with push to dump the commit message into Octopus prior to build-information being available. Note that I was not using user/pass here and it was working just fine.

Here is what I think is happening based on all the debug stuff.

The BITBUCKET_CLONE_DIR=/opt/atlassian/pipelines/agent/build is set when the docker container spins up so the ‘root’ of the git clone is there. But the bash script error fatal: not a git repository (or any parent up to mount point /opt/atlassian/pipelines/agent) has it running in a directory higher for some reason. The question then is why the script is running in a directory higher when it is in the clone_dir in the first place.

Hi @travis_phillips

Thanks for the additional information, including the YML pipeline file.

I’m honestly a bit stumped as to why the git log command inside of the script would be failing since the git log outside of the script only a few commands earlier worked.

This is starting to fall outside of my area of expertise, but there are some other options for you to try:

  • Add an echo $PWD before the first git log command that’s successful.
  • Add an echo $PWD before the call to /bin/sh create-build-info.sh and see what directory you’re in then
  • In the create-build-info.sh script, add the same echo $PWD before the git log to see what directory it thinks it’s running in then. If it does show it’s in a different directory, you could try changing the current working directory, using something like cd $BITBUCKET_CLONE_DIR before the git log

If none of that helps, I only have two other suggestions:

  • Engage with Atlassian to see if they can help shed any further light on the issue. I’ve used their community support before, and this section might be useful - https://community.atlassian.com/t5/Bitbucket-questions/qa-p/bitbucket-questions
  • Try out the octopus bitbucket pipe octopus-cli-run. After seeing the issue you faced, I added support for a username/password for the build information. You need version 0.34.0 or greater if you want to use it. Although it’s marked as beta/experimental, I do (where time allows) provide updates to it, but it’s currently on a best-endeavors basis, as it’s not written by our Engineers and therefore I can’t say it’s “officially supported”. I’ve added details of the variables in the readme here.

Hopefully, we can figure it out!

Best, Mark

Hi @travis_phillips

I managed to recreate the error you were seeing, and I think I know the cause.

Essentially I think this line in your pipeline:

- rm -rf $BITBUCKET_CLONE_DIR/.[!.]* ..?*

Was deleting the .git folder from the directory $BITBUCKET_CLONE_DIR (/opt/atlassian/pipelines/agent/build) before the call to create-build-info.sh. Then when the git log was executing, since the .git folder was no longer present, it didn’t think the folder was a valid git repo.

I commented that line out in my test, and the build info commits were able to be retrieved from running git log.

Looking at your pipeline, I assume you are clearing up folders and files you don’t want packaged up which makes sense, so I think a little re-ordering of that one command can sort this out:

clone:
  depth: full

pipelines:
  default:
    - step:
        name: Deploy to Alpha
        deployment: alpha
        services:
          - docker
        image: octopusdeploy/octo:7.4.3552-alpine
        script:
          - apk update && apk upgrade
          - apk add --no-cache git curl jq
          - export COMMIT_MESSAGE=`git log --format=%B -n 1 $BITBUCKET_COMMIT`
          - export PROJECT_VERSION=`cat $BITBUCKET_CLONE_DIR/.version`
          - export VERSION=$PROJECT_VERSION.$BITBUCKET_BUILD_NUMBER
          - echo "export VERSION=$VERSION" >> set_version.sh
          - rm -f $BITBUCKET_CLONE_DIR/_rels
          - rm -f $BITBUCKET_CLONE_DIR/*.yml
          - echo "$VERSION" > $BITBUCKET_CLONE_DIR/.version
          - /bin/sh create-build-info.sh $BITBUCKET_REPO_OWNER $BITBUCKET_REPO_SLUG $BITBUCKET_BUILD_NUMBER $BITBUCKET_COMMIT $BITBUCKET_BRANCH $BITBUCKET_GIT_HTTP_ORIGIN $BITBUCKET_APP_USER $BITBUCKET_APP_PASSWORD
          - rm -rf $BITBUCKET_CLONE_DIR/.[!.]* ..?*
          - rm -rf $BITBUCKET_CLONE_DIR/*.sh
          - octo pack --id $PROJECT_NAME --version $VERSION --outFolder ./out --format nupkg --overwrite --description="$PROJECT_DESCRIPTION" --releaseNotes="$COMMIT_MESSAGE" --author="$PROJECT_AUTHOR" --title="$OCTOPUS_PROJECT"
          - octo push --package ./out/$PROJECT_NAME.$VERSION.nupkg --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --overwrite-mode=OverwriteExisting
          - octo build-information --package-id=$PROJECT_NAME --version $VERSION --file=octopus.buildinfo --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --overwrite-mode=OverwriteExisting
          - octo create-release --project "$OCTOPUS_PROJECT" --version $VERSION --packageversion $VERSION --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY --ignoreexisting
          - octo deploy-release --project "$OCTOPUS_PROJECT" --version $VERSION --progress --waitfordeployment --deployTo=Alpha --tenant="Cloud" --server $OCTOPUS_SERVER --apiKey $OCTOPUS_APIKEY
        artifacts:
          - set_version.sh

Note the line to remove is now after the call to create-build-info.sh. You can of course move all of the rm commands, but the simplest is to move the one offending item.

I hope that helps!

Best

Hi @travis_phillips

I spent some more time today on the BitBucket pipe to provide better error handling and logging.
I totally understand if you’d prefer not to use it, but if you were interested the latest version of the script is here:
https://bitbucket.org/octopusdeploy/octopus-cli-run/src/master/pipe/build-information.sh

Best,

Well good news @mark.harrison

Moving the rm -rf $BITBUCKET_CLONE_DIR/.[!.]* ..?* command did the trick. What an adventure. Anything you think I should leave up on this thread in case others come across it?

Hi @travis_phillips

I’m really glad to hear that moving the command did the trick!

I think the thread is good as it is, it describes the adventure we’ve had well :grinning_face_with_smiling_eyes:

Hope you have a great weekend.

Best,