Connecting Dataform to GitHub: A Step-by-Step Guide

Connecting your Dataform repository to GitHub is a crucial step for data teams, enabling version control, collaboration, and automated deployments. This guide will walk you through the process, from creating a secure access token to syncing your first change.

Step 1: Create a GitHub Personal Access Token (PAT)

This token acts as a secure password for Dataform to access your GitHub repository. For this guide, we'll use a fine-grained personal access token, which is the most secure option.

1. Log in to your GitHub account and navigate to Settings.
2. In the left sidebar, click Developer settings, then Personal access tokens, and finally Fine-grained tokens.
3. Click Generate new token.
4. Give the token a descriptive name, such as Dataform-Access.
5. Under Repository access, select Only select repositories and choose the specific repository you want to connect to Dataform. This ensures the token can only access that one repository.
6. Under Repository permissions, grant the following:
   • Contents: Set this to Read and write. This is the most important permission, allowing Dataform to read files from and write commits to your repository.
   • Metadata: This is typically set to Read-only by default and is required for many Git operations.
7. Click Generate token.
8. Copy the token immediately. You will not be able to see it again. Store it in a secure location.

Step 2: Store the PAT Securely in Google Cloud Secret Manager

For security, you should not paste your PAT directly into Dataform. Instead, we'll use Google Cloud's Secret Manager.

1. In the Google Cloud console, search for and navigate to Secret Manager. You may need to enable the API.
2. Click Create secret.
3. Name your secret (e.g., github-dataform-pat).
4. In the Secret value field, paste the token you copied in the previous step.
5. Click Create secret.

Step 3: Grant Dataform Access to the Secret

The Dataform service account needs permission to read the secret you just created.

1. Navigate to your newly created secret in Secret Manager and click on the Permissions tab.
2. Click Grant access.
3. For the New principal, enter the email address for your project's default Dataform service account. The format is service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com. You can find your PROJECT_NUMBER on your Google Cloud project's dashboard.
4. For the Role, select Secret Manager Secret Accessor.
5. Click Save.

Step 4: Connect Dataform to Your GitHub Repository

With the PAT securely stored and accessible, you can now link your Dataform repository to GitHub.

1. In the Google Cloud console, navigate to your Dataform repository.
2. Click on Settings in the left-hand menu.
3. Under the Repository section, click Connect to a Git repository.
4. Fill out the pop-up form:
   • Git repository URL: Paste the full HTTPS URL of your GitHub repository (e.g., https://github.com/your-username/your-repo.git).
   • Main branch: Enter the name of your main branch, which is typically main.
   • Secret: A dropdown will appear. Select the secret you created in Step 2.
5. Click Connect.

If the connection is successful, you'll see a confirmation message and a new Git status indicator at the top of your Dataform workspace. Congratulations! Your Dataform project is now connected to GitHub, and you can begin using Git directly from the Dataform UI.