Document How To Install Python Package from a Private Repo

I have a number of Python Packages in private (company) repos and I am using GitHub Actions to run pytest on commits. One of the repos depends on packages from other repos. When pip runs from the Action, I see the following error:

Collecting pyconfig@ git+ssh://git@github.com/sxi/pyconfig@master
Running command git clone -q 'ssh://****@github.com/sxi/pyconfig' /tmp/pip-install-wglwufhp/pyconfig
Cloning ssh://****@github.com/sxi/pyconfig (to revision master) to /tmp/pip-install-wglwufhp/pyconfig
Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sxi/pyconfig' /tmp/pip-install-wglwufhp/pyconfig Check the logs for full command output.
##[error]Process completed with exit code 1.

Please document how the user can grant access to private repos to the Action. For example, I solved the problem using the following:

For every Python package, create a step to check out the private repo

    - name: Checkout pyconfig from a private repo
      uses: actions/checkout@v2
      with:
        repository: <company>/pyconfig
        token: ${{ secrets.ACCESS_TOKEN }}
        path: pyconfig

Modify the “Install dependencies” stop to pip install each Python package sourced from a private repo

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install /home/runner/work/<path>/pyconfig

While this works, it is a little tedious. Tech Support suggested I use HTTPS with a username and password to check out the packages from the private repositories. I would prefer to not use this method. It would likely require me to create and maintain a "fake" user account just for checking repositories in GitHub Actions. I would much prefer to use a personal access token (like I did above), but in a more simplified manner.

Is there a canonical way to do this now in 2022?

I would also love to know if there was a canonical way to do this in 2022.

So far the best way I've found to do this is by creating a Machine User and inviting it to the organization.

Then, in requirements.txt you can add lines like:

git+ssh://git@github.com/<org>/<package>.git

Generate and add an SSH key for the machine user, then use something like

shimataro/ssh-key-action

to set the key in your workflow from a secret.

There is more information at Using organization Python package in Github actions without Python repository although this only seems to work for one package and so that's why I ended up having to use the machine user approach as usually if you depend on one dependency in an organization you will depend on others.

Hope somebody finds this useful.

Not really canonical but this workaround might be a little better:

Just add
git config --global url."https://${{ secrets.GH_TOKEN }}@github".insteadOf https://github
at the beggining of your run step (before installing dependencies).

I tackled the same issue a long time ago. My conclusion at the time was much the same as the one described by croth.

Then I learned that GitHub recommends authentication via HTTPS, and I felt the need to reconsider this issue, but I neglected it.

Recently, I had to do some maintenance on an old project, so I thought about my own best practices at this point. It is a combination of the methods described by croth and victorsevero.

In other words, do the following.

Define requirements.txt and pyproject.toml with the https schema.
Add the following steps at the beginning of the CI
- Register private key with ssh-agent.
- Add URL rewrite setting to Git config
  - ex. git config --global url."ssh://git@github.com/".insteadOf https://github.com/

I checked and found that for GitHub Actions, this can be easily accomplished by using webfactory/ssh-agent.
The README is very carefully written, so I recommend reading it.

I have the following up and running successful:

    steps:
      # https://github.com/actions/checkout
      - name: checkout
        uses: actions/checkout@v3.2.0

      # https://github.com/marketplace/actions/setup-python
      - name: Setup Python
        uses: actions/setup-python@v4.3.1
        with:
          python-version: "3.10"

      - name: replace requirements
        run: sed -i "s/ssh:\/\/git@github\.com/https:\/\/${{ secrets.GH_USERNAME }}:${{ secrets.GH_PAT_WITH_ACCESS_TO_OTHER_REPOS }}@github\.com/g" requirements.txt

      - name: install
        run: pip install -e . && pip install -r requirements.txt

      - name: test
        run: python test.py

will try out another variant with a gh app token.

@victorsevero

Same issue, and had to resort to @victorsevero (and others') solution

There's another option now that I believe is a bit more secure than managing a separate machine user and its associated credentials: Create a new Github App with "Contents" permissions, add it to the private repositories that the Github Action needs to access, and use something like https://github.com/marketplace/actions/action-github-app-token to get an access token that can be used during git checkout. Unfortunately this still requires the app's private key to be saved as a secret for the Action.

@aripollak

@aripollak this solution works Pretty Good as of today, and has a nice restrictive access element to it. Thank you 🙇🏽‍♂️

@tyriis

@tyriis Thank you for this elegant dependency-free solution!

@aripollak

@aripollak How did you manage to successfully use a Github App token with pip? I can use the private_key as a way of configuring the checkout action, but unsure how to re-use the private key to let pip authenticate and clone dependencies.
I figured I could use the private key and setup an ssh agent https://github.com/marketplace/actions/webfactory-ssh-agent using this action, but this does not work. I might have to check how the checkout action does it and go from there

@victorsevero

So I got it all wrong on my previous comment.
If you are authenticating with a Github App, you can follow these steps to create a new one:

https://github.com/proxy/docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app

Then, once your app has been created and added to the required repos (or if you add it to the top level of your organization, you can select which repos the app has access to), you can use create-github-app-token action to generate an installation token that can access multiple repositories.

The only issue is this token can be used to sign-in using https. We can override global urls with git config --global url as stated in the comments above by @victorsevero and @fuji44.
A similar approach is followed here:
https://github.com/actions/checkout/blob/main/src/git-auth-helper.ts

In my case I was using ssh in my requirements.txt file, so these also get overriden by the global git configuration.

    - name: Get our app token from Github App
      uses: actions/create-github-app-token@v1
      id: app_token
      with:
        app-id: ${{ secrets.APP_ID }}
        private-key: ${{ secrets.APP_PRIVATE_KEY }}
        # owner is required, otherwise the creds will fail the checkout step
        owner: ${{ github.repository_owner }}
    - name: Checkout from GitHub
      uses: actions/checkout@v4.1.0
      with:
        submodules: true # In my case I'm also using this token to clone sub-modules
        token: ${{ steps.app_token.outputs.token }}
    # This step is necessary to allow pip install private packages hosted in github.
    - name: Setup token for Python installation
      run: git config --global url."https://oauth2:${GH_TOKEN}@github.com".insteadOf ssh://git@github.com # replace with whatever authentication method you're currently using
      env:
        GH_TOKEN: ${{ steps.app_token.outputs.token }}

Thanks to this person for suggesting this approach:
actions/checkout#287 (comment)

more info on github app authentication:
https://github.com/proxy/docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/making-authenticated-api-requests-with-a-github-app-in-a-github-actions-workflow
and generating authentication tokens for an installation (which is what create-github-app-token does for us):
https://github.com/proxy/docs.github.com/en/rest/apps/apps?apiVersion=2022-11-28#create-an-installation-access-token-for-an-app

Remember also to create your APP_ID and PRIVATE_KEY secret variables from your App configuration:
Store the App's ID in your repository environment variables (example: APP_ID)
Store the App's private key in your repository secrets (example: PRIVATE_KEY)

@victorsevero

So I got it all wrong on my previous comment. If you are authenticating with a Github App, you can follow these steps to create a new one:

https://github.com/proxy/docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app

Then, once your app has been created and added to the required repos (or if you add it to the top level of your organization, you can select which repos the app has access to), you can use create-github-app-token action to generate an installation token that can access multiple repositories.

The only issue is this token can be used to sign-in using https. We can override global urls with git config --global url as stated in the comments above by @victorsevero and @fuji44. A similar approach is followed here: https://github.com/actions/checkout/blob/main/src/git-auth-helper.ts

In my case I was using ssh in my requirements.txt file, so these also get overriden by the global git configuration.
    - name: Get our app token from Github App
      uses: actions/create-github-app-token@v1
      id: app_token
      with:
        app-id: ${{ secrets.APP_ID }}
        private-key: ${{ secrets.APP_PRIVATE_KEY }}
        # owner is required, otherwise the creds will fail the checkout step
        owner: ${{ github.repository_owner }}
    - name: Checkout from GitHub
      uses: actions/checkout@v4.1.0
      with:
        submodules: true # In my case I'm also using this token to clone sub-modules
        token: ${{ steps.app_token.outputs.token }}
    # This step is necessary to allow pip install private packages hosted in github.
    - name: Setup token for Python installation
      run: git config --global url."https://oauth2:${GH_TOKEN}@github.com".insteadOf ssh://git@github.com # replace with whatever authentication method you're currently using
      env:
        GH_TOKEN: ${{ steps.app_token.outputs.token }}
Thanks to this person for suggesting this approach: actions/checkout#287 (comment)

more info on github app authentication: https://github.com/proxy/docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/making-authenticated-api-requests-with-a-github-app-in-a-github-actions-workflow and generating authentication tokens for an installation (which is what create-github-app-token does for us): https://github.com/proxy/docs.github.com/en/rest/apps/apps?apiVersion=2022-11-28#create-an-installation-access-token-for-an-app

Remember also to create your APP_ID and PRIVATE_KEY secret variables from your App configuration: Store the App's ID in your repository environment variables (example: APP_ID) Store the App's private key in your repository secrets (example: PRIVATE_KEY)

Does this method work in an organization setting? I have two private repositories, one is a python package and other one is where I want to install. I did the things mentioned, but it fails. Could it be because the repositories are in organization? I get the following error:

runner@github.com: Permission denied (publickey).
  fatal: Could not read from remote repository.

@mohit2512sharma

@mohit2512sharma . This works in an organization and it is what we're currently using.
Make sure your GitHub app has permissions to access your organization repositories. Also make sure that you're referencing your GitHub repos in your requirements as ssh so that they get replaced correctly by the git global configuration

@mohit2512sharma

@mohit2512sharma . This works in an organization and it is what we're currently using.
Make sure your GitHub app has permissions to access your organization repositories. Also make sure that you're referencing your GitHub repos in your requirements as ssh so that they get replaced correctly by the git global configuration

Thanks for your comment. @pmabres . Your comment does work, unfortunately I had a typo in my requirements.txt and for hours I couldn't find it.

I successfully used the app token method, but ran into issues with conflicting git configs (and token persistence) on a self-hosted runner.

This is the config I ended up with to isolate the git config --global ...insteadOf ... line to a single run. I'm not sure if changing HOME breaks some pip native build process, but it seems like the way to go for git and is working for me so far.

      - name: Generate Repo Access Token
        uses: actions/create-github-app-token@v1
        id: app-token
        with:
          app-id: ${{ vars.GH_APP_ID }}
          private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
          owner: ${{ github.repository_owner }}

      - name: Install Public Python dependencies
        run: |
          source venv/bin/activate
          pip install -r requirements.txt

      - name: Install Private Python dependencies
        run: |
          source venv/bin/activate

          # https://github.com/actions/setup-python/issues/104#issuecomment-2187580389

          mkdir -p "$RUNNER_TEMP/git"
          HOME="$RUNNER_TEMP/git" git config --global url."https://oauth2:${GH_TOKEN}@github.com".insteadOf ssh://git@github.com

          # Setting HOME to a folder with .gitconfig seems the only way to get the git config to work
          # I can't find an option to use the GIT_CONFIG environment variable with git clone
          HOME="$RUNNER_TEMP/git" pip install -r requirements-private.txt
        env:
          GH_TOKEN: ${{ steps.app-token.outputs.token }}

The app_token plus git config oauth step is perfect when you already have the helper bot setup. Thanks for keeping this issue alive.

@dthomas-sensonix

Hello @dthomas-sensonix👋,
Thank you for reporting this issue, and apologies for the delayed response.

To clarify, the setup-python action is designed to install and manage Python versions within GitHub Actions workflows. However, it does not handle authentication for pip or git when accessing private repositories, as this falls outside the direct scope of setup-python action.

For guidance on configuring authentication with pip when installing packages from private repositories, you may find pip’s documentation on VCS support and authentication helpful. For more specific assistance with pip authentication, you might also consider reaching out to the pypa/pip repository or the GitHub Community discussions.

If you have any further questions or suggestions specifically related to setup-python, please feel free to let us know. Thank you again for your understanding!

@dthomas-sensonix

Hello @dthomas-sensonix👋,
As mentioned in my previous comment, since the setup-python action does not handle authentication for pip when installing packages from private repositories, we've updated our documentation to clarify this in the linked PR#1156. Thank you!

@dthomas-sensonix

Hello @dthomas-sensonix,
We are proceeding to close this issue now that the PR#1156 has been merged. If you have any further questions or concerns, please feel free to reach out. Thank you!

Navigation Menu

Repository navigation

Document How To Install Python Package from a Private Repo #104

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Document How To Install Python Package from a Private Repo #104

Description

Activity

ctoth commented on Aug 14, 2022

SJCaldwell commented on Sep 28, 2022

ctoth commented on Sep 29, 2022

victorsevero commented on Oct 24, 2022

fuji44 commented on Dec 2, 2022

tyriis commented on Jan 13, 2023

blieusong commented on Mar 11, 2023

aripollak commented on Apr 27, 2023

kenichi commented on Apr 11, 2024

Aylr commented on May 16, 2024

pmabres commented on Jun 24, 2024

pmabres commented on Jun 24, 2024

mohit2512sharma commented on Sep 17, 2024

pmabres commented on Sep 17, 2024

mohit2512sharma commented on Sep 18, 2024

chrismooredev commented on Apr 24, 2025

sarnold commented on May 24, 2025

v-priya-kinthali commented on Jun 26, 2025

v-priya-kinthali commented on Jul 18, 2025

v-priya-kinthali commented on Jul 25, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions