March 21, 2023
Tutorial

Poetry and Private Git Dependencies

Read on to find out how to add and update a private GitHub repo as a poetry dependency, tried and tested by our very own James Marchant here at The Data Refinery.
Poetry and Private Git Dependencies

If like us at The Data Refinery you have a private repository of code that you use across different projects and use poetry to manage dependencies within one of your python projects then you may have wondered how you would add that dependency.

Like many software or data engineers (or however you identify) the first thing you may do is head over to your favourite search engine and use a term something like “poetry add private git repo”.

For me I was greeted by a few links to the poetry docs which didn’t include what I was looking to do, some blogs which also didn’t detail the specific issue and some github issues within the poetry project.

So, with a failed web search I decided to start looking through the poetry cli documentation and found what I needed nestled in the add command. Makes sense right! So, the examples there are:

For https:
poetry add git+https://github.com/sdispater/pendulum.git
For ssh:
poetry add git+ssh://git@github.com/sdispater/pendulum.git
Or alternatively:
poetry add git+ssh://git@github.com:sdispater/pendulum.git
You can also reference a subdirectory if that’s something you’d like to do:
poetry add git+https://github.com/myorg/mypackage_with_subdirs.git@main#subdirectory=subdir

Fantastic, easy peasey lemon… oh wait! I’ve made a change to my shared repo to support a new feature across our organisation, let me just update that dependency…

This happens frequently for us, we are developing new features all the time and at pace. How does poetry know to update that dependency? Well, it doesn’t. So, what do you do when you want to get the latest changes from your private repo?

Well the first thing I tried was a poetry update thinking that would update the dependency with the new code from our private repo… sadly that didn’t seem to work for me.

I then decided to look in the poetry.lock file and searched for the name of our private repo. There were some lines which looked something like this:

[package.source]
type = "git"
url = "https://github.com/OurOrganisation/our-shared-repo.git"
reference = "main"
resolved_reference = "d5e5df52b7c0d8ddfbd474c7e196c06554568ce7"

So, it would appear poetry is saving a reference to a specific version of our private repo. I say it would appear because I couldn’t find any reference to the resolved_reference in their documentation. Ironic, eh?

What next then? Well the only thing I could think to do next was to remove the private repo.

To do this I first did a poetry show to list the dependencies installed by poetry then found the one I was looking for then performed a poetry remove and re-added it just like I did initially with the poetry add command.

That’s it! I hope this helps anyone in the future who wants to add a private repo to poetry and avoids them from going through the pain I did to get this all working.

Maximise your data value

See how The Data Refinery can unlock the value in your data.
Data Engineer at The Data Refinery

Check out our latest posts

Book your free 30 minute
data value review

Are you ready to supercharge your organisation’s analytics & AI capabilities,
but not quite ready to invest £100,000s for a bespoke solution?