mip: package installation using GitLab URLs for repos in subgroups #1014

Activity

1. agatti commented on Apr 2, 2026

   

   agatti

   on Apr 2, 2026

   Contributor

   More actions

   Ow. This makes things much more complicated than expected.

   If the manifest file format is set in stone, the location of the user/project/path splits in the gitlab: identifier can be obtained by using the REST API offered by gitlab (see https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi_v3.yaml) [1].

   If altering the manifest format is an option, then extra fields can be used to provide more accurate information (eg. ["https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/", "develop"] in your example becomes {"provider": "gitlab", "user": "micropython-iot-project", "repo": "libraries/micropython-network-utils", "name": "micropython-network-utils", "ref": "develop"}), and thus it lets the URL rewriter fill in the same fields as before but with the right values this time.

   ...or the mip installer could be split into mip, mip-gitlab, mip-<provider>, and so on if there's some special handling required, but maybe that's a bit too much, isn't it?

   [1]

   The /api/v4/projects/{id} endpoint could be used to check if a project exists by recursively adding path components to the query until either we run out of components or we get a successful response. Eg. case of gitlab:user/group/subgroup1/subgroup2/project@develop, on a 404 on user/group, it'd call the API endpoint with user/group/subgroup1 and if it still gets a 404 it will try user/group/subgroup1/subgroup2, etc. The main issues with this approach, complexity aside, are increased code size and that probably access patterns like that might be seen as suspicious by the server and possibly trigger an IP block/bot verification.
2. Josverl commented on Apr 2, 2026

   

   Josverl

   on Apr 2, 2026

   SponsorContributor

   More actions

   @andyrids , I actually ran into this with your library as I happened to pick it for an automated test of the different MIP sources.
   I initially tried just copied part of the url to construct the Uri for MIP. And only after failing started to read the documentation 😎 where you used the Uri you mention in the OP.

   I'm not sure what the seize penalty is of adding just-enough-logic to solve this, withouth changing the package format.

   Alternatives if the size is too much are:

   • document it, with examples
   • create a hosted function/lambda to do the complex lookup, return a http redirect and cache it
3. agatti commented on Apr 3, 2026

   

   agatti

   on Apr 3, 2026

   Last edited by agatti

   Contributor

   More actions

   Assuming manifest format stays the same, the size penalty for the extra lookups it probably isn't too much for mpremote but it might be on the order of two/three hundreds of bytes minimum for mip. After all we don't care about what the response looks like but whether the query completes with a 200 or not [1], so that avoids having to bring in the json module too.

   I'm not the right person to judge the development effort for adding a lambda to do this sort of task, but going that route you add an extra point of failure to the whole lot - which is probably not an issue in this case, but worth keeping in mind - and still require changes to mip anyway.

   Still, I'll have a go at adding the extended manifest format to mip and see how much space it takes. Stay tuned.

   [1] Unless they operate like a company I used to work for, whose web services often returned "200 Fatal Error" :)
4. Josverl commented on Apr 3, 2026

   

   Josverl

   on Apr 3, 2026

   Last edited by Josverl

   SponsorContributor

   More actions

   I don`t know gitlab that well - so below may be overly complex.
   A partial solution/workaround seems to be to get GitLab to create redirects for the repo/project by transferring it to a top level group, and then back to wherever you want.
   Gitlab is kind enough to generate HTTP redirects on the repo level, which also work for transfers between groups / subgroups.

   You will need a unique top-group though

   So after transferring it back and forth, you can mip install gitlab:/jv_sample/jv_subgroup/parrot
   using

   • import mip; mip.install("gitlab:jv_sample/parrot")
   • mpremote mip install gitlab:jv_sample/parrot ( Not tested )

   The same does not seem to work in dependencies though :-(

   • import mip; mip.install("gitlab:jv_sample/petshop") fails as it has a dependency on above parrot
     And I have not found a way to make that work yet - other than coding the full URI

   petshop package

   {
     "urls": [
       ["petshop.py", "petshop.py"]
     ],
       "deps": [
           ["logging", "latest"],
           ["gitlab:josverl/jv_sample/parrot", "main"],
           ["github:josverl/micropython-stubs/mip/typing_mpy.json", "main"],
       ],
       "working"= [
           ["https://gitlab.com/jv_sample/jv_subgroup/parrot/-/raw/main/package.json?ref_type=heads", ""]
       ],
       "version": "0.1"
   }
   
   

   offtopic : ** note the .../typing_mpy.json that also pulls in the 2nd level dependencies