Repository navigation
Investigate failure to load caches above 2^32 - 1 bytes #336
Description
Activity
Looked some more into this: the person who reported this said that
torchwas serving 2 GB+ wheels, but I can't see any: https://pypi.org/project/torch/#filesThat being said, I suspect this is still causing unnecessary cache misses due to #200: we end up storing large downloads (such as 700 MB torche wheels) that never get "hit", since the default
msgpackload behavior is to limit binary bodies to ~100MB: https://msgpack-python.readthedocs.io/en/latest/api.htmlHmm, I've still been unable to trigger this: it looks like
msgpack.loads(payload)sets its maximum limits based onlen(payload), so we should never really hit a binary object limit in practice.2^32 - 1 GB
You spooked me -- you probably want to say bytes here.
Reacted by William WoodruffIndeed, sorry!
- changed the title
[-]Investigate failure to load caches above `2^32 - 1` GB[/-][+]Investigate failure to load caches above `2^32 - 1` bytes[/+]on Jul 13, 2024
Metadata
Metadata
Assignees
Labels
No labels
Opening this as a reminder to myself.
This is likely related to #238 and #200: some recent
torchwheels are>= 2.5GB, andpipappears to download them repeatedly without hitting the cache. My only SWAG so far is that this is because thebodyitself overflowsmsgpack's signed 32 bit limit on binary objects, per the spec.Haven't fully diagnosed yet.
See: https://news.ycombinator.com/item?id=40659973