Repository navigation
Switch wget for wget-at for better WARC compliance #17
Description
https://github.com/ArchiveTeam/wget-lua
It's the version that archive team warrior runs and is very well tested + ready to feed directly into archive.org.
https://github.com/ArchiveTeam/wget-lua
It's the version that archive team warrior runs and is very well tested + ready to feed directly into archive.org.
Activity
pirate commentedon Mar 24, 2026
TODO: implement this over on the new plugins repo https://github.com/archiveBox/abx-plugins
devicefricker commentedon Mar 25, 2026
I don't really want to nag you (in another issue, you said you could only work on this on weekends), but can we transfer this to abx-plugins?
pirate commentedon Mar 26, 2026
@claude can you take a crack at adding a new plugin wgetlua thats similar to wget, but it should install wget-at using binprovider overrides defined in config.json.
Install must succeed locally / in the test environment when running
abx-dl install wgetlua(uv pip install -e ../abx-dl. auto-install in macos/linux at runtime is a core part of the user-facing surface area and must succeed in all environments including CI and sandboxes with no mocking or "faked" code paths in tests.Make sure the plugin includes a live integration test that hits a real
https://example.comurl and confirms the warc and html saved actually contain the correct contents. Do not mock the url with a test server and do not mock the wget-at/wgetlua binary, it must succeed an install the real binary and hit a real url and produce real content in the output. Do not simply check that it runs without error, your tests must check that the content saved is actually correct.pirate commentedon Mar 26, 2026
Unlike all our other plugin binary dependencies, it seems wget-at offers no package distribution, prebuilt binaries, or install script. This makes it significantly harder to offer as a plugin, as users would have to install an entire C/C++ build system toolchain.
If that changes and they release a package manager distribution then I'm more likely to add it, otherwise this is very low on my list. I'm more likely to invest in adding a browsertrix/archivweb.page plugin which is great at producing WARC/WACZs.
Add wgetlua plugin for Archive Team wget-lua (wget-at)
pirate commentedon Mar 27, 2026
Vibecoded approach but untested / not going to enable by default with this gnarly of a build step: #19
devicefricker commentedon Mar 27, 2026
Eh, it's only a plugin, nobody will really care if it's vibecoded or not (as long as it doesn't delete the user's D drive!)