Repository navigation
Segfault when worker thread throws an exception #2265
Description
See https://github.com/proxy/gist.github.com/tomhughes/0ebdc537b6a9a390b904d394d796b5e0
@pnorman Please add all the details about what you were doing here.
What version of osm2pgsql are you using?
2.0.0+ds-1~bpo12+1
What operating system and PostgreSQL/PostGIS version are you using?
Debian 12
Tell us something about your system
OpenStack Virtual Machine with 64 x 1 core AMD EPYC Processor, 444GB
What did you do exactly?
export LUA_PATH='/srv/vector.openstreetmap.org/osm2pgsql-themepark/lua/?.lua;/srv/vector.openstreetmap.org/spirit/?.lua;;'
# Import the osm2pgsql file specified as an argument, using the locations for spirit
osm2pgsql \
--output flex \
--style '/srv/vector.openstreetmap.org/spirit/shortbread.lua' \
--slim \
--flat-nodes '/srv/vector.openstreetmap.org/data/nodes.bin' \
-d spirit \
--cache 75000 data.pbf
What did you expect to happen?
The planet to import.
What did happen instead?
A core dump occurred in the post-processing stage
What did you do to try analyzing the problem?
@tomhughes provided the backtrace at https://github.com/proxy/gist.github.com/tomhughes/0ebdc537b6a9a390b904d394d796b5e0
The machine had an unusual postgresql setup at the time. When starting the import it had sufficient connection slots for osm2pgsql to start, but by the time post-processing had started it would not be possible to acquire new connections. When I fixed this error in the machine configuration it worked fine.
Activity
joto commentedon Oct 19, 2024
Strange. The backtrace does not fit with the idea that the database connection failed. I am not sure that's what the problem was. What was this "unusual postgresql setup"? I tried limiting the number of connections and osm2pgsql properly reported an error and exited.
pnorman commentedon Oct 21, 2024
Too few
max_connections. I fixed it by properly stopping chef and the replication process.joto commentedon Oct 21, 2024
I am pretty sure it is not the
max_connectionswhich caused this. At least when I try that it seems to work correctly. I'd rather expect some race condition that went one way in the first try and the other in the seconds. Is there any chance you can re-create the situation with the too smallmax_connectionand try again? At the moment I don't know how to debug this until we get a reproducible way to trigger the bug.pnorman commentedon Oct 23, 2024
I can see if I have a suitable system
joto commentedon Dec 12, 2024
I believe I have figured out what is happening here: There are several threads in the thread pool. Thread A is started and opens a database connection and all is fine so far. Then thread B is started and doesn't get a database connection any more, it throws an exception. The exception is propagated and as part of that propagation lots of things are destructed. At some point the data structure containing the information about tables is destructed. Now thread A gets a chance to run again and wants to build the
CREATE INDEXcommand. It needs the information about the tables, but that was destructed already. And then it segfaults.To solve this we would need to make sure all threads are destructed before anything else. But there is no way to kill a running std::thread. We would have to wait until it is done with its work, which doesn't make much sense, because that situation isn't recoverable anyway. In C++20 there is a new
std::jthreadwhich has some mechanism for stopping it from the outside, but I believe that also only works if the thread cooperates. But if that thread just calledCREATE INDEXit might be a long time before it even gets to run and can notice that it should shutdown.At the moment, I don't have a good idea how to solve this. :-(
tomhughes commentedon Dec 12, 2024
Yes
jthreadstop tokens are essentially co-operative so you'd probably have to combine that with use ofPQcancelCreateto hold a cancellation object from each connection that you could callPQcancelBlockingon at the same time you requested the thread to stop.[-]Segfault happening[/-][+]Segfault when worker thread throws an exception[/+]