I was watching
htop for a while and noticed it was quite active for just monitoring the mempool. It seemed like it was working darn hard for just a few txs/second on the network. So reviewing the code I noticed it was looking up every tx in the mempool every time it cycled through, which is currently set at 5 seconds, and that's 46k odd lookups. Well, that's dumb. It should only lookup new txs in the mempool. I moved a few lines of code around to achieve this and ran some tests. Voila!
I've also had occasional crashes where an RPC call returned "httplib.CannotSendRequest" exceptions. With a bit of Google I figured it was contention between threads causing this; when a new request is attempted before the last has responded. Not sure why, as they should each have their own connection; and yet there it is. So I'm currently testing some new code for an RPC Pool wrapper. This opens a few connections at the start and then they can be grabbed as needed. It also traps the errors and tries to repair failed connections. Previously a bad result could nuke one thread leaving others peddling away aimlessly. Hopefully this will be a step towards more robust behaviour.