Why not analyze your logs, identify illegitimate spiders, and block their IP at your routers or redirect them to junk? That’s what I did when people were stealing my content twenty years ago and it’s a lot easier now.