Screaming Frog spider comes highly beneficial for SEARCH ENGINE OPTIMIZATION and internet designers all over the place.
I've used it on small and enormous websites (using a cloud laptop to maximise its effectivity). You will get a seo software free version too to check it out.