This is part 3 of my selenium exploration trying to fetch stories from the NY Times ((as a subscriber).
- Part I: Selenium Basics
- Part II: Running Headless on a Server
- Part III: Handling Errors and Timeouts (this article)
At the end of Part II, selenium was running on a server with the minimal number of X and GTK libraries installed.
But now that it can run unattended, there's nother problem: there are all kinds of ways this can fail, and your script needs to handle those errors somehow.
Before diving in, I should mention that for my original goal, fetching stories from the NY Times as a subscriber, it turned out I didn't need selenium after all. Since handling selenium errors turned out to be so brittle (as I'll describe in this article), I'm now using requests combined with a Python CookieJar. I'll write about that in a future article. Meanwhile ...
Handling Errors and Timeouts
Timeouts are a particular problem with selenium,
because there doesn't seem to be any reliable way to change them
so the selenium script doesn't hang for ridiculously long periods.
[ 12:07 Nov 11, 2021 More programming | permalink to this entry | ]