Solving problems related to http.client.RemoteDisconnected: Remote end closed connection without response

When encountering this kind of problem, when using the urllib or requests library in the Python crawler library to obtain data, most websites will detect the User-Agent in the user request . If the User-Agent is not set in the request header , then Related exceptions will be thrown

Reason 1:

Most websites will detect the User-Agent in the user request. If the User-Agent is not set in the request header, an exception will be thrown. Improvement method: Add: User-Agent in the headers.

Reason 2:

Here I use the http protocol, and the requested URL is also the http protocol. When the test website at this location is changed to the https://blog.csdn.net/lovlu18 protocol, it can still get a response. This is because there is no SSL encryption application layer intervention.

It is still recommended that you use the http protocol URL.

In layman's terms, if the problem has not been solved after adding UA, check to see if the website you requested is https. If so, try changing it to http. It is recommended that you use the URL of the http protocol.