python/python-Scrapy
SHA256
14
0
Fork 0
forked from pool/python-Scrapy
Code Pull Requests Activity

Accepting request 1002338 from home:yarunachalam:branches:devel:languages:python

Browse Source 
- Update to v2.6.2 
  Security bug fix:
  * When HttpProxyMiddleware processes a request with proxy metadata, and that proxy metadata includes proxy credentials,
    HttpProxyMiddleware sets the Proxy-Authentication header, but only if that header is not already set.
  * There are third-party proxy-rotation downloader middlewares that set different proxy metadata every time they process a request.
  * Because of request retries and redirects, the same request can be processed by downloader middlewares more than once,
    including both HttpProxyMiddleware and any third-party proxy-rotation downloader middleware.
  * These third-party proxy-rotation downloader middlewares could change the proxy metadata of a request to a new value,
    but fail to remove the Proxy-Authentication header from the previous value of the proxy metadata, causing the credentials of one
    proxy to be sent to a different proxy.
  * To prevent the unintended leaking of proxy credentials, the behavior of HttpProxyMiddleware is now as follows when processing a request:
    + If the request being processed defines proxy metadata that includes credentials, the Proxy-Authorization header is always updated 
    to feature those credentials.
    + If the request being processed defines proxy metadata without credentials, the Proxy-Authorization header is removed unless
    it was originally defined for the same proxy URL.
    + To remove proxy credentials while keeping the same proxy URL, remove the Proxy-Authorization header.
    + If the request has no proxy metadata, or that metadata is a falsy value (e.g. None), the Proxy-Authorization header is removed.
    + It is no longer possible to set a proxy URL through the proxy metadata but set the credentials through the Proxy-Authorization header.
    Set proxy credentials through the proxy metadata instead.
  * Also fixes the following regressions introduced in 2.6.0:
    + CrawlerProcess supports again crawling multiple spiders (issue 5435, issue 5436)
    + Installing a Twisted reactor before Scrapy does (e.g. importing twisted.internet.reactor somewhere at the module level)
    no longer prevents Scrapy from starting, as long as a different reactor is not specified in TWISTED_REACTOR (issue 5525, issue 5528)
    + Fixed an exception that was being logged after the spider finished under certain conditions (issue 5437, issue 5440)
    + The --output/-o command-line parameter supports again a value starting with a hyphen (issue 5444, issue 5445)
    + The scrapy parse -h command no longer throws an error (issue 5481, issue 5482)

OBS-URL: https://build.opensuse.org/request/show/1002338
OBS-URL: https://build.opensuse.org/package/show/devel:languages:python/python-Scrapy?expand=0&rev=28
This commit is contained in:
Dirk Mueller
2022-09-12 08:00:07 +00:00
committed by Git OBS Bridge
parent bc00530500
commit a84bf5033f
4 changed files with 34 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
python-Scrapy.changes
View File

@@ -1,3 +1,33 @@
-------------------------------------------------------------------
Fri Sep 9 15:21:20 UTC 2022 - Yogalakshmi Arunachalam <yarunachalam@suse.com>
- Update to v2.6.2
Security bug fix:
* When HttpProxyMiddleware processes a request with proxy metadata, and that proxy metadata includes proxy credentials,
HttpProxyMiddleware sets the Proxy-Authentication header, but only if that header is not already set.
* There are third-party proxy-rotation downloader middlewares that set different proxy metadata every time they process a request.
* Because of request retries and redirects, the same request can be processed by downloader middlewares more than once,
including both HttpProxyMiddleware and any third-party proxy-rotation downloader middleware.
* These third-party proxy-rotation downloader middlewares could change the proxy metadata of a request to a new value,
but fail to remove the Proxy-Authentication header from the previous value of the proxy metadata, causing the credentials of one
proxy to be sent to a different proxy.
* To prevent the unintended leaking of proxy credentials, the behavior of HttpProxyMiddleware is now as follows when processing a request:
+ If the request being processed defines proxy metadata that includes credentials, the Proxy-Authorization header is always updated
to feature those credentials.
+ If the request being processed defines proxy metadata without credentials, the Proxy-Authorization header is removed unless
it was originally defined for the same proxy URL.
+ To remove proxy credentials while keeping the same proxy URL, remove the Proxy-Authorization header.
+ If the request has no proxy metadata, or that metadata is a falsy value (e.g. None), the Proxy-Authorization header is removed.
+ It is no longer possible to set a proxy URL through the proxy metadata but set the credentials through the Proxy-Authorization header.
Set proxy credentials through the proxy metadata instead.
* Also fixes the following regressions introduced in 2.6.0:
+ CrawlerProcess supports again crawling multiple spiders (issue 5435, issue 5436)
+ Installing a Twisted reactor before Scrapy does (e.g. importing twisted.internet.reactor somewhere at the module level)
no longer prevents Scrapy from starting, as long as a different reactor is not specified in TWISTED_REACTOR (issue 5525, issue 5528)
+ Fixed an exception that was being logged after the spider finished under certain conditions (issue 5437, issue 5440)
+ The --output/-o command-line parameter supports again a value starting with a hyphen (issue 5444, issue 5445)
+ The scrapy parse -h command no longer throws an error (issue 5481, issue 5482)
-------------------------------------------------------------------
Fri Mar 4 00:06:54 UTC 2022 - Ben Greiner <code@bnavigator.de>