python/python-Scrapy
SHA256
14
0
Fork 0
forked from pool/python-Scrapy
Code Pull Requests Activity

Commit Graph

  • 8062fb28e7 Accepting request 1296688 from devel:languages:python main Dominique Leuenberger 2025-07-31 15:47:02 +00:00
  • 99a9fd84ae skip another test Steve Kowalik 2025-07-31 05:18:58 +00:00
  • 820484d3f8 - Update to 2.13.3: * Changed the values for DOWNLOAD_DELAY (from 0 to 1) and CONCURRENT_REQUESTS_PER_DOMAIN (from 8 to 1) in the default project template. * Fixed several bugs in the engine initialization and exception handling logic. * Allowed running tests with Twisted 25.5.0+ again and fixed test failures with lxml 6.0.0. * Give callback requests precedence over start requests when priority values are the same. * The asyncio reactor is now enabled by default * Replaced start_requests() (sync) with start() (async) and changed how it is iterated. * Added the allow_offsite request meta key * Spider middlewares that don't support asynchronous spider output are deprecated * Added a base class for universal spider middlewares - Add patch remove-hoverxref.patch: * Do not use deprecated sphinx-hoverxref extension. - Add patch no-dark-mode.patch: * Do not use unavailable sphinx-rtd-dark-mode extension. Steve Kowalik 2025-07-31 04:43:53 +00:00
  • 18faaccdb2 Accepting request 1264848 from devel:languages:python Ana Guerrero 2025-04-16 18:38:35 +00:00
  • 17fd446a8c - Normalize metadata directory name. Steve Kowalik 2025-03-27 05:46:50 +00:00
  • 83613729bd Accepting request 1227933 from devel:languages:python Ana Guerrero 2024-12-03 19:47:04 +00:00
  • 969a5b5698 - Update to 2.12.0: * Dropped support for Python 3.8, added support for Python 3.13 * start_requests can now yield items * Added scrapy.http.JsonResponse * Added the CLOSESPIDER_PAGECOUNT_NO_ITEM setting Steve Kowalik 2024-12-03 08:25:27 +00:00
  • 6dac99b3c7 Accepting request 1186841 from devel:languages:python Ana Guerrero 2024-07-11 18:33:34 +00:00
  • 1c6fbdfae1 OBS-URL: https://build.opensuse.org/package/show/devel:languages:python/python-Scrapy?expand=0&rev=42 Dirk Mueller 2024-07-11 11:08:02 +00:00
  • add99b967c - update to 2.11.2 (bsc#1224474, CVE-2024-1968): * Redirects to non-HTTP protocols are no longer followed. Please, see the 23j4-mw76-5v7h security advisory for more information. (:issue:457) * The Authorization header is now dropped on redirects to a different scheme (http:// or https://) or port, even if the domain is the same. Please, see the 4qqq-9vqf-3h3f security advisory for more information. * When using system proxy settings that are different for http:// and https://, redirects to a different URL scheme will now also trigger the corresponding change in proxy settings for the redirected request. Please, see the jm3v-qxmh-hxwv security advisory for more information. (:issue:767) * :attr:Spider.allowed_domains <scrapy.Spider.allowed_domains> is now enforced for all requests, and not only requests from spider callbacks. * :func:~scrapy.utils.iterators.xmliter_lxml no longer resolves XML entities. * defusedxml is now used to make :class:scrapy.http.request.rpc.XmlRpcRequest more secure. * Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in favor of brotli. (:issue:6261) Note brotlipy is deprecated, both in Scrapy and upstream. Use brotli instead if you can. * Make :setting:METAREFRESH_IGNORE_TAGS ["noscript"] by default. This prevents :class:~scrapy.downloadermiddlewares. redirect.MetaRefreshMiddleware from following redirects that would not be followed by web browsers with JavaScript enabled. Dirk Mueller 2024-07-11 10:53:38 +00:00
  • 053a125313 Accepting request 1164153 from devel:languages:python Ana Guerrero 2024-04-03 15:19:30 +00:00
  • 542431ad3b Accepting request 1161494 from devel:languages:python Ana Guerrero 2024-03-27 19:41:53 +00:00
  • 04481ebc46 baserev update by copy to link target OBS User buildservice-autocommit 2024-03-27 19:41:53 +00:00
  • 3087fe5d77 Updating link to change in openSUSE:Factory/python-Scrapy revision 19 OBS User buildservice-autocommit 2024-03-27 19:41:53 +00:00
  • 3df144e0aa - update to 2.11.1 (bsc#1220514, CVE-2024-1892, bsc#1221986): advisory`_ for more information. (bsc#1221986) Dirk Mueller 2024-03-26 15:10:26 +00:00
  • 119328cdce - update to 2.11.1 (bsc#1220514, CVE-2024-1892): * Addressed ReDoS vulnerabilities (bsc#1220514, CVE-2024-1892) - `scrapy.utils.iterators.xmliter is now deprecated in favor of :func:~scrapy.utils.iterators.xmliter_lxml, which :class:~scrapy.spiders.XMLFeedSpider now uses. To minimize the impact of this change on existing code, :func:~scrapy.utils.iterators.xmliter_lxml now supports indicating the node namespace with a prefix in the node name, and big files with highly nested trees when using libxml2 2.7+. - Fixed regular expressions in the implementation of the :func:~scrapy.utils.response.open_in_browser function. .. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS * :setting:DOWNLOAD_MAXSIZE and :setting:DOWNLOAD_WARNSIZE now also apply to the decompressed response body. Please, see the 7j7m-v7m3-jqm7 security advisory_ for more information. .. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7 * Also in relation with the 7j7m-v7m3-jqm7 security advisory_, the deprecated scrapy.downloadermiddlewares.decompression module has been removed. * The Authorization header is now dropped on redirects to a different domain. Please, see the cw9j-q3vf-hrrv security advisory_ for more information. * The OS signal handling code was refactored to no longer use private Twisted functions. (:issue:6024, :issue:6064, :issue:6112) * Improved documentation for :class:~scrapy.crawler.Crawler initialization changes made in the 2.11.0 release. (:issue:6057, :issue:6147) * Extended documentation for :attr:Request.meta <scrapy.http.Request.meta>. * Fixed the :reqmeta:dont_merge_cookies documentation. (:issue:5936, * Added a link to Zyte's export guides to the :ref:feed exports * Added a missing note about backward-incompatible changes in Dirk Mueller 2024-03-25 15:36:37 +00:00
  • 0ad62694dd Accepting request 1137882 from devel:languages:python Ana Guerrero 2024-01-10 20:52:52 +00:00
  • feb6ce6077 - Disable flaky test Daniel Garcia 2024-01-10 08:44:17 +00:00
  • f93a35cd30 - Add patch twisted-23.8.0-compat.patch gh#scrapy/scrapy#6064 - Update to 2.11.0: - Spiders can now modify settings in their from_crawler methods, e.g. based on spider arguments. - Periodic logging of stats. - Bug fixes. - 2.10.0: - Added Python 3.12 support, dropped Python 3.7 support. - The new add-ons framework simplifies configuring 3rd-party components that support it. - Exceptions to retry can now be configured. - Many fixes and improvements for feed exports. - 2.9.0: - Per-domain download settings. - Compatibility with new cryptography and new parsel. - JMESPath selectors from the new parsel. - Bug fixes. - 2.8.0: - This is a maintenance release, with minor features, bug fixes, and cleanups. Daniel Garcia 2024-01-10 07:53:57 +00:00
  • 7967a165bb Accepting request 1034478 from devel:languages:python Dominique Leuenberger 2022-11-09 11:56:49 +00:00
  • 47fd8f7029 Accepting request 1034369 from home:yarunachalam:branches:devel:languages:python Markéta Machová 2022-11-08 09:53:03 +00:00
  • e6503c2be7 Accepting request 1032071 from devel:languages:python Dominique Leuenberger 2022-10-29 18:16:47 +00:00
  • d9752627a9 Accepting request 1031641 from home:yarunachalam:branches:devel:languages:python Matej Cepl 2022-10-28 22:27:39 +00:00
  • c966983550 Accepting request 1002736 from devel:languages:python Dominique Leuenberger 2022-09-12 17:08:23 +00:00
  • a84bf5033f Accepting request 1002338 from home:yarunachalam:branches:devel:languages:python Dirk Mueller 2022-09-12 08:00:07 +00:00
  • d418d5a4b7 Accepting request 959733 from devel:languages:python Dominique Leuenberger 2022-03-06 17:48:43 +00:00
  • bc00530500 Accepting request 959304 from home:bnavigator:branches:devel:languages:python Dirk Mueller 2022-03-06 16:31:19 +00:00
  • 2e78e2e37d Accepting request 958587 from devel:languages:python Dominique Leuenberger 2022-03-03 23:17:11 +00:00
  • b1973a8506 Fix changelogs Matej Cepl 2022-03-03 06:01:02 +00:00
  • 7533e3a14a - Upgrade to 2.6.1: - Remove unnecessary patches: - remove-h2-version-restriction.patch - add-peak-method-to-queues.patch Matej Cepl 2022-03-02 23:14:08 +00:00
  • 548e19fdca Accepting request 946882 from devel:languages:python Dominique Leuenberger 2022-01-17 21:34:04 +00:00
  • edd1727cd3 Accepting request 946843 from home:bnavigator:branches:devel:languages:python Matej Cepl 2022-01-17 06:30:18 +00:00
  • e3d3aaef29 Accepting request 924057 from devel:languages:python Dominique Leuenberger 2021-10-07 22:06:30 +00:00
  • 0e50613c60 Accepting request 923811 from home:bnavigator:branches:devel:languages:python Matej Cepl 2021-10-07 16:57:39 +00:00
  • bacd15a4d3 Accepting request 917717 from devel:languages:python Dominique Leuenberger 2021-09-09 21:07:43 +00:00
  • ed9c5a3da0 Accepting request 917688 from home:fusionfuture:branches:devel:languages:python Matej Cepl 2021-09-09 12:02:15 +00:00
  • 59e61f6a88 Accepting request 889037 from devel:languages:python Dominique Leuenberger 2021-04-28 23:38:33 +00:00
  • 278632db39 Accepting request 889030 from home:bnavigator:branches:devel:languages:python Markéta Machová 2021-04-28 13:47:21 +00:00
  • 281dc651f3 Accepting request 819355 from devel:languages:python Dominique Leuenberger 2020-07-08 17:13:43 +00:00
  • 50bf9ddc84 Accepting request 818747 from home:winski:python Tomáš Chvátal 2020-07-08 06:42:00 +00:00
  • 769bc702cf Accepting request 807286 from devel:languages:python Dominique Leuenberger 2020-05-19 12:58:12 +00:00
  • 554a84a443 Accepting request 807242 from home:pgajdos:python Tomáš Chvátal 2020-05-19 12:14:06 +00:00
  • 6dcfb7e77c Accepting request 790737 from devel:languages:python Dominique Leuenberger 2020-04-02 15:43:41 +00:00
  • 5577430fb1 - Update to 2.0.1: * Python 2 support has been removed * Partial coroutine syntax support and experimental asyncio support * New Response.follow_all method * FTP support for media pipelines * New Response.certificate attribute * IPv6 support through DNS_RESOLVER * Response.follow_all now supports an empty URL iterable as input * Removed top-level reactor imports to prevent errors about the wrong Twisted reactor being installed when setting a different Twisted reactor using TWISTED_REACTOR - Add zope-exception-test_crawler.patch, rewriting one testcase to pass with our version of Zope. - Update BuildRequires based on test requirements. Steve Kowalik 2020-04-02 03:41:29 +00:00
  • f1afecd802 Accepting request 765023 from devel:languages:python Dominique Leuenberger 2020-01-16 17:24:00 +00:00
  • 535b71edfe Accepting request 765007 from home:mcalabkova:branches:devel:languages:python Tomáš Chvátal 2020-01-16 15:35:55 +00:00
  • 55d75182b8 Accepting request 725773 from devel:languages:python Dominique Leuenberger 2019-08-24 16:48:44 +00:00
  • 9f5d599005 Accepting request 725578 from home:polslinux:branches:devel:languages:python Tomáš Chvátal 2019-08-24 07:38:09 +00:00
  • 1cb1df21ea Accepting request 718179 from devel:languages:python Dominique Leuenberger 2019-07-24 18:36:43 +00:00
  • ddab9089be - Format with spec-cleaner - Use just python3 version of Sphinx Tomáš Chvátal 2019-07-24 10:27:28 +00:00
  • df0b142876 Accepting request 718147 from home:pgajdos Tomáš Chvátal 2019-07-24 09:00:54 +00:00
  • e0bb11edf0 Accepting request 703535 from devel:languages:python Dominique Leuenberger 2019-05-17 21:43:32 +00:00
  • c4b3eee51a Accepting request 703534 from home:anandrit:branches:devel:languages:python Tomáš Chvátal 2019-05-16 19:37:41 +00:00
  • 62ba8354a5 Accepting request 677379 from devel:languages:python Stephan Kulow 2019-02-28 20:41:23 +00:00
  • 8dfa7191b3 Accepting request 677210 from home:frispete:python Tomáš Chvátal 2019-02-19 14:19:30 +00:00