python - Aiohttp, Asyncio: RuntimeError: Event loop is closed -
i have 2 scripts, scraper.py , db_control.py. in scraper.py have this:
... def scrap(category, field, pages, search, use_proxy, proxy_file): ... loop = asyncio.get_event_loop() to_do = [ get_pages(url, params, conngen) url in urls ] wait_coro = asyncio.wait(to_do) res, _ = loop.run_until_complete(wait_coro) ... loop.close() return [ x.result() x in res ] ...
and in db_control.py:
from scraper import scrap ... while new < 15: data = scrap(category, field, pages, search, use_proxy, proxy_file) ... ...
theoretically, scrapper should started unknown-times until enough of data have been obtained. when new
not imidiatelly > 15
error occurs:
file "/usr/lib/python3.4/asyncio/base_events.py", line 293, in run_until_complete self._check_closed() file "/usr/lib/python3.4/asyncio/base_events.py", line 265, in _check_closed raise runtimeerror('event loop closed') runtimeerror: event loop closed
but scripts works fine if run scrap() once. guess there problem recreating loop = asyncio.get_event_loop()
, have tried this nothing changed. how can fix this? of course snippets of code, if think problem can elsewhere, full code available here.
methods run_until_complete
, run_forever
, run_in_executor
, create_task
, call_at
explicitly check loop , throw exception if it's closed.
quote docs - baseevenloop.close
:
this idempotent , irreversible
unless have some(good) reasons, might omit close line:
def scrap(category, field, pages, search, use_proxy, proxy_file): #... loop = asyncio.get_event_loop() to_do = [ get_pages(url, params, conngen) url in urls ] wait_coro = asyncio.wait(to_do) res, _ = loop.run_until_complete(wait_coro) #... # loop.close() return [ x.result() x in res ]
if want have each time brand new loop, have t create manually , set default:
def scrap(category, field, pages, search, use_proxy, proxy_file): #... loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) to_do = [ get_pages(url, params, conngen) url in urls ] wait_coro = asyncio.wait(to_do) res, _ = loop.run_until_complete(wait_coro) #... return [ x.result() x in res ]
Comments
Post a Comment