Я использую API-интерфейс поиска новостей bing для сбора новостей по заданным ключевым словам, однако я довольно часто теряю время.

Это код, о котором идет речь:

@app.route('/getAcadNews')
def getAadNews():
    headers = {
        'accept': "application/json",
        'cache-control': "no-cache",
        'Ocp-Apim-Subscription-Key': "<API KEY HERE>"
    }

    news_items = {}

    query = removeBadOrdinal(request.args.get('kws'))
    url = "https://api.cognitive.microsoft.com/bing/v5.0/news/search?"

    querystring = {"q":"\""+query+"\" and \"papers published\"","count":"10000","freshness":"Month"}
    try:
        papernews = json.loads(requests.request("GET", url, headers=headers, params=querystring).text)
    except requests.exceptions.ConnectionError:
        papernews = {}

    if "totalEstimatedMatches" in papernews:
        news_items['papers_published_news'] = papernews["totalEstimatedMatches"]
        news_items['papers_published_news_item'] = getTopNews(papernews)
    else:
        news_items['papers_published_news'] = 0
        news_items['papers_published_news_item'] = {}

    querystring = {"q":"\""+query+"\" and \"patent\"","count":"10000","freshness":"Month"}
    try:
        patentnews = json.loads(requests.request("GET", url, headers=headers, params=querystring).text)
    except requests.exceptions.ConnectionError:
        patentnews = {}
    if "totalEstimatedMatches" in patentnews:
        news_items['patent_news'] = patentnews["totalEstimatedMatches"]
        news_items['patent_news_item'] = getTopNews(patentnews)
    else:
        news_items['patent_news'] = 0
        news_items['patent_news_item'] = {}

    resp1 = (json.dumps(news_items, indent=4))
    resp = Response(response=resp1,
        status=200, \
        mimetype="application/json")
    return(resp)

Иногда это работает, но иногда я сталкиваюсь со следующей ошибкой:

ERROR in app: Exception on /getAcadNews [GET]
  Traceback (most recent call last):
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 142, in _new_conn
      (self.host, self.port), self.timeout, **extra_kw)
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py", line 98, in create_connection
      raise err
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py", line 88, in create_connection
      sock.connect(sa)
  TimeoutError: [Errno 110] Connection timed out
  During handling of the above exception, another exception occurred:
  Traceback (most recent call last):
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
      chunked=chunked)
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 352, in _make_request
      self._validate_conn(conn)
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 831, in _validate_conn
      conn.connect()
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 254, in connect
      conn = self._new_conn()
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 151, in _new_conn
      self, "Failed to establish a new connection: %s" % e)
  requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d7a40b080>: Failed to establish a new connection: [Errno 110] Connection timed out
  During handling of the above exception, another exception occurred:
  Traceback (most recent call last):
    File "/root/DataParser/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
      timeout=timeout
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 640, in urlopen
      _stacktrace=sys.exc_info()[2])
    File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 287, in increment
      raise MaxRetryError(_pool, url, error or ResponseError(cause))
  requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.cognitive.microsoft.com', port=443): Max retries exceeded with url: /bing/v5.0/news/search?count=10000&q=%22+mobile+payments+platform%22+and+%22papers+published%22&freshness=Month (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d7a40b080>: Failed to establish a new connection: [Errno 110] Connection timed out',))
  During handling of the above exception, another exception occurred:
  Traceback (most recent call last):
    File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app
      response = self.full_dispatch_request()
    File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request
      rv = self.handle_user_exception(e)
    File "/root/DataParser/lib/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
      return cors_after_request(app.make_response(f(*args, **kwargs)))
    File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception
      reraise(exc_type, exc_value, tb)
    File "/root/DataParser/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
      raise value
    File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request
      rv = self.dispatch_request()
    File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request
      return self.view_functions[rule.endpoint](**req.view_args)
    File "./webAppDisplay.py", line 978, in getAadNews
      papernews = json.loads(requests.request("GET", url, headers=headers, params=querystring).text)
    File "/root/DataParser/lib/python3.5/site-packages/requests/api.py", line 56, in request
      return session.request(method=method, url=url, **kwargs)
    File "/root/DataParser/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
      resp = self.send(prep, **send_kwargs)
    File "/root/DataParser/lib/python3.5/site-packages/requests/sessions.py", line 596, in send
      r = adapter.send(request, **kwargs)
    File "/root/DataParser/lib/python3.5/site-packages/requests/adapters.py", line 487, in send
      raise ConnectionError(e, request=request)
  requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.cognitive.microsoft.com', port=443): Max retries exceeded with url: /bing/v5.0/news/search?count=10000&q=%22+mobile+payments+platform%22+and+%22papers+published%22&freshness=Month (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d7a40b080>: Failed to establish a new connection: [Errno 110] Connection timed out',))
  SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /getAcadNews?kws=mobile%20payments%20platform (ip 14.102.84.2) !!!
  uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during GET /getAcadNews?kws=mobile%20payments%20platform (14.102.84.2)
  OSError: write error

Было бы очень полезно, если бы кто-нибудь помог мне решить эту проблему. Я нахожусь на платном уровне S2. Эта проблема возникает в основном, когда эта функция вызывается 3+ раза подряд.


person Kaustabh Datta Choudhury    schedule 14.04.2017    source источник


Ответы (1)


К сожалению, ни вы, ни я ничего не можем сделать, это вина Microsoft. Хотя вам может не понравиться этот ответ, они ограничивают запросы и пропускную способность. Если у вас медленный интернет, это может быть фактором.

person James    schedule 14.04.2017