Content-Type: multipart/alternative;
 boundary="------------haF1UhFbpCBPIW0a10ZSoQUx"
Message-ID: <25b24513-cf6f-406e-a04e-b3d834ad5398@pgug.de>
Date: Thu, 7 Dec 2023 17:49:46 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: cannot download mbox with python
To: Pierre Forstmann <pierre.forstmann@gmail.com>, daniel@yesql.se
Cc: pgsql-www@postgresql.org
References: 
 <CAM-sOH8qZxmza4xm29jbNNpfXYDu7eUUZSXHgJ9dCoUbHK86OQ@mail.gmail.com>
 <EB2B3047-FAB1-4F5A-A535-6C3A82DFD945@yesql.se>
 <CAM-sOH_TOtSDVHbC76hyr3akFt2P6o8myFcFdOiGpq6GCDc9JA@mail.gmail.com>
Content-Language: en-GB
From: Andreas 'ads' Scherbaum <ads@pgug.de>
In-Reply-To: 
 <CAM-sOH_TOtSDVHbC76hyr3akFt2P6o8myFcFdOiGpq6GCDc9JA@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/25b24513-cf6f-406e-a04e-b3d834ad5398%40pgug.de>
Precedence: bulk

This is a multi-part message in MIME format.
--------------haF1UhFbpCBPIW0a10ZSoQUx
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 07/12/2023 17:39, Pierre Forstmann wrote:
> I've tried this:
>
> import requests
> from urllib.parse import urlparse
>
> url = 'https://www.postgresql.org/list/pgsql-bugs/mbox/pgsql-bugs.202312'
> #response = requests.get(url, auth=('xxx','yyy'))
> session = requests.session()
> session.auth = ('xxx','yyy')
> response = session.get(url)
> print('status: ' + str(response.status_code))
> print('... done')
> print(response.content)

The session.auth is still doing a basic http auth, not what you need here.

Try opening your link in a browser in an anonymous window:

https://www.postgresql.org/list/pgsql-bugs/mbox/pgsql-bugs.202312

It redirects you to the login, you need to emulate that path in your script,
login into the website and then you can retrieve the mbox.

-- 
				Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
Volunteer Regional Contact, Germany - PostgreSQL Project

--------------haF1UhFbpCBPIW0a10ZSoQUx
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 07/12/2023 17:39, Pierre Forstmann
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAM-sOH_TOtSDVHbC76hyr3akFt2P6o8myFcFdOiGpq6GCDc9JA@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">
            <div dir="ltr">
              <div>I've tried this:</div>
              <div><br>
              </div>
              <div><span style="font-family:monospace">import requests<br>
                  from urllib.parse import urlparse<br>
                  <br>
                  url = '<a
href="https://www.postgresql.org/list/pgsql-bugs/mbox/pgsql-bugs.202312"
                    moz-do-not-send="true" class="moz-txt-link-freetext">https://www.postgresql.org/list/pgsql-bugs/mbox/pgsql-bugs.202312</a>'<br>
                  #response = requests.get(url, auth=('xxx','yyy'))<br>
                  session = requests.session()<br>
                  session.auth = ('xxx','yyy')<br>
                  response = session.get(url)<br>
                  print('status: ' + str(response.status_code))<br>
                  print('... done')<br>
                  print(response.content)</span></div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    The session.auth is still doing a basic http auth, not what you need
    here.<br>
    <br>
    Try opening your link in a browser in an anonymous window:<br>
    <br>
    <a class="moz-txt-link-freetext" href="https://www.postgresql.org/list/pgsql-bugs/mbox/pgsql-bugs.202312">https://www.postgresql.org/list/pgsql-bugs/mbox/pgsql-bugs.202312</a><br>
    <br>
    It redirects you to the login, you need to emulate that path in your
    script,<br>
    login into the website and then you can retrieve the mbox.<br>
    <br>
    <pre class="moz-signature" cols="72">-- 
				Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
Volunteer Regional Contact, Germany - PostgreSQL Project
</pre>
  </body>
</html>

--------------haF1UhFbpCBPIW0a10ZSoQUx--