Public buckets?

[JoshCu]

Is it possible to connect to public buckets? e.g. s3 = s3fs.S3FileSystem(anon=True)

[michalc]

Hi @JoshCu,

I don’t see why not, as long as they support versioning. Have you tried and it doesn’t work? Can you post details if so, enough so I can run the code myself and see any errors?

Michal

[JoshCu]

I was getting key errors because I didn't have the AWS env vars set. I got past that by passing a lambda returning ("","","",None) but eventually had to just comment out all of the header signing.

    @contextmanager
    def make_auth_request(http_client, method, params, headers):
        #now = datetime.utcnow()
        #region, access_key_id, secret_access_key, session_token = get_credentials(now)
        #to_auth_headers = headers + (
        #    (('x-amz-security-token', session_token),) if session_token is not None else \
        #    ()
        #)
        #request_headers = aws_sigv4_headers(
        #    now, access_key_id, secret_access_key, region, method, to_auth_headers, params,
        #)
        url = f'{scheme}://{netloc}{path}'
        print(method)
        print(url)
        print(headers)
        with http_client.stream(method, url, params=params, headers=headers) as response:
            response.raise_for_status()
            yield response

Which gave me permission denied because I needed to set public allow on s3:GetObjectVersion on my bucket.
Now it's working, but it's reading a huge amount of data. The db has an index, but my file might just impractically large?

[michalc]

I got past that by passing a lambda returning ("","","",None)

Ah yes I see - I assumed that AWS would allow this for public buckets, but I was wrong. Addressing this in #95

Now it's working, but it's reading a huge amount of data. The db has an index, but my file might just impractically large?

It all depends on the query - is it definitely using the index? Does EXPLAIN QUERY PLAN shed any light? What query are you running?

[JoshCu]

I just tested out #95 and it works great! Thanks for adding it so quickly :)
And thanks for pointing me back to the index, embarrassingly, there isn't one.
The column I was searching by shares a name with another table that is indexed. Modifying the query to select by the correctly indexed id worked as expected and didn't try to scan the whole table.
Thanks for your help!

[michalc]

Thanks for adding it so quickly :)

No problem! Now released in v0.0.82