Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Populate the FileInfo.XS field with the file checksum #1868

Merged
merged 38 commits into from
Jul 9, 2021
Merged

Populate the FileInfo.XS field with the file checksum #1868

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
ec7dada
Added changelog for eosgrpc
Feb 11, 2021
8c7c894
Added changelog pull req
Feb 11, 2021
c37c7db
First working implementation of HTTP GET/PUT towards EOS
Feb 24, 2021
16c57f4
Add comment
Feb 24, 2021
c6b7c6a
eogrpc: change to full streaming behaviour for reads
Feb 24, 2021
63c8b00
Fix comment
Feb 24, 2021
c14a3ca
Fix comment
Feb 24, 2021
1637eb6
Implement Get/SetQuota (pending cleanup)
Mar 8, 2021
4466183
Shut up a warning from Hound
Mar 12, 2021
cecc20f
Shut up another warning from Hound
Mar 12, 2021
5301f81
eosgrpc: remove some test code, cleanup, introduce the parms ReadUses…
Mar 15, 2021
1e7d823
eosfs: Fix two minor warnings
Mar 19, 2021
3e8d68f
Fix a few warnings
Mar 29, 2021
f3db75b
If EOS complains, extract the real error. More logging discipline and…
Feb 9, 2021
55dfffd
First working implementation of HTTP GET/PUT towards EOS
Feb 24, 2021
6b4398c
Implement Get/SetQuota (pending cleanup)
Mar 8, 2021
92c76df
Code cleanup, logging, implemet Rename
Mar 12, 2021
70becb0
Shut up a warning from Hound
Mar 12, 2021
c26f1d7
Shut up another warning from Hound
Mar 12, 2021
39812ca
Use the eos grpc defs from github
Mar 12, 2021
a353488
eosgrpc: remove some test code, cleanup, introduce the parms ReadUses…
Mar 15, 2021
ef1e6f9
Allow the grpc auto connect mechanism to work also when eos is down o…
Mar 23, 2021
9e7867f
Lint cosmetics
May 11, 2021
b9c9135
Lint cosmetics (maligned)
May 11, 2021
40f281c
Pass the upload file size to the http layer if WriteUsesLocalTemp is …
May 27, 2021
f576b3b
Fix minor warning
Jun 2, 2021
be5990d
Use the embedded eos grpc proto
Jun 9, 2021
8294b0e
mod tidy
ffurano Jun 11, 2021
e85adad
make the http options local to Client
ffurano Jun 11, 2021
21da204
Keep the http transport struct on a per EOS instance base, and make a…
ffurano Jun 14, 2021
beb6c82
Don't reinitialize the http options and transport when creating a cli…
ffurano Jun 17, 2021
a181d4f
Http EOS client: more robust url parsing
ffurano Jun 17, 2021
76d5283
remove double func
ffurano Jul 1, 2021
f6f6356
remove duplicated lines
ffurano Jul 1, 2021
d57dbe6
Populate the FileInfo.XS field with the file checksum
ffurano Jul 7, 2021
1008052
http client: Escape the url path being sent to eos
ffurano Jul 7, 2021
094a603
http client: Fix changelog
ffurano Jul 7, 2021
f7e5f57
http client: Fix changelog
ffurano Jul 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog/unreleased/eosgrpc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Bugfix: Minor fixes to the grpc/http plugin: checksum, url escaping

https://github.com/cs3org/reva/pull/1868
11 changes: 10 additions & 1 deletion pkg/eosclient/eosgrpc/eos_http/eoshttp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ func (c *Client) buildFullURL(urlpath, uid, gid string) (string, error) {
return "", err
}

u, err = u.Parse(urlpath)
u, err = u.Parse(url.PathEscape(urlpath))
if err != nil {
return "", err
}
Expand Down Expand Up @@ -426,6 +426,15 @@ func (c *Client) PUTFile(ctx context.Context, httptransport *http.Transport, rem
log.Debug().Str("func", "PUTFile").Int64("Content-Length", length).Msg("setting header")
req.Header.Set("Content-Length", strconv.FormatInt(length, 10))

}
if err != nil {
log.Error().Str("func", "PUTFile").Str("url", loc.String()).Str("err", err.Error()).Msg("can't create redirected request")
return err
}
if length >= 0 {
log.Debug().Str("func", "PUTFile").Int64("Content-Length", length).Msg("setting header")
req.Header.Set("Content-Length", strconv.FormatInt(length, 10))

}

log.Debug().Str("func", "PUTFile").Str("location", loc.String()).Msg("redirection")
Expand Down
13 changes: 11 additions & 2 deletions pkg/eosclient/eosgrpc/eosgrpc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ package eosgrpc
import (
"bytes"
"context"
"encoding/hex"
"fmt"
"io"
"io/ioutil"
Expand Down Expand Up @@ -1490,6 +1491,14 @@ func (c *Client) grpcMDResponseToFileInfo(st *erpc.MDResponse, namepfx string) (

fi.Size = st.Fmd.Size

xs := &eosclient.Checksum{
XSSum: hex.EncodeToString(st.Fmd.Checksum.Value),
XSType: st.Fmd.Checksum.Type,
}
fi.XS = xs

log.Debug().Str("stat info - path", fi.File).Uint64("inode", fi.Inode).Uint64("uid", fi.UID).Uint64("gid", fi.GID).Str("etag", fi.ETag).Str("checksum", fi.XS.XSType+":"+fi.XS.XSSum).Msg("grpc response")

} else {
fi.Inode = st.Cmd.Id
fi.UID = st.Cmd.Uid
Expand All @@ -1513,9 +1522,9 @@ func (c *Client) grpcMDResponseToFileInfo(st *erpc.MDResponse, namepfx string) (
}

fi.Size = 0
}

log.Debug().Str("stat info - path", fi.File).Uint64("inode:", fi.Inode).Uint64("uid:", fi.UID).Uint64("gid:", fi.GID).Str("etag:", fi.ETag).Msg("grpc response")
log.Debug().Str("stat info - path", fi.File).Uint64("inode", fi.Inode).Uint64("uid", fi.UID).Uint64("gid", fi.GID).Str("etag", fi.ETag).Msg("grpc response")
}

return fi, nil
}
Expand Down