Add new option in debug tool to get a histogram of key and value sizes. #2844
Conversation
|
|
|
Example output below (fyi, it looks justified in the terminal but not when I pasted it here): Opening DB: ./p Histogram of key sizes (in bytes) Histogram of value sizes (in bytes) |
There was a problem hiding this comment.
Reviewable status: 0 of 1 files reviewed, 8 unresolved discussions (waiting on @martinmr and @manishrjain)
dgraph/cmd/debug/run.go, line 82 at r1 (raw file):
flag.BoolVarP(&opt.keyHistory, "history", "y", false, "Show all versions of a key.") flag.StringVarP(&opt.pdir, "postings", "p", "", "Directory where posting lists are stored.") flag.BoolVar(&opt.sizeHistogram, "histogram", false, "Show a histogram of the key and value sizes.")
100 chars.
dgraph/cmd/debug/run.go, line 530 at r1 (raw file):
// Return a new instance of HistogramData with properly initialized fields. func NewHistogramData(bounds []float64) *HistogramData { histogram := new(HistogramData)
Can also be written as:
return &HistogramData{
Bounds: bounds,
Min: math.MaxInt64,
}
Slightly easier to read.
dgraph/cmd/debug/run.go, line 539 at r1 (raw file):
// Update the Min and Max fields if value is less than or greater than the // current values. func (histogram *HistogramData) UpdateMinMax(value int64) {
Also, find the right bucket, and +1 it.
dgraph/cmd/debug/run.go, line 555 at r1 (raw file):
// Simple exporter function that copies the view data to the exporter. func (exporter HistogramExporter) ExportView(viewData *view.Data) {
Remove all this.
dgraph/cmd/debug/run.go, line 611 at r1 (raw file):
func sizeHistogram(db *badger.DB) { txn := db.NewTransactionAt(math.MaxUint64, false)
Use opt.readTs instead of maxuint64.
dgraph/cmd/debug/run.go, line 628 at r1 (raw file):
keySizeMeasure := stats.Int64("debug/key_sizes", "The size of the keys in bytes", "1") keySizeView := &view.View{
Get rid of the opencensus stuff.
dgraph/cmd/debug/run.go, line 673 at r1 (raw file):
keySize := int64(len(item.Key())) valueSize := item.ValueSize() keySizeHistogram.UpdateMinMax(keySize)
.Update
dgraph/cmd/debug/run.go, line 684 at r1 (raw file):
// printing. sleepDuration, _ := time.ParseDuration("500ms") time.Sleep(sleepDuration)
time.Sleep(500*time.Millisecond), or better still time.Sleep(time.Second).
dgraph/cmd/debug/run.go
Outdated
| var ( | ||
| Debug x.SubCommand | ||
| opt flagOptions | ||
| keySizeViewName = "debug/key_size_histogram" |
There was a problem hiding this comment.
keySizeViewName is unused (from varcheck)
dgraph/cmd/debug/run.go
Outdated
| Debug x.SubCommand | ||
| opt flagOptions | ||
| keySizeViewName = "debug/key_size_histogram" | ||
| valueSizeViewName = "debug/value_size_histogram" |
There was a problem hiding this comment.
valueSizeViewName is unused (from varcheck)
There was a problem hiding this comment.
Reviewable status: 0 of 1 files reviewed, 10 unresolved discussions (waiting on @manishrjain and @golangcibot)
dgraph/cmd/debug/run.go, line 82 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
100 chars.
Done.
dgraph/cmd/debug/run.go, line 530 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Can also be written as:
return &HistogramData{
Bounds: bounds,
Min: math.MaxInt64,
}Slightly easier to read.
Done.
dgraph/cmd/debug/run.go, line 539 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Also, find the right bucket, and +1 it.
Done.
dgraph/cmd/debug/run.go, line 555 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Remove all this.
Done.
dgraph/cmd/debug/run.go, line 611 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Use opt.readTs instead of maxuint64.
Done.
dgraph/cmd/debug/run.go, line 628 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Get rid of the opencensus stuff.
Done.
dgraph/cmd/debug/run.go, line 673 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
.Update
Done.
dgraph/cmd/debug/run.go, line 684 at r1 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
time.Sleep(500*time.Millisecond), or better still time.Sleep(time.Second).
Done . Got rid of all the opencensus stuff so this is no longer needed.
dgraph/cmd/debug/run.go, line 43 at r2 (raw file):
Previously, golangcibot (Bot from GolangCI) wrote…
keySizeViewNameis unused (fromvarcheck)
Done.
dgraph/cmd/debug/run.go, line 44 at r2 (raw file):
Previously, golangcibot (Bot from GolangCI) wrote…
valueSizeViewNameis unused (fromvarcheck)
Done.
dgraph/cmd/debug/run.go
Outdated
|
|
||
| var opt flagOptions | ||
| var ( | ||
| Debug x.SubCommand |
There was a problem hiding this comment.
File is not goimports-ed (from goimports)
|
I compared the output with the previous version using opencensus to verify that they are the same. |
There was a problem hiding this comment.
Reviewable status: 0 of 1 files reviewed, 11 unresolved discussions (waiting on @manishrjain and @golangcibot)
dgraph/cmd/debug/run.go, line 41 at r3 (raw file):
Previously, golangcibot (Bot from GolangCI) wrote…
File is not
goimports-ed (fromgoimports)
This was due to a formatting issue. I formatted the file and the warning went away.
There was a problem hiding this comment.
Reviewable status: 0 of 1 files reviewed, 11 unresolved discussions (waiting on @manishrjain and @golangcibot)
dgraph/cmd/debug/run.go, line 41 at r3 (raw file):
Previously, martinmr (Martin Martinez Rivera) wrote…
This was due to a formatting issue. I formatted the file and the warning went away.
Done.
The new option can be accessed via the --histogram flag. It will print a histogram of the key and value sizes, as well as the min, max, mean and standard deviation of their distributions.
martinmr
force-pushed
the
martinmr/histogram
branch
from
248b9c0
to
7550e06
Compare
martinmr
force-pushed
the
martinmr/histogram
branch
from
7550e06
to
f9c57b4
Compare
There was a problem hiding this comment.
Reviewed 1 of 1 files at r5.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @manishrjain, @golangcibot, and @martinmr)
dgraph/cmd/debug/run.go, line 562 at r5 (raw file):
fmt.Printf("Min value: %d\n", histogram.Min) fmt.Printf("Max value: %d\n", histogram.Max) fmt.Printf("Mean: %.2f\n", histogram.Mean)
calculate the mean here.
dgraph/cmd/debug/run.go, line 617 at r5 (raw file):
item := itr.Item() keySize := int64(len(item.Key()))
keySize := ...
keyHistogram.Update(keySize)
valSize := ...
valHistogram.Update(valSize)
Define the variable as close as possible to usage.
dgraph/cmd/debug/run.go, line 620 at r5 (raw file):
valueSize := item.ValueSize() keySizeHistogram.Update(keySize) valueSizeHistogram.Update(valueSize)
valueSizeHistogra
There was a problem hiding this comment.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @manishrjain and @golangcibot)
dgraph/cmd/debug/run.go, line 562 at r5 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
calculate the mean here.
Done.
dgraph/cmd/debug/run.go, line 617 at r5 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
keySize := ...
keyHistogram.Update(keySize)valSize := ...
valHistogram.Update(valSize)Define the variable as close as possible to usage.
Done.
dgraph/cmd/debug/run.go, line 620 at r5 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
valueSizeHistogra
Done.
…s. (dgraph-io#2844) The new option can be accessed via the --histogram flag. It will print a histogram of the key and value sizes, as well as the min, max, mean and standard deviation of their distributions.
The new option can be accessed via the --histogram flag. It will print a
histogram of the key and value sizes, as well as the min, max, mean and
standard deviation of their distributions.
This change is