data and scripts for amazon-google dataset

[navinrathore]

No description provided.

[navinrathore]

Run the pyspark scripts from dirctory - examples/amazon-google/scripts
spark-submit prepareTestDataSpark.py

training file has cluster_id and label at the end right now. trying to move them in the beginning...selection column list needs to be adjusted accordingly.

[sonalgoyal]

why do we need two configs - dont we need just one for linking?

[navinrathore]

Removed.
It was added if somebody wants to generate own training/labelled data.

[navinrathore]

File removal will be part of next checkin. Version from first checkin is still there.

[sonalgoyal]

what is the need for config.json?

[sonalgoyal]

call it aws?

[navinrathore]

ok

[navinrathore]

Yes. I've made the changes.
For other datasets, these fields may be updated accordingly.

[navinrathore]

The code was updated yesterday with some changes. Some improvements were made and some mistakes were rectified.
Some analysis of the results was also done.
Result
Test1 (Google): Total 2293 records -> 771 unique ==> After Linking 565 records found matched
Test2 (AWS) : Total 2293 records -> 1090 unique ==> 2287 records found matched
Observations: in teset2 records (we did have matched records, we had other partially matched records). Perhaps updating model with some parameters, results can be improved. the score/probability was around 0.45 mostly.
Training data: both train,csv (6874 recs)and valid.csv(2293) was used to train the model.

[navinrathore]

Some queries/notes about changes:

• field 'price' is 'exact'/double and all others are 'fuzzy'/string
• In configWithTrainingSample.json, the training sample data comprises both train.csv and valid.csv data
• In configWithTrainingSample.json, is "data" attributes compulsory? it currently has both tableA and tableB. Or it could have data corresponding to train.csv and valid.csv
• What could be ideal location of the script? Some minor changes can be made it absolutely generic in terms of location
• Same script works with all dataset without any change.
• Naming is simple testA, TestB etc.