Question: ElasticsearchClient.Bulk with object array containing Deserialized JSON

[JamieDixon]

Hi there.

I'm trying to use the bulk API and am hitting a problem with the way dynamic entries are being handled. It looks like it's a serialization issue.

Given the following:

var bulkEntries = new List<object>();
var bulkEntry = new {index = new { _index = "ourIndex", _type = "ourType", _id = "ourId"}}
bulkEntries.Add(bulkEntry);

Here's two examples, one that works as expected and one that doesn't:

Working

bulkEntries.Add(new { id = 1 });

Not working

var doc = JsonConvert.DeserializeObject<dynamic>(document);
doc.someprop = "some value";

bulkEntries.Add(doc);

In the second example, when we call:

var bulkResponse = client.Bulk(bulkEntries);

The response looks like:

Request: {"index":{"_index":"ourIndex","_type":"ourType","_id":"ourId"}}
[[[]],[[]]]

I'm probably just doing something wrong. Any pointers would be appreciated.

Thank you.

[gmarz]

@JamieDixon I can't quite reproduce this- works fine for me. Can you share the string (document) that you are deserializing?

[JamieDixon]

@gmarz The document string is coming from a test and so it looks like "{ \"id\" : \"3\" }"

[gmarz]

Hmm, okay. Wanted to make sure it didn't contain invalid json or anything funky that would throw off serialization.

Still can't reproduce this. I'm going to add a test, maybe you can take a look at it and see if you can get it to fail/repro?

[JamieDixon]

Thanks. I'll take a look once you've added the test. Cheers for the help.

[gmarz]

^^ @JamieDixon. The expected json is in Issue1464.json.

[JamieDixon]

Your tests pass just fine. The only possible difference I can see at the moment is the version of elastic search. We're using 1.1.1 with lucene 4.7. Perhaps that makes a difference?

[JamieDixon]

Ok, I'm getting somewhere now. I've got it working but I don't really know why the broken version doesn't work. Here's the differences:

##Broken

var client = new ElasticsearchClient(new ConnectionSettings(new Uri("http://example.com"), defaultIndex: "foo"));

var bulkResponse = client.Bulk(bulkEntries);

##Working

var client = new ElasticClient(new ConnectionSettings(new Uri("http://example.com"), defaultIndex: "foo"));

var bulkResponse = client.Raw.Bulk(bulkEntries);

The working version is using ElasticClient with client.Raw.Bulk. The broken version uses ElasticsearchClient with client.Bulk.

[gmarz]

Okay, now that makes sense...this is a difference in the serializer that's being used.

Elasticsearch.Net (ElasticsearchClient) uses SimpleJson for handling serialization. This is embedded in the client and allows the library to be dependency-free.

When ElasticsearchClient is used via the Raw property on ElasticClient, Newtonsoft.Json is used as the serializer instead.

This is possibly a bug in the SimpleJson library, considering we do have dynamic support enabled. We'll need to look into this further.

However, if you're already using NEST (ElasticClient) and have a reference to it, then it's best to just access the low-level client via Raw for stuff like this.

[wbsimms]

@gmarz Thanks for this analysis. Any suggestions on how we get the IBulkResponse object from the Raw.Bulk response.

Another observation:

This fails too with the same deserialization issues.

var request = new SearchRequest() {...};
elasticClient.Serializer.Serialize(request);

[Mpdreamz]

When you call:

var doc = JsonConvert.DeserializeObject<dynamic>(document); the dynamic is actually a Json.NET JObject. If you pass that to the SimpleJson serilazer it sees it as an IEnumerable.

You need to call client.Serializer.Deserialize<dynamic>() instead to give SimpleJson a fair fight! I updated gmarz tests to showcase this.