Elasticsearch replacement character test fail

[ghost]

======================================================================
ERROR: Checks that dots in field names are replaced with the replacement character if one is set.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/certtools/intelmq/intelmq/tests/bots/outputs/elasticsearch/test_output.py", line 160, in test_replacement_characters
    body=ES_SEARCH_REPLACEMENT_CHARS)['hits']['hits'][0]
IndexError: list index out of range
-------------------- >> begin captured logging << --------------------
test-bot: INFO: ElasticsearchOutputBot initialized with id test-bot and intelmq 2.2.1 and python 3.5.6 (default, Jan  9 2019, 10:45:27) as process 8251.
test-bot: INFO: Bot is starting.
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:9200
urllib3.connectionpool: DEBUG: http://127.0.0.1:9200 "HEAD /intelmq HTTP/1.1" 200 0
elasticsearch: INFO: HEAD http://127.0.0.1:9200/intelmq [status:200 request:0.006s]
elasticsearch: DEBUG: > None
elasticsearch: DEBUG: < 
test-bot: INFO: Bot initialization completed.
urllib3.connectionpool: DEBUG: http://127.0.0.1:9200 "POST /intelmq/_doc HTTP/1.1" 201 174
elasticsearch: INFO: POST http://127.0.0.1:9200/intelmq/_doc [status:201 request:0.061s]
elasticsearch: DEBUG: > {"source_asn":64496,"source_ip":"192.0.2.1","extra_foo_bar":"test","feed_name":"Example Feed","classification_type":"infected-system"}
elasticsearch: DEBUG: < {"_index":"intelmq","_type":"_doc","_id":"N5kFoHMBohz-sL_iTnEE","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1}
test-bot: INFO: Bot stopped.
urllib3.connectionpool: DEBUG: http://localhost:9200 "POST /intelmq/_search HTTP/1.1" 200 160
elasticsearch: INFO: POST http://localhost:9200/intelmq/_search [status:200 request:0.005s]
elasticsearch: DEBUG: > {"query":{"constant_score":{"filter":{"term":{"source_asn":64496}}}}}
elasticsearch: DEBUG: < {"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
--------------------- >> end captured logging << ---------------------

https://travis-ci.org/github/certtools/intelmq/jobs/713307930

It looks like this test does not fail in every build, only sometimes.

@stone-z @navtej @cert-lv

[stone-z]

Seems like the error is that there aren't any hits, not that the hit is incorrect. This makes me think the document isn't indexed yet (ES does this every 1 second by default, so it could be a race condition with the test's current sleep of 1 second).

I'm not sure how to verify this other than to see if the test still flaps. In any case, it's probably better than sleeping.

I opened #1595 for this, let me know if it still seems flaky