Merge pull request #37 from DevoInc/release-next

Version 1.6.3 - Fixes in documentation, and codestyle

Author: Worvast

.travis.yml

@@ -8,13 +8,6 @@ install:
 script:
 - python run_tests.py
 deploy:
-  - provider: releases
-    api_key:
-      secure: "ON5p7zLOp8YlGxvv9+saMVChEdyJnPdlY0Cvm4MuiJU0t1hVlcnREsX/QuztQM/uAs2Tug8rkWKlhwbUNt0gOz/N0aA5JqojS8jGYrB0at9rOAPoWj+MMbKlj8YfPxfWJwHAffxurwhKzklb/ni4tc2SFV7D8uUC1BUbj+agETV93MHYUAPrMNEhFmXFExTsNQw/5Bww145mrzRu1Kctz9IR7UtCj8a2KCeiFVxk3kaSuvtYBMHRfaVgkbDtIV8rMzZYBzM9OAEDcOBRqBAp9sFSvVxnG6l630Lv0VoRhmH1mvnpw/BM7BxWn4t2QVYAkMrTC//w6C6VIOMonQI9RaD6xEqu44PyedOVwtTpFTP1S71vob+u05K5uCs14zNCd3O8fVIZtCi8x/7t82UCaV/iCISy3YZqeeNioN5EsPEY0wLekmWOZG8dY3CBgukWsNeEe9MsteOvrPaJ59yqmJDw1JFbd7BU64aEeHj82/e75wf0TvIpfydwJAWFQLF9yTcvK+i/YxEwrjkpoIJ8sQLE7wrnpa5MVtOzHMomG5eVHI9tagrV3thwHwtq0tVBRr38L4BC4AHYSMtbH8g6a7jdPJMoy99jm1pfvrMRSd6dacYAF7yEn2wNYmwScwovI4vdaZkVanSvIR94YYKeAMoJfu8CRRXQVa0zz3jDz9w="
-    file: dist/devo_sdk-*.whl
-    on:
-      repo: DevoInc/python-sdk
-      branch: master
   - provider: pypi
     user: worvast
     password:

CHANGELOG.md

@@ -4,6 +4,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [1.6.3] - 2019-01-17
+#### Fixed
+ * Broken urls in documentations
+ 
+#### Changed
+ * Inconsistences in api.base class
+ * PEP8/Pylint styles in api and common packages
+
 ## [1.6.2] - 2019-01-16
 #### Added
  * Documentation about API "destination" parameter

README.md

@@ -1,7 +1,7 @@
 
 [![master Build Status](https://travis-ci.com/DevoInc/python-sdk.svg?branch=master)](https://travis-ci.com/DevoInc/python-sdk) [![LICENSE](https://img.shields.io/dub/l/vibe-d.svg)](https://github.com/DevoInc/python-sdk/blob/master/LICENSE)
 
-[![wheel](https://img.shields.io/badge/wheel-yes-brightgreen.svg)](https://pypi.org/project/devo-sdk/) [![version](https://img.shields.io/badge/version-1.6.2-blue.svg)](https://pypi.org/project/devo-sdk/) [![python](https://img.shields.io/badge/python-2.7%20%7C%203.3%20%7C%203.4%20%7C%203.5%20%7C%203.6%20%7C%203.7-blue.svg)](https://pypi.org/project/devo-sdk/)
+[![wheel](https://img.shields.io/badge/wheel-yes-brightgreen.svg)](https://pypi.org/project/devo-sdk/) [![version](https://img.shields.io/badge/version-1.6.3-blue.svg)](https://pypi.org/project/devo-sdk/) [![python](https://img.shields.io/badge/python-2.7%20%7C%203.3%20%7C%203.4%20%7C%203.5%20%7C%203.6%20%7C%203.7-blue.svg)](https://pypi.org/project/devo-sdk/)
 
 
 # Devo Python SDK
@@ -42,9 +42,16 @@ You can use sources files, clonning the project too:
 ### Documentation
 
 You has specific documentation in _[docs](docs)_ folder for each part of SDK:
-* [Api Query](docs/api.md)
 * [Sender](docs/sender.md)
 * [Common](docs/common.md)
+* API:
+    * [Api query](docs/api/api.md)
+    * [Api tasks management](docs/api/task.md)
+        * [Destination: email](docs/api/destination_email.md)
+        * [Destination: redis](docs/api/destination_redis.md)
+        * [Destination: S3](docs/api/destination_s3.md)
+
+
 
 
 ## Contributing
@@ -57,18 +64,18 @@ To send data with Devo SDK, first choose the required endpoint depending on the
  * **USA:** 	us.elb.relay.logtrust.net:443
  * **EU:**   	eu.elb.relay.logtrust.net:443
 
-You has more info in [Relays article](https://docs.devo.com/confluence/docs/system-configuration/relays).
+You have more information in the official documentation of Devo, [Sending data to Devo](https://docs.devo.com/confluence/ndt/sending-data-to-devo).
 
 ##### API
 To perform a request with API, first choose the required endpoint depending on the region your are accessing from:
  * **USA:** 	https://api-us.logtrust.com/search/query
  * **EU:**   	https://api-eu.logtrust.com/search/query
 
-You has more info in [REST API article](https://docs.devo.com/confluence/docs/api/rest-api-v2) .
+You have more information in the official documentation of Devo, [REST API v2](https://docs.devo.com/confluence/ndt/api-reference/rest-api-v2) .
 
 ## Credentials
 To obtain the access credentials necessary to use this SDK, you must have an account in [DEVO](https://www.devo.com/).<br/>
-Check the [Credentials article](https://docs.devo.com/confluence/docs/administration/administration-credentials) for more details. 
+Check the [security credentials](https://docs.devo.com/confluence/ndt/domain-administration/security-credentials) info for more details. 
 
 ##### Certificates
 You need use a three files (Cert, key and chain) to secure send data to Devo. 

devo/__version__.py

@@ -1,6 +1,6 @@
 __description__ = 'Devo Python Library.'
 __url__ = 'http://www.devo.com'
-__version__ = "1.6.2"
+__version__ = "1.6.3"
 __author__ = 'Devo'
 __author_email__ = 'support@devo.com'
 __license__ = 'MIT'

devo/api/base.py

@@ -72,6 +72,7 @@ def __init__(self, *args, **kwargs):
         self.retries = 3
         self.timeout = 30
         self.sleep = 5
+        self.buffer = None
 
     def __set_url_query(self):
         """
@@ -92,30 +93,15 @@ def __get_url_parts(self):
             self.url.split("//")[-1].split("/", maxsplit=1) if PY3
             else self.url.split("//")[-1].split("/", 1))
 
-    def __verify_url_complement(self, url_list):
+    @staticmethod
+    def __verify_url_complement(url_list):
         """
         Verify if only has main domain or full url
         :param url_list: One or two part of the url
         """
         return url_list if len(url_list) == 2 \
             else [url_list[0], URL_QUERY_COMPLEMENT]
 
-    def _generate_pragmas(self, comment=None):
-        """
-        Generate pragmas to add to query
-        :comment: Pragma comment free
-        :user: Pragma comment user
-        :app_name: Pragma comment id. App name.
-        """
-        str_pragmas = ' pragma comment.id:"{}" ' \
-                      'pragma comment.user:"{}"'\
-            .format(self.app_name, self.user)
-
-        if comment:
-            return str_pragmas + ' pragma comment.free:"{}"'.format(comment)
-
-        return str_pragmas
-
     @staticmethod
     def _generate_dates(dates):
         """
@@ -186,10 +172,3 @@ def close(self):
         if self.socket is not None:
             self.socket.close()
             self.socket = None
-
-    def _get_job_url_part(self):
-        URL_JOB = '/search/job'
-        URL_JOBS = '/search/jobs'
-        URL_JOB_START = '/start/'
-        URL_JOB_STOP = '/start/'
-        URL_JOB_REMOVE = '/start/'

devo/api/client.py

@@ -4,8 +4,8 @@
 import hashlib
 import time
 import json
-import requests
 from socket import timeout as socket_timeout
+import requests
 from devo.common import Buffer, default_from, default_to
 from .base import Base, DevoClientException, CLIENT_DEFAULT_USER, \
     CLIENT_DEFAULT_APP_NAME, URL_JOB, URL_JOBS, URL_JOB_START, URL_JOB_STOP, \
@@ -151,7 +151,7 @@ def _call_stream(self, payload=None):
                     while not self.buffer.is_empty() or self.buffer.close:
                         time.sleep(1)
             else:
-                raise DevoClientException("Devo-Client|%s" % str(data))
+                self.buffer.close = True
 
     @staticmethod
     def _get_payload(query, query_id, dates, opts):
@@ -275,6 +275,22 @@ def _get_sign(self, data, tstamp):
                         hashlib.sha256)
         return sign.hexdigest()
 
+    def _generate_pragmas(self, comment=None):
+        """
+        Generate pragmas to add to query
+        :comment: Pragma comment free
+        :user: Pragma comment user
+        :app_name: Pragma comment id. App name.
+        """
+        str_pragmas = ' pragma comment.id:"{}" ' \
+                      'pragma comment.user:"{}"'\
+            .format(self.app_name, self.user)
+
+        if comment:
+            return str_pragmas + ' pragma comment.free:"{}"'.format(comment)
+
+        return str_pragmas
+
     def _call_jobs(self, url):
         """
         Make the call
@@ -349,4 +365,3 @@ def remove_job(self, job_id):
         :return: bool"""
         return self._call_jobs("{}{}{}{}".format(self.url, URL_JOB,
                                                  URL_JOB_REMOVE, job_id))
-

devo/common/data/buffer.py

@@ -12,7 +12,6 @@
 
 class DevoBufferException(Exception):
     """ Default Devo Client Exception """
-    pass
 
 
 def clrf():
@@ -23,17 +22,16 @@ def clrf():
 def get_response_split(api_response):
     """Get split for each event, if csv or other, and in bytes or str"""
     if version_info[0] > 2:
-        return b"\n" if api_response is "csv" else b"\r\n"
-    else:
-        return "\n" if api_response is "csv" else "\r\n"
+        return b"\n" if api_response == "csv" else b"\r\n"
+    return "\n" if api_response == "csv" else "\r\n"
 
 
 def empty():
     """Empty string in bytes or str if py2 or py3"""
     return bytes() if version_info[0] > 2 else str()
 
 
-class Buffer(object):
+class Buffer:
     """ Simple Buffer class """
     def __init__(self, buffer_max_size=1000, api_response="json/compact"):
         self.queue = Queue.Queue(maxsize=buffer_max_size)
@@ -49,13 +47,16 @@ def __init__(self, buffer_max_size=1000, api_response="json/compact"):
         self.api_response = api_response
 
     def is_alive(self):
+        """Revise is buffer is alive"""
         return self.thread.isAlive()
 
     def set_timeout(self, timeout):
+        """Set timeout for socket call"""
         self.timeout = timeout
 
     def is_empty(self):
-        return False if self.size() > 0 else True
+        """Revise is buffer queue is empty"""
+        return True if self.size() else False
 
     def create_thread(self, target, kwargs):
         """ Function for create one separate thread for Queue"""
@@ -85,17 +86,18 @@ def process_first_line(self, data):
         """ process first line of the Query call (For delete headers) """
 
         if version_info[0] > 2:
-            ok = b"200 OK"
+            response_ok = b"200 OK"
         else:
-            ok = "200 OK"
+            response_ok = "200 OK"
 
-        if ok in data.split(self.clrf+self.clrf)[0]:
+        if response_ok in data.split(self.clrf+self.clrf)[0]:
             return self.buffering(data[data.find(self.clrf+self.clrf)+4:]), None
 
         self.error = data
         return False, data
 
     def buffering(self, data):
+        """Saving temp date from system IO, waiting for all octet response"""
         if not self.octet:
             pointer = data.find(self.clrf) + 2
             size = int(data[:pointer], 16)
@@ -122,8 +124,8 @@ def process_recv(self, data):
         if data[-(len(self.response_split)):] != self.response_split:
             self.temp_event = data_list.pop()
 
-        for aux in range(0, len(data_list)):
-            self.queue.put(data_list[aux].strip(), block=True)
+        for item in data_list:
+            self.queue.put(item.strip(), block=True)
 
         self.octet = 0
         if len(self.temp.strip()):

devo/common/data/chain_dict.py

@@ -14,7 +14,6 @@ def set_key(self, key_list, value):
         if not isinstance(key_list, list):
             key_list = [key_list]
         self.set_key_chain(self, key_list, value)
-        return
 
     @staticmethod
     def set_key_chain(cdict, key_list, value):
@@ -30,6 +29,6 @@ def set_key_chain(cdict, key_list, value):
             cdict[key_list[0]] = value
         else:
             cdict[key_list[0]] = ChainDict.set_key_chain(dict(),
-                                                           key_list[1:],
-                                                           value)
+                                                         key_list[1:],
+                                                         value)
         return cdict

devo/common/dates/dateoperations.py

@@ -1,14 +1,9 @@
 # -*- coding: utf-8 -*-
-"""Utils for date operations."""
+"""A collection of allowed operations on date parsing"""
 from datetime import datetime as dt, timedelta
 from .dateutils import to_millis, trunc_time, trunc_time_minute
 
 
-"""
-This class is a collection of allowed operations on date parsing
-"""
-
-
 def month():
     """
     Return millis for a month
@@ -90,6 +85,7 @@ def yesterday():
 
 
 def parse_functions():
+    """Return object with all functions availables"""
     return {
         "month": month,
         "week": week,
@@ -101,4 +97,4 @@ def parse_functions():
         "now_without_ms": now_without_ms,
         "today": today,
         "yesterday": yesterday
-    }
+    }

devo/common/dates/dateparser.py

@@ -32,7 +32,7 @@ def parse_string(date_string):
     if test_date_format(date_string, '%Y-%m-%d %H:%M:%S'):
         date = dt.strptime(date_string, '%Y-%m-%d %H:%M:%S')
         return to_millis(date)
-    elif test_date_format(date_string, '%Y-%m-%d'):
+    if test_date_format(date_string, '%Y-%m-%d'):
         date = dt.strptime(date_string, '%Y-%m-%d')
         return to_millis(trunc_time(date))
     return parse_expression(date_string)