backend doesn't support absolute paths: Support for non local filesystem

[smereddy]

When using data-wizard with S3 as default storage I get the following error.

Django 1.11
Python 2.7

/Users/AEM/venv/lib/python2.7/site-packages/storages/backends/s3boto3.py:282: UserWarning: The default behavior of S3Boto3Storage is insecure and will change in django-storages 2.0. By default files and new buckets are saved with an ACL of 'public-read' (globally publicly readable). Version 2.0 will default to using the bucket's ACL. To opt into the new behavior set AWS_DEFAULT_ACL = None, otherwise to silence this warning explicitly set AWS_DEFAULT_ACL.
  "The default behavior of S3Boto3Storage is insecure and will change "
ERROR:root:This backend doesn't support absolute paths.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/backends/base.py", line 39, in try_run_sync
    result = self.run_sync(task_name, run_id, user_id, post)
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/backends/base.py", line 30, in run_sync
    result = fn(run_id, user_id)
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/tasks.py", line 72, in wrapped
    return fn(run, user, **kwargs)
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/tasks.py", line 282, in read_columns
    matched = get_columns(run)
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/tasks.py", line 303, in get_columns
    return parse_columns(run)
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/tasks.py", line 387, in parse_columns
    table = run.load_io()
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/models.py", line 53, in load_io
    self._io_data = loader.load_io()
  File "/Users/AEM/venv/lib/python2.7/site-packages/data_wizard/loaders.py", line 39, in load_io
    return load_file(self.file.path, options=options)
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/db/models/fields/files.py", line 65, in path
    return self.storage.path(self.name)
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/core/files/storage.py", line 111, in path
    raise NotImplementedError("This backend doesn't support absolute paths.")
NotImplementedError: This backend doesn't support absolute paths.
DEBUG:django.template:Exception while resolving variable 'is_popup' in template 'data_wizard/run_columns.html'.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
    (bit, current))  # missing attribute
VariableDoesNotExist: Failed lookup for key [is_popup] in u'[{\'False\': False, \'None\': None, \'True\': True}, {u\'csrf_token\': <SimpleLazyObject: <function _get_val at 0x110e71140>>, \'perms\': <django.contrib.auth.context_processors.PermWrapper object at 0x110d6e950>, \'messages\': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x110d54f50>, u\'request\': <rest_framework.request.Request object at 0x110d53390>, u\'TIME_ZONE\': \'UTC\', u\'STATIC_URL\': \'/static/\', u\'LANGUAGES\': [(u\'af\', u\'Afrikaans\'), (u\'ar\', u\'Arabic\'), (u\'ast\', u\'Asturian\'), (u\'az\', u\'Azerbaijani\'), (u\'bg\', u\'Bulgarian\'), (u\'be\', u\'Belarusian\'), (u\'bn\', u\'Bengali\'), (u\'br\', u\'Breton\'), (u\'bs\', u\'Bosnian\'), (u\'ca\', u\'Catalan\'), (u\'cs\', u\'Czech\'), (u\'cy\', u\'Welsh\'), (u\'da\', u\'Danish\'), (u\'de\', u\'German\'), (u\'dsb\', u\'Lower Sorbian\'), (u\'el\', u\'Greek\'), (u\'en\', u\'English\'), (u\'en-au\', u\'Australian English\'), (u\'en-gb\', u\'British English\'), (u\'eo\', u\'Esperanto\'), (u\'es\', u\'Spanish\'), (u\'es-ar\', u\'Argentinian Spanish\'), (u\'es-co\', u\'Colombian Spanish\'), (u\'es-mx\', u\'Mexican Spanish\'), (u\'es-ni\', u\'Nicaraguan Spanish\'), (u\'es-ve\', u\'Venezuelan Spanish\'), (u\'et\', u\'Estonian\'), (u\'eu\', u\'Basque\'), (u\'fa\', u\'Persian\'), (u\'fi\', u\'Finnish\'), (u\'fr\', u\'French\'), (u\'fy\', u\'Frisian\'), (u\'ga\', u\'Irish\'), (u\'gd\', u\'Scottish Gaelic\'), (u\'gl\', u\'Galician\'), (u\'he\', u\'Hebrew\'), (u\'hi\', u\'Hindi\'), (u\'hr\', u\'Croatian\'), (u\'hsb\', u\'Upper Sorbian\'), (u\'hu\', u\'Hungarian\'), (u\'ia\', u\'Interlingua\'), (u\'id\', u\'Indonesian\'), (u\'io\', u\'Ido\'), (u\'is\', u\'Icelandic\'), (u\'it\', u\'Italian\'), (u\'ja\', u\'Japanese\'), (u\'ka\', u\'Georgian\'), (u\'kk\', u\'Kazakh\'), (u\'km\', u\'Khmer\'), (u\'kn\', u\'Kannada\'), (u\'ko\', u\'Korean\'), (u\'lb\', u\'Luxembourgish\'), (u\'lt\', u\'Lithuanian\'), (u\'lv\', u\'Latvian\'), (u\'mk\', u\'Macedonian\'), (u\'ml\', u\'Malayalam\'), (u\'mn\', u\'Mongolian\'), (u\'mr\', u\'Marathi\'), (u\'my\', u\'Burmese\'), (u\'nb\', u\'Norwegian Bokm\\xe5l\'), (u\'ne\', u\'Nepali\'), (u\'nl\', u\'Dutch\'), (u\'nn\', u\'Norwegian Nynorsk\'), (u\'os\', u\'Ossetic\'), (u\'pa\', u\'Punjabi\'), (u\'pl\', u\'Polish\'), (u\'pt\', u\'Portuguese\'), (u\'pt-br\', u\'Brazilian Portuguese\'), (u\'ro\', u\'Romanian\'), (u\'ru\', u\'Russian\'), (u\'sk\', u\'Slovak\'), (u\'sl\', u\'Slovenian\'), (u\'sq\', u\'Albanian\'), (u\'sr\', u\'Serbian\'), (u\'sr-latn\', u\'Serbian Latin\'), (u\'sv\', u\'Swedish\'), (u\'sw\', u\'Swahili\'), (u\'ta\', u\'Tamil\'), (u\'te\', u\'Telugu\'), (u\'th\', u\'Thai\'), (u\'tr\', u\'Turkish\'), (u\'tt\', u\'Tatar\'), (u\'udm\', u\'Udmurt\'), (u\'uk\', u\'Ukrainian\'), (u\'ur\', u\'Urdu\'), (u\'vi\', u\'Vietnamese\'), (u\'zh-hans\', u\'Simplified Chinese\'), (u\'zh-hant\', u\'Traditional Chinese\')], \'user\': <SimpleLazyObject: <User: sreddy>>, u\'agent\': <django_agent_trust.models.Agent object at 0x110d54a10>, u\'LANGUAGE_CODE\': \'en-us\', \'DEFAULT_MESSAGE_LEVELS\': {\'DEBUG\': 10, \'INFO\': 20, \'WARNING\': 30, \'SUCCESS\': 25, \'ERROR\': 40}, \'srns\': 4254, u\'LANGUAGE_BIDI\': False, u\'MEDIA_URL\': \'https://local-aem-test.s3.amazonaws.com/\'}, {}, {\'record_count\': None, \'content_type_id\': u\'sources.filesource\', u\'LANGUAGE_CODE\': \'en-us\', u\'LANGUAGE_BIDI\': False, \'serializer_label\': u\'Customer\', \'label\': u\'Run for customer\', \'object_id\': 2, \'last_update\': datetime.datetime(2020, 6, 17, 23, 53, 49, 397460, tzinfo=<UTC>), \'object_label\': u\'customer\', \'error\': "NotImplementedError: This backend doesn\'t support absolute paths.", \'loader\': u\'data_wizard.loaders.FileLoader\', u\'id\': 2, \'serializer\': \'data_wizard.registry.CustomerSerializer\'}]'
DEBUG:django.template:Exception while resolving variable 'is_popup' in template 'data_wizard/run_columns.html'.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
    (bit, current))  # missing attribute
VariableDoesNotExist: Failed lookup for key [is_popup] in u'[{\'False\': False, \'None\': None, \'True\': True}, {u\'csrf_token\': <SimpleLazyObject: <function _get_val at 0x110e71140>>, \'perms\': <django.contrib.auth.context_processors.PermWrapper object at 0x110d6e950>, \'messages\': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x110d54f50>, u\'request\': <rest_framework.request.Request object at 0x110d53390>, u\'TIME_ZONE\': \'UTC\', u\'STATIC_URL\': \'/static/\', u\'LANGUAGES\': [(u\'af\', u\'Afrikaans\'), (u\'ar\', u\'Arabic\'), (u\'ast\', u\'Asturian\'), (u\'az\', u\'Azerbaijani\'), (u\'bg\', u\'Bulgarian\'), (u\'be\', u\'Belarusian\'), (u\'bn\', u\'Bengali\'), (u\'br\', u\'Breton\'), (u\'bs\', u\'Bosnian\'), (u\'ca\', u\'Catalan\'), (u\'cs\', u\'Czech\'), (u\'cy\', u\'Welsh\'), (u\'da\', u\'Danish\'), (u\'de\', u\'German\'), (u\'dsb\', u\'Lower Sorbian\'), (u\'el\', u\'Greek\'), (u\'en\', u\'English\'), (u\'en-au\', u\'Australian English\'), (u\'en-gb\', u\'British English\'), (u\'eo\', u\'Esperanto\'), (u\'es\', u\'Spanish\'), (u\'es-ar\', u\'Argentinian Spanish\'), (u\'es-co\', u\'Colombian Spanish\'), (u\'es-mx\', u\'Mexican Spanish\'), (u\'es-ni\', u\'Nicaraguan Spanish\'), (u\'es-ve\', u\'Venezuelan Spanish\'), (u\'et\', u\'Estonian\'), (u\'eu\', u\'Basque\'), (u\'fa\', u\'Persian\'), (u\'fi\', u\'Finnish\'), (u\'fr\', u\'French\'), (u\'fy\', u\'Frisian\'), (u\'ga\', u\'Irish\'), (u\'gd\', u\'Scottish Gaelic\'), (u\'gl\', u\'Galician\'), (u\'he\', u\'Hebrew\'), (u\'hi\', u\'Hindi\'), (u\'hr\', u\'Croatian\'), (u\'hsb\', u\'Upper Sorbian\'), (u\'hu\', u\'Hungarian\'), (u\'ia\', u\'Interlingua\'), (u\'id\', u\'Indonesian\'), (u\'io\', u\'Ido\'), (u\'is\', u\'Icelandic\'), (u\'it\', u\'Italian\'), (u\'ja\', u\'Japanese\'), (u\'ka\', u\'Georgian\'), (u\'kk\', u\'Kazakh\'), (u\'km\', u\'Khmer\'), (u\'kn\', u\'Kannada\'), (u\'ko\', u\'Korean\'), (u\'lb\', u\'Luxembourgish\'), (u\'lt\', u\'Lithuanian\'), (u\'lv\', u\'Latvian\'), (u\'mk\', u\'Macedonian\'), (u\'ml\', u\'Malayalam\'), (u\'mn\', u\'Mongolian\'), (u\'mr\', u\'Marathi\'), (u\'my\', u\'Burmese\'), (u\'nb\', u\'Norwegian Bokm\\xe5l\'), (u\'ne\', u\'Nepali\'), (u\'nl\', u\'Dutch\'), (u\'nn\', u\'Norwegian Nynorsk\'), (u\'os\', u\'Ossetic\'), (u\'pa\', u\'Punjabi\'), (u\'pl\', u\'Polish\'), (u\'pt\', u\'Portuguese\'), (u\'pt-br\', u\'Brazilian Portuguese\'), (u\'ro\', u\'Romanian\'), (u\'ru\', u\'Russian\'), (u\'sk\', u\'Slovak\'), (u\'sl\', u\'Slovenian\'), (u\'sq\', u\'Albanian\'), (u\'sr\', u\'Serbian\'), (u\'sr-latn\', u\'Serbian Latin\'), (u\'sv\', u\'Swedish\'), (u\'sw\', u\'Swahili\'), (u\'ta\', u\'Tamil\'), (u\'te\', u\'Telugu\'), (u\'th\', u\'Thai\'), (u\'tr\', u\'Turkish\'), (u\'tt\', u\'Tatar\'), (u\'udm\', u\'Udmurt\'), (u\'uk\', u\'Ukrainian\'), (u\'ur\', u\'Urdu\'), (u\'vi\', u\'Vietnamese\'), (u\'zh-hans\', u\'Simplified Chinese\'), (u\'zh-hant\', u\'Traditional Chinese\')], \'user\': <SimpleLazyObject: <User: sreddy>>, u\'agent\': <django_agent_trust.models.Agent object at 0x110d54a10>, u\'LANGUAGE_CODE\': \'en-us\', \'DEFAULT_MESSAGE_LEVELS\': {\'DEBUG\': 10, \'INFO\': 20, \'WARNING\': 30, \'SUCCESS\': 25, \'ERROR\': 40}, \'srns\': 4254, u\'LANGUAGE_BIDI\': False, u\'MEDIA_URL\': \'https://local-aem-test.s3.amazonaws.com/\'}, {}, {\'record_count\': None, \'content_type_id\': u\'sources.filesource\', u\'LANGUAGE_CODE\': \'en-us\', u\'LANGUAGE_BIDI\': False, \'serializer_label\': u\'Customer\', \'label\': u\'Run for customer\', \'object_id\': 2, \'last_update\': datetime.datetime(2020, 6, 17, 23, 53, 49, 397460, tzinfo=<UTC>), \'object_label\': u\'customer\', \'error\': "NotImplementedError: This backend doesn\'t support absolute paths.", \'loader\': u\'data_wizard.loaders.FileLoader\', u\'id\': 2, \'serializer\': \'data_wizard.registry.CustomerSerializer\'}]'
DEBUG:django.template:Exception while resolving variable 'has_permission' in template 'data_wizard/run_columns.html'.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
    (bit, current))  # missing attribute
VariableDoesNotExist: Failed lookup for key [has_permission] in u'[{\'False\': False, \'None\': None, \'True\': True}, {u\'csrf_token\': <SimpleLazyObject: <function _get_val at 0x110e71140>>, \'perms\': <django.contrib.auth.context_processors.PermWrapper object at 0x110d6e950>, \'messages\': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x110d54f50>, u\'request\': <rest_framework.request.Request object at 0x110d53390>, u\'TIME_ZONE\': \'UTC\', u\'STATIC_URL\': \'/static/\', u\'LANGUAGES\': [(u\'af\', u\'Afrikaans\'), (u\'ar\', u\'Arabic\'), (u\'ast\', u\'Asturian\'), (u\'az\', u\'Azerbaijani\'), (u\'bg\', u\'Bulgarian\'), (u\'be\', u\'Belarusian\'), (u\'bn\', u\'Bengali\'), (u\'br\', u\'Breton\'), (u\'bs\', u\'Bosnian\'), (u\'ca\', u\'Catalan\'), (u\'cs\', u\'Czech\'), (u\'cy\', u\'Welsh\'), (u\'da\', u\'Danish\'), (u\'de\', u\'German\'), (u\'dsb\', u\'Lower Sorbian\'), (u\'el\', u\'Greek\'), (u\'en\', u\'English\'), (u\'en-au\', u\'Australian English\'), (u\'en-gb\', u\'British English\'), (u\'eo\', u\'Esperanto\'), (u\'es\', u\'Spanish\'), (u\'es-ar\', u\'Argentinian Spanish\'), (u\'es-co\', u\'Colombian Spanish\'), (u\'es-mx\', u\'Mexican Spanish\'), (u\'es-ni\', u\'Nicaraguan Spanish\'), (u\'es-ve\', u\'Venezuelan Spanish\'), (u\'et\', u\'Estonian\'), (u\'eu\', u\'Basque\'), (u\'fa\', u\'Persian\'), (u\'fi\', u\'Finnish\'), (u\'fr\', u\'French\'), (u\'fy\', u\'Frisian\'), (u\'ga\', u\'Irish\'), (u\'gd\', u\'Scottish Gaelic\'), (u\'gl\', u\'Galician\'), (u\'he\', u\'Hebrew\'), (u\'hi\', u\'Hindi\'), (u\'hr\', u\'Croatian\'), (u\'hsb\', u\'Upper Sorbian\'), (u\'hu\', u\'Hungarian\'), (u\'ia\', u\'Interlingua\'), (u\'id\', u\'Indonesian\'), (u\'io\', u\'Ido\'), (u\'is\', u\'Icelandic\'), (u\'it\', u\'Italian\'), (u\'ja\', u\'Japanese\'), (u\'ka\', u\'Georgian\'), (u\'kk\', u\'Kazakh\'), (u\'km\', u\'Khmer\'), (u\'kn\', u\'Kannada\'), (u\'ko\', u\'Korean\'), (u\'lb\', u\'Luxembourgish\'), (u\'lt\', u\'Lithuanian\'), (u\'lv\', u\'Latvian\'), (u\'mk\', u\'Macedonian\'), (u\'ml\', u\'Malayalam\'), (u\'mn\', u\'Mongolian\'), (u\'mr\', u\'Marathi\'), (u\'my\', u\'Burmese\'), (u\'nb\', u\'Norwegian Bokm\\xe5l\'), (u\'ne\', u\'Nepali\'), (u\'nl\', u\'Dutch\'), (u\'nn\', u\'Norwegian Nynorsk\'), (u\'os\', u\'Ossetic\'), (u\'pa\', u\'Punjabi\'), (u\'pl\', u\'Polish\'), (u\'pt\', u\'Portuguese\'), (u\'pt-br\', u\'Brazilian Portuguese\'), (u\'ro\', u\'Romanian\'), (u\'ru\', u\'Russian\'), (u\'sk\', u\'Slovak\'), (u\'sl\', u\'Slovenian\'), (u\'sq\', u\'Albanian\'), (u\'sr\', u\'Serbian\'), (u\'sr-latn\', u\'Serbian Latin\'), (u\'sv\', u\'Swedish\'), (u\'sw\', u\'Swahili\'), (u\'ta\', u\'Tamil\'), (u\'te\', u\'Telugu\'), (u\'th\', u\'Thai\'), (u\'tr\', u\'Turkish\'), (u\'tt\', u\'Tatar\'), (u\'udm\', u\'Udmurt\'), (u\'uk\', u\'Ukrainian\'), (u\'ur\', u\'Urdu\'), (u\'vi\', u\'Vietnamese\'), (u\'zh-hans\', u\'Simplified Chinese\'), (u\'zh-hant\', u\'Traditional Chinese\')], \'user\': <SimpleLazyObject: <User: sreddy>>, u\'agent\': <django_agent_trust.models.Agent object at 0x110d54a10>, u\'LANGUAGE_CODE\': \'en-us\', \'DEFAULT_MESSAGE_LEVELS\': {\'DEBUG\': 10, \'INFO\': 20, \'WARNING\': 30, \'SUCCESS\': 25, \'ERROR\': 40}, \'srns\': 4254, u\'LANGUAGE_BIDI\': False, u\'MEDIA_URL\': \'https://local-aem-test.s3.amazonaws.com/\'}, {}, {\'record_count\': None, \'content_type_id\': u\'sources.filesource\', u\'LANGUAGE_CODE\': \'en-us\', u\'LANGUAGE_BIDI\': False, \'serializer_label\': u\'Customer\', \'label\': u\'Run for customer\', \'object_id\': 2, \'last_update\': datetime.datetime(2020, 6, 17, 23, 53, 49, 397460, tzinfo=<UTC>), \'object_label\': u\'customer\', \'error\': "NotImplementedError: This backend doesn\'t support absolute paths.", \'loader\': u\'data_wizard.loaders.FileLoader\', u\'id\': 2, \'serializer\': \'data_wizard.registry.CustomerSerializer\'}, {\'block\': <Block Node: usertools. Contents: [<TextNode: u\'\\n        \'>, <IfNode>, <TextNode: u\'\\n        \'>]>}]'
DEBUG:django.template:Exception while resolving variable 'result' in template 'data_wizard/run_columns.html'.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
    (bit, current))  # missing attribute
VariableDoesNotExist: Failed lookup for key [result] in u'[{\'False\': False, \'None\': None, \'True\': True}, {u\'csrf_token\': <SimpleLazyObject: <function _get_val at 0x110e71140>>, \'perms\': <django.contrib.auth.context_processors.PermWrapper object at 0x110d6e950>, \'messages\': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x110d54f50>, u\'request\': <rest_framework.request.Request object at 0x110d53390>, u\'TIME_ZONE\': \'UTC\', u\'STATIC_URL\': \'/static/\', u\'LANGUAGES\': [(u\'af\', u\'Afrikaans\'), (u\'ar\', u\'Arabic\'), (u\'ast\', u\'Asturian\'), (u\'az\', u\'Azerbaijani\'), (u\'bg\', u\'Bulgarian\'), (u\'be\', u\'Belarusian\'), (u\'bn\', u\'Bengali\'), (u\'br\', u\'Breton\'), (u\'bs\', u\'Bosnian\'), (u\'ca\', u\'Catalan\'), (u\'cs\', u\'Czech\'), (u\'cy\', u\'Welsh\'), (u\'da\', u\'Danish\'), (u\'de\', u\'German\'), (u\'dsb\', u\'Lower Sorbian\'), (u\'el\', u\'Greek\'), (u\'en\', u\'English\'), (u\'en-au\', u\'Australian English\'), (u\'en-gb\', u\'British English\'), (u\'eo\', u\'Esperanto\'), (u\'es\', u\'Spanish\'), (u\'es-ar\', u\'Argentinian Spanish\'), (u\'es-co\', u\'Colombian Spanish\'), (u\'es-mx\', u\'Mexican Spanish\'), (u\'es-ni\', u\'Nicaraguan Spanish\'), (u\'es-ve\', u\'Venezuelan Spanish\'), (u\'et\', u\'Estonian\'), (u\'eu\', u\'Basque\'), (u\'fa\', u\'Persian\'), (u\'fi\', u\'Finnish\'), (u\'fr\', u\'French\'), (u\'fy\', u\'Frisian\'), (u\'ga\', u\'Irish\'), (u\'gd\', u\'Scottish Gaelic\'), (u\'gl\', u\'Galician\'), (u\'he\', u\'Hebrew\'), (u\'hi\', u\'Hindi\'), (u\'hr\', u\'Croatian\'), (u\'hsb\', u\'Upper Sorbian\'), (u\'hu\', u\'Hungarian\'), (u\'ia\', u\'Interlingua\'), (u\'id\', u\'Indonesian\'), (u\'io\', u\'Ido\'), (u\'is\', u\'Icelandic\'), (u\'it\', u\'Italian\'), (u\'ja\', u\'Japanese\'), (u\'ka\', u\'Georgian\'), (u\'kk\', u\'Kazakh\'), (u\'km\', u\'Khmer\'), (u\'kn\', u\'Kannada\'), (u\'ko\', u\'Korean\'), (u\'lb\', u\'Luxembourgish\'), (u\'lt\', u\'Lithuanian\'), (u\'lv\', u\'Latvian\'), (u\'mk\', u\'Macedonian\'), (u\'ml\', u\'Malayalam\'), (u\'mn\', u\'Mongolian\'), (u\'mr\', u\'Marathi\'), (u\'my\', u\'Burmese\'), (u\'nb\', u\'Norwegian Bokm\\xe5l\'), (u\'ne\', u\'Nepali\'), (u\'nl\', u\'Dutch\'), (u\'nn\', u\'Norwegian Nynorsk\'), (u\'os\', u\'Ossetic\'), (u\'pa\', u\'Punjabi\'), (u\'pl\', u\'Polish\'), (u\'pt\', u\'Portuguese\'), (u\'pt-br\', u\'Brazilian Portuguese\'), (u\'ro\', u\'Romanian\'), (u\'ru\', u\'Russian\'), (u\'sk\', u\'Slovak\'), (u\'sl\', u\'Slovenian\'), (u\'sq\', u\'Albanian\'), (u\'sr\', u\'Serbian\'), (u\'sr-latn\', u\'Serbian Latin\'), (u\'sv\', u\'Swedish\'), (u\'sw\', u\'Swahili\'), (u\'ta\', u\'Tamil\'), (u\'te\', u\'Telugu\'), (u\'th\', u\'Thai\'), (u\'tr\', u\'Turkish\'), (u\'tt\', u\'Tatar\'), (u\'udm\', u\'Udmurt\'), (u\'uk\', u\'Ukrainian\'), (u\'ur\', u\'Urdu\'), (u\'vi\', u\'Vietnamese\'), (u\'zh-hans\', u\'Simplified Chinese\'), (u\'zh-hant\', u\'Traditional Chinese\')], \'user\': <SimpleLazyObject: <User: sreddy>>, u\'agent\': <django_agent_trust.models.Agent object at 0x110d54a10>, u\'LANGUAGE_CODE\': \'en-us\', \'DEFAULT_MESSAGE_LEVELS\': {\'DEBUG\': 10, \'INFO\': 20, \'WARNING\': 30, \'SUCCESS\': 25, \'ERROR\': 40}, \'srns\': 4254, u\'LANGUAGE_BIDI\': False, u\'MEDIA_URL\': \'https://local-aem-test.s3.amazonaws.com/\'}, {}, {\'record_count\': None, \'content_type_id\': u\'sources.filesource\', u\'LANGUAGE_CODE\': \'en-us\', u\'LANGUAGE_BIDI\': False, \'serializer_label\': u\'Customer\', \'label\': u\'Run for customer\', \'object_id\': 2, \'last_update\': datetime.datetime(2020, 6, 17, 23, 53, 49, 397460, tzinfo=<UTC>), \'object_label\': u\'customer\', \'error\': "NotImplementedError: This backend doesn\'t support absolute paths.", \'loader\': u\'data_wizard.loaders.FileLoader\', u\'id\': 2, \'serializer\': \'data_wizard.registry.CustomerSerializer\'}, {\'block\': <Block Node: content. Contents: [<TextNode: u\'\\n\'>, <IfNode>, <TextNode: u\'\\n<table class="mappings">\'>, <For Node: for col in result.columns, tail_len: 7>, <TextNode: u\'\\n  </tbody>\\n</table>\\n\'>, <IfNode>, <TextNode: u\'\\n\'>]>}]'
DEBUG:django.template:Exception while resolving variable 'result' in template 'data_wizard/run_columns.html'.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
    (bit, current))  # missing attribute
VariableDoesNotExist: Failed lookup for key [result] in u'[{\'False\': False, \'None\': None, \'True\': True}, {u\'csrf_token\': <SimpleLazyObject: <function _get_val at 0x110e71140>>, \'perms\': <django.contrib.auth.context_processors.PermWrapper object at 0x110d6e950>, \'messages\': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x110d54f50>, u\'request\': <rest_framework.request.Request object at 0x110d53390>, u\'TIME_ZONE\': \'UTC\', u\'STATIC_URL\': \'/static/\', u\'LANGUAGES\': [(u\'af\', u\'Afrikaans\'), (u\'ar\', u\'Arabic\'), (u\'ast\', u\'Asturian\'), (u\'az\', u\'Azerbaijani\'), (u\'bg\', u\'Bulgarian\'), (u\'be\', u\'Belarusian\'), (u\'bn\', u\'Bengali\'), (u\'br\', u\'Breton\'), (u\'bs\', u\'Bosnian\'), (u\'ca\', u\'Catalan\'), (u\'cs\', u\'Czech\'), (u\'cy\', u\'Welsh\'), (u\'da\', u\'Danish\'), (u\'de\', u\'German\'), (u\'dsb\', u\'Lower Sorbian\'), (u\'el\', u\'Greek\'), (u\'en\', u\'English\'), (u\'en-au\', u\'Australian English\'), (u\'en-gb\', u\'British English\'), (u\'eo\', u\'Esperanto\'), (u\'es\', u\'Spanish\'), (u\'es-ar\', u\'Argentinian Spanish\'), (u\'es-co\', u\'Colombian Spanish\'), (u\'es-mx\', u\'Mexican Spanish\'), (u\'es-ni\', u\'Nicaraguan Spanish\'), (u\'es-ve\', u\'Venezuelan Spanish\'), (u\'et\', u\'Estonian\'), (u\'eu\', u\'Basque\'), (u\'fa\', u\'Persian\'), (u\'fi\', u\'Finnish\'), (u\'fr\', u\'French\'), (u\'fy\', u\'Frisian\'), (u\'ga\', u\'Irish\'), (u\'gd\', u\'Scottish Gaelic\'), (u\'gl\', u\'Galician\'), (u\'he\', u\'Hebrew\'), (u\'hi\', u\'Hindi\'), (u\'hr\', u\'Croatian\'), (u\'hsb\', u\'Upper Sorbian\'), (u\'hu\', u\'Hungarian\'), (u\'ia\', u\'Interlingua\'), (u\'id\', u\'Indonesian\'), (u\'io\', u\'Ido\'), (u\'is\', u\'Icelandic\'), (u\'it\', u\'Italian\'), (u\'ja\', u\'Japanese\'), (u\'ka\', u\'Georgian\'), (u\'kk\', u\'Kazakh\'), (u\'km\', u\'Khmer\'), (u\'kn\', u\'Kannada\'), (u\'ko\', u\'Korean\'), (u\'lb\', u\'Luxembourgish\'), (u\'lt\', u\'Lithuanian\'), (u\'lv\', u\'Latvian\'), (u\'mk\', u\'Macedonian\'), (u\'ml\', u\'Malayalam\'), (u\'mn\', u\'Mongolian\'), (u\'mr\', u\'Marathi\'), (u\'my\', u\'Burmese\'), (u\'nb\', u\'Norwegian Bokm\\xe5l\'), (u\'ne\', u\'Nepali\'), (u\'nl\', u\'Dutch\'), (u\'nn\', u\'Norwegian Nynorsk\'), (u\'os\', u\'Ossetic\'), (u\'pa\', u\'Punjabi\'), (u\'pl\', u\'Polish\'), (u\'pt\', u\'Portuguese\'), (u\'pt-br\', u\'Brazilian Portuguese\'), (u\'ro\', u\'Romanian\'), (u\'ru\', u\'Russian\'), (u\'sk\', u\'Slovak\'), (u\'sl\', u\'Slovenian\'), (u\'sq\', u\'Albanian\'), (u\'sr\', u\'Serbian\'), (u\'sr-latn\', u\'Serbian Latin\'), (u\'sv\', u\'Swedish\'), (u\'sw\', u\'Swahili\'), (u\'ta\', u\'Tamil\'), (u\'te\', u\'Telugu\'), (u\'th\', u\'Thai\'), (u\'tr\', u\'Turkish\'), (u\'tt\', u\'Tatar\'), (u\'udm\', u\'Udmurt\'), (u\'uk\', u\'Ukrainian\'), (u\'ur\', u\'Urdu\'), (u\'vi\', u\'Vietnamese\'), (u\'zh-hans\', u\'Simplified Chinese\'), (u\'zh-hant\', u\'Traditional Chinese\')], \'user\': <SimpleLazyObject: <User: sreddy>>, u\'agent\': <django_agent_trust.models.Agent object at 0x110d54a10>, u\'LANGUAGE_CODE\': \'en-us\', \'DEFAULT_MESSAGE_LEVELS\': {\'DEBUG\': 10, \'INFO\': 20, \'WARNING\': 30, \'SUCCESS\': 25, \'ERROR\': 40}, \'srns\': 4254, u\'LANGUAGE_BIDI\': False, u\'MEDIA_URL\': \'https://local-aem-test.s3.amazonaws.com/\'}, {}, {\'record_count\': None, \'content_type_id\': u\'sources.filesource\', u\'LANGUAGE_CODE\': \'en-us\', u\'LANGUAGE_BIDI\': False, \'serializer_label\': u\'Customer\', \'label\': u\'Run for customer\', \'object_id\': 2, \'last_update\': datetime.datetime(2020, 6, 17, 23, 53, 49, 397460, tzinfo=<UTC>), \'object_label\': u\'customer\', \'error\': "NotImplementedError: This backend doesn\'t support absolute paths.", \'loader\': u\'data_wizard.loaders.FileLoader\', u\'id\': 2, \'serializer\': \'data_wizard.registry.CustomerSerializer\'}, {\'block\': <Block Node: content. Contents: [<TextNode: u\'\\n\'>, <IfNode>, <TextNode: u\'\\n<table class="mappings">\'>, <For Node: for col in result.columns, tail_len: 7>, <TextNode: u\'\\n  </tbody>\\n</table>\\n\'>, <IfNode>, <TextNode: u\'\\n\'>]>}, {}]'
DEBUG:django.template:Exception while resolving variable 'result' in template 'data_wizard/run_columns.html'.
Traceback (most recent call last):
  File "/Users/AEM/venv/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
    (bit, current))  # missing attribute
VariableDoesNotExist: Failed lookup for key [result] in u'[{\'False\': False, \'None\': None, \'True\': True}, {u\'csrf_token\': <SimpleLazyObject: <function _get_val at 0x110e71140>>, \'perms\': <django.contrib.auth.context_processors.PermWrapper object at 0x110d6e950>, \'messages\': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x110d54f50>, u\'request\': <rest_framework.request.Request object at 0x110d53390>, u\'TIME_ZONE\': \'UTC\', u\'STATIC_URL\': \'/static/\', u\'LANGUAGES\': [(u\'af\', u\'Afrikaans\'), (u\'ar\', u\'Arabic\'), (u\'ast\', u\'Asturian\'), (u\'az\', u\'Azerbaijani\'), (u\'bg\', u\'Bulgarian\'), (u\'be\', u\'Belarusian\'), (u\'bn\', u\'Bengali\'), (u\'br\', u\'Breton\'), (u\'bs\', u\'Bosnian\'), (u\'ca\', u\'Catalan\'), (u\'cs\', u\'Czech\'), (u\'cy\', u\'Welsh\'), (u\'da\', u\'Danish\'), (u\'de\', u\'German\'), (u\'dsb\', u\'Lower Sorbian\'), (u\'el\', u\'Greek\'), (u\'en\', u\'English\'), (u\'en-au\', u\'Australian English\'), (u\'en-gb\', u\'British English\'), (u\'eo\', u\'Esperanto\'), (u\'es\', u\'Spanish\'), (u\'es-ar\', u\'Argentinian Spanish\'), (u\'es-co\', u\'Colombian Spanish\'), (u\'es-mx\', u\'Mexican Spanish\'), (u\'es-ni\', u\'Nicaraguan Spanish\'), (u\'es-ve\', u\'Venezuelan Spanish\'), (u\'et\', u\'Estonian\'), (u\'eu\', u\'Basque\'), (u\'fa\', u\'Persian\'), (u\'fi\', u\'Finnish\'), (u\'fr\', u\'French\'), (u\'fy\', u\'Frisian\'), (u\'ga\', u\'Irish\'), (u\'gd\', u\'Scottish Gaelic\'), (u\'gl\', u\'Galician\'), (u\'he\', u\'Hebrew\'), (u\'hi\', u\'Hindi\'), (u\'hr\', u\'Croatian\'), (u\'hsb\', u\'Upper Sorbian\'), (u\'hu\', u\'Hungarian\'), (u\'ia\', u\'Interlingua\'), (u\'id\', u\'Indonesian\'), (u\'io\', u\'Ido\'), (u\'is\', u\'Icelandic\'), (u\'it\', u\'Italian\'), (u\'ja\', u\'Japanese\'), (u\'ka\', u\'Georgian\'), (u\'kk\', u\'Kazakh\'), (u\'km\', u\'Khmer\'), (u\'kn\', u\'Kannada\'), (u\'ko\', u\'Korean\'), (u\'lb\', u\'Luxembourgish\'), (u\'lt\', u\'Lithuanian\'), (u\'lv\', u\'Latvian\'), (u\'mk\', u\'Macedonian\'), (u\'ml\', u\'Malayalam\'), (u\'mn\', u\'Mongolian\'), (u\'mr\', u\'Marathi\'), (u\'my\', u\'Burmese\'), (u\'nb\', u\'Norwegian Bokm\\xe5l\'), (u\'ne\', u\'Nepali\'), (u\'nl\', u\'Dutch\'), (u\'nn\', u\'Norwegian Nynorsk\'), (u\'os\', u\'Ossetic\'), (u\'pa\', u\'Punjabi\'), (u\'pl\', u\'Polish\'), (u\'pt\', u\'Portuguese\'), (u\'pt-br\', u\'Brazilian Portuguese\'), (u\'ro\', u\'Romanian\'), (u\'ru\', u\'Russian\'), (u\'sk\', u\'Slovak\'), (u\'sl\', u\'Slovenian\'), (u\'sq\', u\'Albanian\'), (u\'sr\', u\'Serbian\'), (u\'sr-latn\', u\'Serbian Latin\'), (u\'sv\', u\'Swedish\'), (u\'sw\', u\'Swahili\'), (u\'ta\', u\'Tamil\'), (u\'te\', u\'Telugu\'), (u\'th\', u\'Thai\'), (u\'tr\', u\'Turkish\'), (u\'tt\', u\'Tatar\'), (u\'udm\', u\'Udmurt\'), (u\'uk\', u\'Ukrainian\'), (u\'ur\', u\'Urdu\'), (u\'vi\', u\'Vietnamese\'), (u\'zh-hans\', u\'Simplified Chinese\'), (u\'zh-hant\', u\'Traditional Chinese\')], \'user\': <SimpleLazyObject: <User: sreddy>>, u\'agent\': <django_agent_trust.models.Agent object at 0x110d54a10>, u\'LANGUAGE_CODE\': \'en-us\', \'DEFAULT_MESSAGE_LEVELS\': {\'DEBUG\': 10, \'INFO\': 20, \'WARNING\': 30, \'SUCCESS\': 25, \'ERROR\': 40}, \'srns\': 4254, u\'LANGUAGE_BIDI\': False, u\'MEDIA_URL\': \'https://local-aem-test.s3.amazonaws.com/\'}, {}, {\'record_count\': None, \'content_type_id\': u\'sources.filesource\', u\'LANGUAGE_CODE\': \'en-us\', u\'LANGUAGE_BIDI\': False, \'serializer_label\': u\'Customer\', \'label\': u\'Run for customer\', \'object_id\': 2, \'last_update\': datetime.datetime(2020, 6, 17, 23, 53, 49, 397460, tzinfo=<UTC>), \'object_label\': u\'customer\', \'error\': "NotImplementedError: This backend doesn\'t support absolute paths.", \'loader\': u\'data_wizard.loaders.FileLoader\', u\'id\': 2, \'serializer\': \'data_wizard.registry.CustomerSerializer\'}, {\'block\': <Block Node: content. Contents: [<TextNode: u\'\\n\'>, <IfNode>, <TextNode: u\'\\n<table class="mappings">\'>, <For Node: for col in result.columns, tail_len: 7>, <TextNode: u\'\\n  </tbody>\\n</table>\\n\'>, <IfNode>, <TextNode: u\'\\n\'>]>}]'
INFO:django.server:"GET /datawizard/2/columns/ HTTP/1.1" 200 1697
INFO:django.server:"GET /static/data_wizard/css/base.css HTTP/1.1" 304 0

[techdragon]

Because of the way this library leans on itertable, this is trickier than i expected. So far I've got to subclass/replace just as many parts of itertable as data_wizard so I'm probably going to package my final fix up like the separate data_wizard.sources app since otherwise it seems like I'd just be adding a lot of otherwise useless functionality to itertable to support data_wizard in order to get data_wizard to support django-storages.

[sheppard]

I have integrated Django Data Wizard with S3 before, but that was with very large (multi-GB) CSV files and some heavy customization to parallelize the import across several Lambda workers. For the simple case with django-storages, I think all that is needed is to get the file data from S3 into a BytesIO on the file attribute of a custom itertable class. Perhaps something like this:

# myapp/wizard.py

from itertable import ExcelFileIter  # or whichever format
from data_wizard.loaders import FileLoader

class ExcelS3Iter(ExcelFileIter):
    def load(self):
        response = s3.get_object(
            Bucket=self.bucket,
            Key=self.filename,
        )
        self.file = response['Body']  # I think this is already file-like

class S3Loader(FileLoader):
    def load_iter(self):
        return ExcelS3Iter(
            bucket=settings.AWS_STORAGE_BUCKET_NAME,
            filename=self.file.name,
        )

# myproject/settings.py
DATA_WIZARD = {
    'LOADER': 'myapp.wizard.S3Loader',
}

I believe no other customization of itertable should be necessary. That said, itertable exists primarily to support data_wizard - so anything to make that support better is a valid contribution IMO. The only caveat is that itertable shouldn't need to know anything about Django - so it could for example have an optional boto3 integration, but not django-storages specifically.

The goal with having itertable be its own library is to make it easier to test the file loading and parsing code in isolation, without worrying about the complexity introduced by data_wizard's task runner.

[sheppard]

Actually, I think a better approach in this specific case is to update itertable.load_file() to accept arbitrary file-like objects (wq/itertable@5a47f32). Then in data_wizard it's just a matter of passing the file object directly from the storage backend to itertable (4a63066).

I haven't tested this with django-storages specifically, but it should just work. If you would like to try it out before the next release, be sure to update both itertable and data_wizard to the latest development builds.

[sheppard]

This fix has been released.