flattening not working - `string indices must be integers` & `TypeError: can't compare offset-naive and offset-aware datetimes` #19

fekle · 2023-05-30T13:42:14Z

Hi there,

as promised on Slack, I'm opening an issue here regarding the issue with flattening.

When using the tap without flattening enabled, everything works great, but most of the properties are buried in a json field in the resulting database table, as expected. In order to improve performance etc when using the data, flattening while extracting would be great.

The documentation for this tap mentions the options flattening_enabled and flattening_max_depth.
I tried enabling flattening with flattening_enabled, and setting flattening_max_depth to various values such as 1, 3, 9 etc - but always get the same error as shown below.

I dug into the errors a bit further, and the problem seems to be related to this tap's schema - specifically this: https://github.com/MeltanoLabs/tap-klaviyo/blob/d049862b4d8d3426481c6739f48ec98a06dda09c/tap_klaviyo/schemas/profiles.json#L175C19-L189.

When commenting out this part of the schmema (I set up the tap source locally), flattening works.
As I'm not familiar with the workings of the meltano SDK, I'm not sure on how to proceed here - but I'm sure someone with more experience should be able to find the issue :-)

This is the log output when invoking the tap with flattening enabled:

> meltano invoke tap-klaviyo                                                                                              15:37:11 
2023-05-30T13:37:15.794503Z [info     ] Environment 'dev' is active
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Catalog discovery failed: command ['/REDCACTED/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo', '--config', '/REDCACTED/.meltano/run/tap-klaviyo/tap.6aa80ce5-ba97-46d2-8691-3cdc6da9e962.config.json', '--discover'] returned 1 with stderr:
 2023-05-30 15:37:18,372 | INFO     | tap-klaviyo          | Added 'listperson' as child stream to 'lists'
Traceback (most recent call last):
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo", line 8, in <module>
    sys.exit(TapKlaviyo.cli())
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 516, in cli
    tap = cls(  # type: ignore[operator]
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 103, in __init__
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 628, in register_raw_streams_from_catalog
    self.register_raw_stream_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 671, in register_raw_stream_schema
    self.default_mapper_type(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 93, in __init__
    self.transformed_schema = self.flatten_schema(self.transformed_schema)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 141, in flatten_schema
    return flatten_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 205, in flatten_schema
    new_schema["properties"] = _flatten_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 244, in _flatten_schema
    _flatten_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 255, in _flatten_schema
    if list(v.values())[0][0]["type"] == "string":
TypeError: string indices must be integers

I'm using Postgres as the target, so i can use the tap without flattening and using jsonb selectors - so this is not a big deal, but I think it should either be fixed, or flattening removed from the documentation.
Thanks for the great tap!

The text was updated successfully, but these errors were encountered:

This PR fixes this issue reported in #19 but it also needs to be used with meltano/sdk#2243.

fekle · 2024-08-20T16:11:43Z

Hi there,

I ran into another issue using this tap: TypeError: can't compare offset-naive and offset-aware datetimes.
This occurs on all runs after the initial run or a full refresh.

> meltano run tap-klaviyo target-postgres                                                                                                                                                                                    18:06:55 
2024-08-20T16:06:59.053012Z [info     ] Environment 'dev' is active   
2024-08-20T16:06:59.177303Z [info     ] Installing extractor 'tap-klaviyo'
2024-08-20T16:07:06.272142Z [info     ] Installed extractor 'tap-klaviyo'
2024-08-20T16:07:06.272532Z [info     ] Installed 1/2 plugins         
2024-08-20T16:07:06.272589Z [info     ] Skipped installing 1/2 plugins
2024-08-20T16:07:08.731408Z [info     ] 2024-08-20 18:07:08,731 | INFO     | tap-klaviyo          | Added 'listperson' as child stream to 'lists' cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.731877Z [info     ] 2024-08-20 18:07:08,731 | INFO     | tap-klaviyo          | Skipping deselected stream 'campaigns'. cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.731970Z [info     ] 2024-08-20 18:07:08,731 | INFO     | tap-klaviyo          | Skipping deselected stream 'events'. cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732046Z [info     ] 2024-08-20 18:07:08,731 | INFO     | tap-klaviyo          | Skipping deselected stream 'flows'. cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732107Z [info     ] 2024-08-20 18:07:08,731 | INFO     | tap-klaviyo.lists    | Beginning incremental sync of 'lists'... cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732167Z [info     ] 2024-08-20 18:07:08,731 | INFO     | tap-klaviyo.lists    | Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732282Z [info     ] 2024-08-20 18:07:08,731 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 1.1682510375976562e-05, "tags": {"stream": "lists", "context": {}, "status": "failed"}} cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732394Z [info     ] 2024-08-20 18:07:08,731 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "lists", "context": {}}} cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732497Z [info     ] 2024-08-20 18:07:08,731 | ERROR    | tap-klaviyo.lists    | An unhandled error occurred while syncing 'lists' cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732555Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732612Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1203, in sync cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732667Z [info     ]     for _ in self._sync_records(context=context): cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732721Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1092, in _sync_records cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732833Z [info     ]     self._write_starting_replication_value(current_context) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.732974Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 416, in _write_starting_replication_value cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733072Z [info     ]     value = self.compare_start_date(value, start_date_value) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733131Z [info     ]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733185Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 390, in compare_start_date cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733238Z [info     ]     return max(value, start_date_value, key=datetime_fromisoformat) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733288Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733337Z [info     ] TypeError: can't compare offset-naive and offset-aware datetimes cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733385Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733435Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo", line 8, in <module> cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733484Z [info     ]     sys.exit(TapKlaviyo.cli()) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733531Z [info     ]              ^^^^^^^^^^^^^^^^  cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733579Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733719Z [info     ]     return self.main(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733830Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733892Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.733962Z [info     ]     rv = self.invoke(ctx)      cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734017Z [info     ]          ^^^^^^^^^^^^^^^^      cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734069Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/plugin_base.py", line 80, in invoke cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734122Z [info     ]     return super().invoke(ctx) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734171Z [info     ]            ^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734222Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734270Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734321Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734402Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734456Z [info     ]     return __callback(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734509Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734568Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 507, in invoke cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734618Z [info     ]     tap.sync_all()             cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734666Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 466, in sync_all cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734716Z [info     ]     stream.sync()              cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734764Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1203, in sync cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734819Z [info     ]     for _ in self._sync_records(context=context): cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734866Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1092, in _sync_records cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734915Z [info     ]     self._write_starting_replication_value(current_context) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.734965Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 416, in _write_starting_replication_value cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.735013Z [info     ]     value = self.compare_start_date(value, start_date_value) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.735061Z [info     ]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.735108Z [info     ]   File "REDACTED/meltano/.meltano/extractors/tap-klaviyo/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 390, in compare_start_date cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.735166Z [info     ]     return max(value, start_date_value, key=datetime_fromisoformat) cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.735257Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.735317Z [info     ] TypeError: can't compare offset-naive and offset-aware datetimes cmd_type=elb consumer=False job_name=dev:tap-klaviyo-to-target-postgres name=tap-klaviyo producer=True run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=tap-klaviyo
2024-08-20T16:07:08.779700Z [info     ] time=2024-08-20 18:07:08 name=target_postgres level=INFO message=Table '"lists"' exists cmd_type=elb consumer=True job_name=dev:tap-klaviyo-to-target-postgres name=target-postgres producer=False run_id=[2e07b7cc-5956-4f32-bbe2-f2b3a7b0d1ad] stdio=stderr string_id=target-postgres
2024-08-20T16:07:08.801133Z [info     ] Incremental state has been updated at 2024-08-20 16:07:08.801113+00:00.
2024-08-20T16:07:08.815490Z [error    ] Extractor failed              
2024-08-20T16:07:08.815646Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={<PluginType.EXTRACTORS: 'extractors'>: 1} set_number=0 success=False
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Extractor failed

Seems that there is a mismatch in the date types returned by this tap?

edgarrmondragon · 2024-08-20T22:37:28Z

I think the issue is caused by the SDK parsing these two values into a naive and an aware datetime objects respectively:

2024-01-01T00:00:00Z
2024-01-01T00:00:00

@fekle as a workaround, could you try changing the value of the start_date setting to have an offset? i.e. either Z, +00:00 or the more appropriate one. I'll also raise an issue in meltano/sdk for better handling of this type of edge cases.

edgarrmondragon · 2024-08-20T22:59:18Z

Hi there,

as promised on Slack, I'm opening an issue here regarding the issue with flattening.

When using the tap without flattening enabled, everything works great, but most of the properties are buried in a json field in the resulting database table, as expected. In order to improve performance etc when using the data, flattening while extracting would be great.

The documentation for this tap mentions the options flattening_enabled and flattening_max_depth. I tried enabling flattening with flattening_enabled, and setting flattening_max_depth to various values such as 1, 3, 9 etc - but always get the same error as shown below.

I dug into the errors a bit further, and the problem seems to be related to this tap's schema - specifically this: https://github.com/MeltanoLabs/tap-klaviyo/blob/d049862b4d8d3426481c6739f48ec98a06dda09c/tap_klaviyo/schemas/profiles.json#L175C19-L189.

When commenting out this part of the schmema (I set up the tap source locally), flattening works. As I'm not familiar with the workings of the meltano SDK, I'm not sure on how to proceed here - but I'm sure someone with more experience should be able to find the issue :-)

This is the log output when invoking the tap with flattening enabled:
> meltano invoke tap-klaviyo                                                                                              15:37:11 
2023-05-30T13:37:15.794503Z [info     ] Environment 'dev' is active
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Catalog discovery failed: command ['/REDCACTED/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo', '--config', '/REDCACTED/.meltano/run/tap-klaviyo/tap.6aa80ce5-ba97-46d2-8691-3cdc6da9e962.config.json', '--discover'] returned 1 with stderr:
 2023-05-30 15:37:18,372 | INFO     | tap-klaviyo          | Added 'listperson' as child stream to 'lists'
Traceback (most recent call last):
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo", line 8, in <module>
    sys.exit(TapKlaviyo.cli())
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 516, in cli
    tap = cls(  # type: ignore[operator]
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 103, in __init__
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 628, in register_raw_streams_from_catalog
    self.register_raw_stream_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 671, in register_raw_stream_schema
    self.default_mapper_type(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 93, in __init__
    self.transformed_schema = self.flatten_schema(self.transformed_schema)
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 141, in flatten_schema
    return flatten_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 205, in flatten_schema
    new_schema["properties"] = _flatten_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 244, in _flatten_schema
    _flatten_schema(
  File "/REDCACTED/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 255, in _flatten_schema
    if list(v.values())[0][0]["type"] == "string":
TypeError: string indices must be integers
I'm using Postgres as the target, so i can use the tap without flattening and using jsonb selectors - so this is not a big deal, but I think it should either be fixed, or flattening removed from the documentation. Thanks for the great tap!

For this, do you know which is the offending stream @fekle?

fekle · 2024-08-21T08:10:13Z

I think the issue is caused by the SDK parsing these two values into a naive and an aware datetime objects respectively:

2024-01-01T00:00:00Z

2024-01-01T00:00:00

@fekle as a workaround, could you try changing the value of the start_date setting to have an offset? i.e. either Z, +00:00 or the more appropriate one. I'll also raise an issue in meltano/sdk for better handling of this type of edge cases.

Hey, yes that worked! Changing start_date to 2023-01-01T00:00:00Z worked!
Thank you very much. Strange that just passing a date worked before, probably due to a meltano update?
Should we update the docs that a timezone is required in the start_date (and that it needs to be a timestamp and not a date), or can we update the tap to handle this (for example just assuming the timestamp to be UTC when there is no timezone)?

edgarrmondragon · 2024-08-22T01:23:31Z

can we update the tap to handle this (for example just assuming the timestamp to be UTC when there is no timezone)?

In a future version of singer-sdk that will be the case:

fix: Mismatch between timezone-aware and naive datetimes in start date and bookmarks is now correctly handled meltano/sdk#2613
refactor: Default to UTC when parsing dates without a known timezone meltano/sdk#2619

fekle · 2024-08-23T07:56:52Z

can we update the tap to handle this (for example just assuming the timestamp to be UTC when there is no timezone)?

In a future version of singer-sdk that will be the case:

fix: Mismatch between timezone-aware and naive datetimes in start date and bookmarks is now correctly handled meltano/sdk#2613

refactor: Default to UTC when parsing dates without a known timezone meltano/sdk#2619

Ah great, thanks for the info

fekle · 2024-08-23T07:59:00Z

For this, do you know which is the offending stream @fekle?

I just tried flattening again now, and it seems to work :-) - At least for the profiles, lists and listperson streams.

So I think we can now finally close this issue? :-D

edgarrmondragon · 2024-08-23T19:53:13Z

So I think we can now finally close this issue? :-D

Yup!

edgarrmondragon · 2024-08-23T19:53:20Z

Thanks for reporting @fekle!

fekle changed the title ~~flattening not working~~ flattening not working - string indices must be integers May 30, 2023

joaopamaral mentioned this issue Feb 15, 2024

fix: Profiles schema #49

Merged

edgarrmondragon pushed a commit that referenced this issue Feb 15, 2024

fix: Profiles schema (#49)

8314d37

This PR fixes this issue reported in #19 but it also needs to be used with meltano/sdk#2243.

fekle changed the title ~~flattening not working - string indices must be integers~~ flattening not working - string indices must be integers & TypeError: can't compare offset-naive and offset-aware datetimes Aug 20, 2024

edgarrmondragon mentioned this issue Aug 20, 2024

bug: Tap crashes when start_date and replication value are not both tz-aware or viceversa meltano/sdk#2612

Closed

1 task

edgarrmondragon closed this as completed Aug 23, 2024

edgarrmondragon self-assigned this Aug 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

flattening not working - `string indices must be integers` & `TypeError: can't compare offset-naive and offset-aware datetimes` #19

flattening not working - `string indices must be integers` & `TypeError: can't compare offset-naive and offset-aware datetimes` #19

fekle commented May 30, 2023

fekle commented Aug 20, 2024

edgarrmondragon commented Aug 20, 2024

edgarrmondragon commented Aug 20, 2024

fekle commented Aug 21, 2024

edgarrmondragon commented Aug 22, 2024

fekle commented Aug 23, 2024

fekle commented Aug 23, 2024

edgarrmondragon commented Aug 23, 2024

edgarrmondragon commented Aug 23, 2024

flattening not working - string indices must be integers & TypeError: can't compare offset-naive and offset-aware datetimes #19

flattening not working - string indices must be integers & TypeError: can't compare offset-naive and offset-aware datetimes #19

Comments

fekle commented May 30, 2023

fekle commented Aug 20, 2024

edgarrmondragon commented Aug 20, 2024

edgarrmondragon commented Aug 20, 2024

fekle commented Aug 21, 2024

edgarrmondragon commented Aug 22, 2024

fekle commented Aug 23, 2024

fekle commented Aug 23, 2024

edgarrmondragon commented Aug 23, 2024

edgarrmondragon commented Aug 23, 2024

flattening not working - `string indices must be integers` & `TypeError: can't compare offset-naive and offset-aware datetimes` #19

flattening not working - `string indices must be integers` & `TypeError: can't compare offset-naive and offset-aware datetimes` #19