[CT-2748] [Feature] A single Structured Logging JSON dictionary per Exception

[moltar]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I am not 100% sure if this can be considered a bug. I am filing it as a bug, as the end user and a system admin.

It appears that the structured logging (as JSON) feature is splitting error messages coming from the database into separate entries.

Here's an example of one database error being represented by 6 separate JSON events:

Expected Behavior

One database error should be packed into a single JSON event.

Steps To Reproduce

1. Use dbt-postgres
2. Set env DBT_LOG_FORMAT=json
3. Create a model that produces a database error (e.g. select 1/0)
4. Do a dbt run
5. Observe the log output

Relevant log output

Environment

- OS: Linux
- Python: 3.11
- dbt: 1.5.2

Which database adapter are you using with dbt?

postgres

Additional Context

N/A

[peterallenwebb]

I agree that this is not a desirable behavior. It is an artifact of the initial move to structured logging, which took the approach of converting each log line to its own independent message. We have since abandoned that approach, but it lingers in this issue. We should adopt the behavior @moltar suggests.

[dbeatty10]

Thank you for raising this @moltar! And thanks for the insight @peterallenwebb 🧠

Per @gshank, here is the relevant section of code:

dbt-core/core/dbt/task/printer.py

Lines 120 to 129 in 6a1e3a6

	elif result.message is not None:
	first = True
	for line in result.message.split("\n"):
	# TODO: why do we format like this? Is there a reason this needs to
	# be split instead of sending it as a single log line?
	if first:
	fire_event(FirstRunResultError(msg=line))
	first = False
	else:
	fire_event(AfterFirstRunResultError(msg=line))

I'm going to re-categorize this as a feature in the meantime.

[graciegoheen]

Note: This is a fix to improve Cloud's ingestion of the structured logging

[peterallenwebb]

Noting that the Experience team has raised concerns about our plan to back-port this, and would like time to work with an RC first. That shouldn't stop us from merging to main, however.