[CT-3385] E: Credentials in profile "invalid: 'database' is a required property" with dbt-core 1.7.x

[VarunSharma15]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

With dbt-teradata adapter,

We are seeing an issue while running a sample project(jaffle shop) on local branch with dbt-core 1.7 .
We are getting the below mentioned error, although database is an optional property in our connections.py file.

Schema and database are same in Teradata database, sample project is working fine with previous versions until v1.6.7

(failure) PS C:\dbt-teradata\jaffle_shop\jaffle_shop> dbt debug
15:25:52  Running with dbt=1.7.0
15:25:52  dbt version: 1.7.0
15:25:52  python version: 3.9.12
15:25:52  python path: C:\dbt-teradata\failure\Scripts\python.exe
15:25:52  os info: Windows-10-10.0.19045-SP0
15:25:52  Using profiles dir at C:\Users\mt255026\.dbt
15:25:52  Using profiles.yml file at C:\Users\mt255026\.dbt\profiles.yml
15:25:52  Using dbt_project.yml file at C:\dbt-teradata\jaffle_shop\jaffle_shop\dbt_project.yml
15:25:52  Configuration:
15:25:52    profiles.yml file [ERROR invalid]
15:25:52    dbt_project.yml file [OK found and valid]
15:25:52  Required dependencies:
15:25:52   - git [OK found]

15:25:52  Connection test skipped since no profile was found
15:25:52  1 check failed:
15:25:52  Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "jaffle_shop", target "dev" invalid: 'database' is a required property

Expected Behavior

dbt commands should pass without database property mentioned

C:\dbt-teradata\jaffle_shop\jaffle_shop> dbt debug
06:31:27  Running with dbt=1.6.7
06:31:27  dbt version: 1.6.7
06:31:27  python version: 3.9.12
06:31:27  python path: C:\Users\mt255026\AppData\Local\Programs\Python\Python39\python.exe
06:31:27  os info: Windows-10-10.0.19045-SP0
06:31:27  Using profiles dir at C:\Users\mt255026\.dbt
06:31:27  Using profiles.yml file at C:\Users\mt255026\.dbt\profiles.yml
06:31:27  Using dbt_project.yml file at C:\dbt-teradata\jaffle_shop\jaffle_shop\dbt_project.yml
06:31:27  adapter type: teradata
06:31:27  adapter version: 1.0.0
06:31:27  Configuration:
06:31:27    profiles.yml file [OK found and valid]
06:31:27    dbt_project.yml file [OK found and valid]
06:31:27  Required dependencies:
06:31:27   - git [OK found]

06:31:27  Connection:
06:31:27    server: sdt47039.labs.teradata.com
06:31:27    port: None
06:31:27    database: None
06:31:27    schema: jaffle_shop
06:31:27    user: mt255026
06:31:27    tmode: ANSI
06:31:27    logmech: TD2
06:31:27    account: None
06:31:27    column_name: None
06:31:27    cop: None
06:31:27    coplast: None
06:31:27    encryptdata: None
06:31:27    fake_result_sets: None
06:31:27    field_quote: None
06:31:27    field_sep: None
06:31:27    lob_support: None
06:31:27    log: None
06:31:27    logdata: None
06:31:27    max_message_body: None
06:31:27    partition: None
06:31:27    sip_support: None
06:31:27    teradata_values: None
06:31:27    sslmode: None
06:31:27    sslca: None
06:31:27    sslcapath: None
06:31:27    sslcrc: None
06:31:27    sslcipher: None
06:31:27    sslprotocol: None
06:31:27    browser: None
06:31:27    browser_tab_timeout: None
06:31:27    browser_timeout: None
06:31:27    sp_spl: None
06:31:27    sessions: None
06:31:27    runstartup: None
06:31:27    logon_timeout: None
06:31:27    https_port: None
06:31:27    connect_timeout: None
06:31:27    request_timeout: None
06:31:27  Registered adapter: teradata=1.0.0
06:31:39    Connection test: [OK connection ok]

06:31:39  All checks passed!

Steps To Reproduce

1. Clone dbt-teradata from https://github.com/Teradata/dbt-teradata.git
2. Make changes in setup.py file at line no. 48. Make dbt-core==1.7.1
3. Change requirements_dev.txt at line number 6 and 9 with dbt-tests-adapter==1.7.1 and dbt-core==1.7.1
4. Run : pip install -r requirements_dev.txt
5. Run any functional testcase with : pytest -v tests/functional/adapter/test_basic.py

Relevant log output

No response

Environment

- OS: Windows
- Python: 3.10
- dbt: 1.7.x

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

dbt-teradata adapter

We found a similar issue on dbt-databricks
databricks/dbt-databricks#469

[dbeatty10]

Thanks for reaching out @VarunSharma15 !

The issue title says "database", but the error message you shared says "schema":

  Credentials in profile "jaffle_shop", target "dev" invalid: 'schema' is a required property

Is the issue title supposed to say "schema"?

[VarunSharma15]

Hi @dbeatty10,
Sorry, I mistakenly shared different code snippet, let me share the exact error.
Let me correct the above snippet as well.
It's an issue with database property.

(failure) PS C:\dbt-teradata\jaffle_shop\jaffle_shop> dbt debug
15:25:52  Running with dbt=1.7.0
15:25:52  dbt version: 1.7.0
15:25:52  python version: 3.9.12
15:25:52  python path: C:\dbt-teradata\failure\Scripts\python.exe
15:25:52  os info: Windows-10-10.0.19045-SP0
15:25:52  Using profiles dir at C:\Users\mt255026\.dbt
15:25:52  Using profiles.yml file at C:\Users\mt255026\.dbt\profiles.yml
15:25:52  Using dbt_project.yml file at C:\dbt-teradata\jaffle_shop\jaffle_shop\dbt_project.yml
15:25:52  Configuration:
15:25:52    profiles.yml file [ERROR invalid]
15:25:52    dbt_project.yml file [OK found and valid]
15:25:52  Required dependencies:
15:25:52   - git [OK found]

15:25:52  Connection test skipped since no profile was found
15:25:52  1 check failed:
15:25:52  Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "jaffle_shop", target "dev" invalid: 'database' is a required property

[dbeatty10]

Thanks for that update @VarunSharma15 👍

Could you take a look in your logs (logs/dbt.log) and look for the associated Python stacktrace as well? If you can share that, it will help us see the code paths that are involved in raising this exception.

[VarunSharma15]

Hi @dbeatty10
There isn't much I can see in dbt.log
dbt.log

I am also attaching result of a testcase run
testcase_output.txt

[dbeatty10]

Looking at a portion of your test case output, it looks like the underlying issue is a JSON schema validation error:

jsonschema.exceptions.ValidationError: 'database' is a required property

Failed validating 'required' in schema:
    {'additionalProperties': True,
...
                    'database': {'anyOf': [{'type': 'string'},
                                           {'type': 'null'}]},
    

But the {'type': 'null'} portion above makes it look like it can be null though.

Did you try providing a value for database, but leaving it as None?

[dataders]

@VarunSharma15 can you try and see ifhttps://github.com/Teradata/dbt-teradata/pull/127 resolves things?

[tallamohan]

@dataders , we already tried this solution, but failed with the below error:

non-default argument follows default argument

[dataders]

@tallamohan I'm going to close this issue in favor of our thread in the upgrade discussion, given that this is not an issue with dbt-core: #8307 (comment)