Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Added new Error to TypedDict #14225

Merged
merged 20 commits into from
Jan 27, 2023
Merged

✨ Added new Error to TypedDict #14225

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
c7824be
✨ Added new Error to TypedDict
JoaquimEsteves Nov 30, 2022
f19a7f7
🐛 Fixed 'missing keys' not being reported
JoaquimEsteves Nov 30, 2022
2a4220c
🐛 Early return in case there's missing or extra keys
JoaquimEsteves Nov 30, 2022
fcf4909
🧑‍🔬 Fixed simple error-code tests
JoaquimEsteves Nov 30, 2022
deb6543
Update test-data/unit/check-typeddict.test
JoaquimEsteves Dec 2, 2022
222869d
Update test-data/unit/check-errorcodes.test
JoaquimEsteves Dec 2, 2022
70c0a18
✨ We now typecheck despite having an extra-key
JoaquimEsteves Dec 5, 2022
eda0c8b
🧹 Setting an extra value on a TypedDict now has the correct error code
JoaquimEsteves Dec 5, 2022
25f3dbc
✨ We can now set an item and get the correct error code
JoaquimEsteves Dec 5, 2022
c7c687f
🧑‍🔬 Redid the `IgnotErrorCodeTypedDictNoteIgnore` test
JoaquimEsteves Dec 5, 2022
a275c86
🐈‍⬛ Black
JoaquimEsteves Dec 8, 2022
f7a7f4f
🐛 Added Review Suggestions
JoaquimEsteves Dec 8, 2022
06a3866
📝 Fixed docstring not being in the correct format.
JoaquimEsteves Dec 8, 2022
780e10d
🐛 Fixed ignoring unknown-key not silencing the 'Did you mean' message
JoaquimEsteves Dec 8, 2022
fb98524
Review Feedback - Better Docstrings
JoaquimEsteves Jan 23, 2023
708d322
📝 Added `unknown-key` to the `error_code_list`
JoaquimEsteves Jan 23, 2023
e4b82b6
🐛 Black and sphinx bugs
JoaquimEsteves Jan 23, 2023
1c9a623
Merge branch 'master' into master
JoaquimEsteves Jan 23, 2023
f4f0cd0
Merge branch 'master' into master
JoaquimEsteves Jan 23, 2023
957ba17
Apply suggestions from code review (docs)
ilevkivskyi Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions mypy/checkexpr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -789,17 +789,22 @@ def check_typeddict_call_with_kwargs(
context: Context,
orig_callee: Type | None,
) -> Type:
if not (callee.required_keys <= set(kwargs.keys()) <= set(callee.items.keys())):
actual_keys = kwargs.keys()
found_set = set(actual_keys)
if not (callee.required_keys <= found_set <= set(callee.items.keys())):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really need to convert these to sets? IIRC dictionary .keys() already behave like sets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're quite right. I had no idea.

Fixed in f7a7f4f

expected_keys = [
key
for key in callee.items.keys()
if key in callee.required_keys or key in kwargs.keys()
if key in callee.required_keys or key in found_set
]
actual_keys = kwargs.keys()
self.msg.unexpected_typeddict_keys(
callee, expected_keys=expected_keys, actual_keys=list(actual_keys), context=context
)
return AnyType(TypeOfAny.from_error)
if callee.required_keys > found_set:
# found_set is not a sub-set of the required_keys
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment contradicts the condition, the if condition specifically checks that found_set is a subset of required_keys.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: f7a7f4f#diff-42eda644ce5003e33ea0766d7601fcfb4934ad106429abfe115954a1fccddd07R804

# This means we're dealing with something weird we can't
# properly type
return AnyType(TypeOfAny.from_error)

orig_callee = get_proper_type(orig_callee)
if isinstance(orig_callee, CallableType):
Expand Down Expand Up @@ -3735,7 +3740,9 @@ def nonliteral_tuple_index_helper(self, left_type: TupleType, index: Expression)
return self.chk.named_generic_type("builtins.tuple", [union])
return union

def visit_typeddict_index_expr(self, td_type: TypedDictType, index: Expression) -> Type:
def visit_typeddict_index_expr(
self, td_type: TypedDictType, index: Expression, setitem: bool = False
) -> Type:
if isinstance(index, StrExpr):
key_names = [index.value]
else:
Expand Down Expand Up @@ -3764,7 +3771,7 @@ def visit_typeddict_index_expr(self, td_type: TypedDictType, index: Expression)
for key_name in key_names:
value_type = td_type.items.get(key_name)
if value_type is None:
self.msg.typeddict_key_not_found(td_type, key_name, index)
self.msg.typeddict_key_not_found(td_type, key_name, index, setitem)
return AnyType(TypeOfAny.from_error)
else:
value_types.append(value_type)
Expand Down
4 changes: 3 additions & 1 deletion mypy/checkmember.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1068,7 +1068,9 @@ def analyze_typeddict_access(
if isinstance(mx.context, IndexExpr):
# Since we can get this during `a['key'] = ...`
# it is safe to assume that the context is `IndexExpr`.
item_type = mx.chk.expr_checker.visit_typeddict_index_expr(typ, mx.context.index)
item_type = mx.chk.expr_checker.visit_typeddict_index_expr(
typ, mx.context.index, setitem=True
)
else:
# It can also be `a.__setitem__(...)` direct call.
# In this case `item_type` can be `Any`,
Expand Down
3 changes: 3 additions & 0 deletions mypy/errorcodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ def __str__(self) -> str:
TYPEDDICT_ITEM: Final = ErrorCode(
"typeddict-item", "Check items when constructing TypedDict", "General"
)
TYPPEDICT_UNKNOWN_KEY: Final = ErrorCode(
"typeddict-unknown-key", "Check unknown keys when constructing TypedDict", "General"
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned on the issue, for consistency this error code should be also used for errors in case like:

d["unknown"]
d["unknown"] = 42

Copy link
Contributor Author

@JoaquimEsteves JoaquimEsteves Dec 2, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, that makes sense, but I'm afraid I don't like it. I'm worried about what would happen when the user would ignore this error. For example

Running mypy --disable-error-code=typeddict-unknown-key on the snippet:

A = T.TypedDict("A", {"x": int})

def f(x: A) -> None:
    X["obvious error"]

f({"x": 1, "y": "foo"})  # I want this to be OK

Would result in an obvious error being completely ignored by mypy.

So to my view we have to options:

1 - rename the unknown-key to something that better reflects the code (unknown-anon-key for example)
2 - Create another error code for this usecase.

Open to suggestions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, reading an unknown key may be too much, but setting it (i.e. d["unknown"] = 42) should definitely use the same error code. I don't see a difference between:

d: TD = {"known": 1, "unknown": 2}

and

d: TD = {"known": 1}
d["unknown"] = 2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's very reasonable. Will ping you when I have that sorted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Not sure if the solution is pretty. Added a test for it on the typeddict file.

HAS_TYPE: Final = ErrorCode(
"has-type", "Check that type of reference can be determined", "General"
)
Expand Down
42 changes: 24 additions & 18 deletions mypy/messages.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1612,30 +1612,28 @@ def unexpected_typeddict_keys(
expected_set = set(expected_keys)
if not typ.is_anonymous():
# Generate simpler messages for some common special cases.
if actual_set < expected_set:
# Use list comprehension instead of set operations to preserve order.
missing = [key for key in expected_keys if key not in actual_set]
# Use list comprehension instead of set operations to preserve order.
missing = [key for key in expected_keys if key not in actual_set]
if missing:
self.fail(
"Missing {} for TypedDict {}".format(
format_key_list(missing, short=True), format_type(typ)
),
context,
code=codes.TYPEDDICT_ITEM,
)
extra = [key for key in actual_keys if key not in expected_set]
if extra:
self.fail(
"Extra {} for TypedDict {}".format(
format_key_list(extra, short=True), format_type(typ)
),
context,
code=codes.TYPPEDICT_UNKNOWN_KEY,
)
if missing or extra:
# No need to check for further errors
return
else:
extra = [key for key in actual_keys if key not in expected_set]
if extra:
# If there are both extra and missing keys, only report extra ones for
# simplicity.
self.fail(
"Extra {} for TypedDict {}".format(
format_key_list(extra, short=True), format_type(typ)
),
context,
code=codes.TYPEDDICT_ITEM,
)
return
found = format_key_list(actual_keys, short=True)
if not expected_keys:
self.fail(f"Unexpected TypedDict {found}", context)
Expand All @@ -1655,8 +1653,16 @@ def typeddict_key_must_be_string_literal(self, typ: TypedDictType, context: Cont
)

def typeddict_key_not_found(
self, typ: TypedDictType, item_name: str, context: Context
self, typ: TypedDictType, item_name: str, context: Context, setitem: bool = False
) -> None:
"""
Handles error messages.
Note, that we differentiate in between reading a value and setting
a value.
Setting a value on a TypedDict is an 'unknown-key' error,
whereas reading it is the more serious/general 'item' error.
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use common convention for docstrings. It should be like this.

def func() -> None:
    """Short description in form of "do something".

    After empty line, long description indented with function body.
    The closing quotes n a separate line.
    """

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in: 06a3866

if typ.is_anonymous():
self.fail(
'"{}" is not a valid TypedDict key; expected one of {}'.format(
Expand All @@ -1668,7 +1674,7 @@ def typeddict_key_not_found(
self.fail(
f'TypedDict {format_type(typ)} has no key "{item_name}"',
context,
code=codes.TYPEDDICT_ITEM,
code=codes.TYPPEDICT_UNKNOWN_KEY if setitem else codes.TYPEDDICT_ITEM,
)
matches = best_matches(item_name, typ.items.keys())
if matches:
Expand Down
13 changes: 10 additions & 3 deletions test-data/unit/check-errorcodes.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,11 +455,15 @@ class E(TypedDict):
y: int

a: D = {'x': ''} # E: Incompatible types (expression has type "str", TypedDict item "x" has type "int") [typeddict-item]
b: D = {'y': ''} # E: Extra key "y" for TypedDict "D" [typeddict-item]
b: D = {'y': ''} # E: Missing key "x" for TypedDict "D" [typeddict-item] \
# E: Extra key "y" for TypedDict "D" [typeddict-unknown-key]
c = D(x=0) if int() else E(x=0, y=0)
c = {} # E: Expected TypedDict key "x" but found no keys [typeddict-item]
d: D = {'x': '', 'y': 1} # E: Extra key "y" for TypedDict "D" [typeddict-unknown-key] \
# E: Incompatible types (expression has type "str", TypedDict item "x" has type "int") [typeddict-item]

a['y'] = 1 # E: TypedDict "D" has no key "y" [typeddict-item]

a['y'] = 1 # E: TypedDict "D" has no key "y" [typeddict-unknown-key]
a['x'] = 'x' # E: Value of "x" has incompatible type "str"; expected "int" [typeddict-item]
a['y'] # E: TypedDict "D" has no key "y" [typeddict-item]
[builtins fixtures/dict.pyi]
Expand All @@ -472,7 +476,10 @@ class A(TypedDict):
two_commonparts: int

a: A = {'one_commonpart': 1, 'two_commonparts': 2}
a['other_commonpart'] = 3 # type: ignore[typeddict-item]
a['other_commonpart'] = 3 # type: ignore[typeddict-unknown-key] \
# N: Did you mean "one_commonpart" or "two_commonparts"? \
# N: Error code "typeddict-item" not covered by "type: ignore" comment
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks wrong. The whole idea is that if an error is suppressed, the related notes should be suppressed as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented your suggestion in commit 780e10d

Originally, I did it on purpose as I liked the 'Did you mean' feature. It would be helpful for situations like so:

# Assume we're running mypy while ignoring the typpedict-unknown-key

F = TypedDict('F', { 'foo':int })

foo: F = { 'foo': 1 }

foo['bar'] = 2

# OOPS! I made a typo
foo['fooo'] = 2

My rationale being that it would be quite rare for a user to both want to add extra keys AND have those keys be very similar looking.

not_exist = a['not_exist'] # type: ignore[typeddict-item]
[builtins fixtures/dict.pyi]
[typing fixtures/typing-typeddict.pyi]

Expand Down
3 changes: 2 additions & 1 deletion test-data/unit/check-typeddict.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2020,7 +2020,8 @@ v = {union: 2} # E: Expected TypedDict key to be string literal
num2: Literal['num']
v = {num2: 2}
bad2: Literal['bad']
v = {bad2: 2} # E: Extra key "bad" for TypedDict "Value"
v = {bad2: 2} # E: Missing key "num" for TypedDict "Value" \
# E: Extra key "bad" for TypedDict "Value"

[builtins fixtures/dict.pyi]
[typing fixtures/typing-typeddict.pyi]
Expand Down