Missing py.typed declaration?

[andrewmwhite]

mypy is complaining about not being able to find type annotations for httpx:

error: Cannot find module named 'httpx'

I'm somewhat new to using type annotations/static type checking in Python, but from the mypy documentation here it looks like there may be a missing declaration in setup.py?

[sethmlarson]

Hmmm, we definitely want our type annotations to be discoverable by mypy. Do you think that this is something you'd be able to tackle? :)

[ChristopherMillones]

Did you properly install the module via pip?
I had an issue occurring where I installed a module via pip but my Python wasn't in my
environment variables. Preventing my IDE from locating the installed module.

Hope this helps or makes sense :)

[sethmlarson]

It looks like we need a single py.typed file within our package to get mypy to recognize inline types? Am I reading that right?

[andrewmwhite]

Basically -- that marker file plus a package_data entry seems to have done the trick.

[kylebebak]

@andrewmwhite @sethmlarson

Hey all, I just did pip install httpx to install httpx v0.7.2.

There is still no py.typed file in /usr/local/lib/python3.7/site-packages/httpx/. All of the other files, like api.py, __init__.py, etc are present, but py.typed isn't.

Consequently, mypy still can't find types for httpx.

From what I can tell, this PR was merged before 0.7.2 was released, so this should work in v0.7.2. If this is the case, this issue needs to be reopened.

[sethmlarson]

Yep, verified this is a bug. We need to add an entry to MANIFEST.in.

@florimondmanca can we get this done before release?

[florimondmanca]

@sethmlarson Sure :)

I’m curious why package_data isn’t enough to include the py.typed file, though. It has worked for me in a few other projects.

[sethmlarson]

That was my big question too, weird.

[sethmlarson]

Thanks for reporting this @kylebebak, it'll be fixed in 0.7.3! :)

[kylebebak]

Right on, thanks all!

@sethmlarson @florimondmanca
This library is excellent btw

[sethmlarson]

@kylebebak Thank you, we appreciate your kind words! 🙇‍♂️

[andrewmwhite]

I think this is still an issue in 0.7.4 -- not sure why, but py.typed isn't getting into the installed package.

$ python3 -m venv venv
$ . ./venv/bin/activate
(venv) $ pip install httpx==0.7.4
Collecting httpx==0.7.4
  Downloading https://files.pythonhosted.org/packages/c9/f1/fdc9c48f22df5f87898826b7b1217a37e78cbccb3f1925f5d0fe28b0b4d4/httpx-0.7.4.tar.gz (57kB)
    100% |████████████████████████████████| 61kB 2.9MB/s 
Collecting certifi (from httpx==0.7.4)
  Downloading https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl (154kB)
    100% |████████████████████████████████| 163kB 4.2MB/s 
Collecting chardet==3.* (from httpx==0.7.4)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting h11==0.8.* (from httpx==0.7.4)
  Using cached https://files.pythonhosted.org/packages/f9/f3/8e4cf5fa1a3d8bda942a0b1cf92f87815494216fd439f82eb99073141ba0/h11-0.8.1-py2.py3-none-any.whl
Collecting h2==3.* (from httpx==0.7.4)
  Downloading https://files.pythonhosted.org/packages/fb/e3/823a574b33aac578c3a171b1a2225ee8e1ad0c3842f3871bdc34e270f352/h2-3.1.1-py2.py3-none-any.whl (64kB)
    100% |████████████████████████████████| 71kB 8.4MB/s 
Collecting hstspreload>=2019.8.27 (from httpx==0.7.4)
  Downloading https://files.pythonhosted.org/packages/dc/63/971f5d4eedc7ecc53dae066ca45c29fc74ed2929dad2a9290ba7cf8e9df8/hstspreload-2019.9.30.tar.gz (677kB)
    100% |████████████████████████████████| 686kB 7.9MB/s 
Collecting idna==2.* (from httpx==0.7.4)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting rfc3986==1.* (from httpx==0.7.4)
  Using cached https://files.pythonhosted.org/packages/00/8d/9d56bfe43997f1864fe0891be69bc239ded98e69c9f56eb9eaa5b1789660/rfc3986-1.3.2-py2.py3-none-any.whl
Collecting hpack<4,>=2.3 (from h2==3.*->httpx==0.7.4)
  Using cached https://files.pythonhosted.org/packages/8a/cc/e53517f4a1e13f74776ca93271caef378dadec14d71c61c949d759d3db69/hpack-3.0.0-py2.py3-none-any.whl
Collecting hyperframe<6,>=5.2.0 (from h2==3.*->httpx==0.7.4)
  Using cached https://files.pythonhosted.org/packages/19/0c/bf88182bcb5dce3094e2f3e4fe20db28a9928cb7bd5b08024030e4b140db/hyperframe-5.2.0-py2.py3-none-any.whl
Installing collected packages: certifi, chardet, h11, hpack, hyperframe, h2, hstspreload, idna, rfc3986, httpx
  Running setup.py install for hstspreload ... done
  Running setup.py install for httpx ... done
Successfully installed certifi-2019.9.11 chardet-3.0.4 h11-0.8.1 h2-3.1.1 hpack-3.0.0 hstspreload-2019.9.30 httpx-0.7.4 hyperframe-5.2.0 idna-2.8 rfc3986-1.3.2
You are using pip version 19.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(venv) $ find venv -name 'py.typed'
(venv) $ ls venv/lib/python3.7/site-packages/httpx
__init__.py  __pycache__  __version__.py  api.py  client.py  concurrency  config.py  decoders.py  dispatch  exceptions.py  middleware  models.py  multipart.py  status_codes.py  utils.py

[andrewmwhite]

Update: If you check out the repo and install from the local directory (either master or 0.7.4), the file is there. It's not there if you install via from pypi pip, nor is it in the tarball that's on pypi (https://pypi.org/project/httpx/#files).

[sethmlarson]

Before we mark this as closed again I guess we'll have to test by uploading the sdist and wheel to test PyPI and then installing from there. I think all we need is a modification to MANIFEST.in? 🤷‍♂️

[andrewmwhite]

That’s what my testing indicates, but /shrug.

…

On Thu, Oct 3, 2019 at 6:25 PM Seth Michael Larson ***@***.***> wrote: Before we mark this as closed again I guess we'll have to test by uploading the sdist and wheel to test PyPI and then installing from there. I think all we need is a modification to MANIFEST.in? 🤷‍♂ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#193?email_source=notifications&email_token=AAKC45WCNM3X3LO53NR6FSTQM2LQJA5CNFSM4IJTP272YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAKCLRI#issuecomment-538191301>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKC45XIT3SFQRGF5Q3EMDLQM2LQJANCNFSM4IJTP27Q> .

[tomchristie]

Related: It's a bit of a gross hack that we need a py.typed file to indicate that we're a well typed package. I don't really dig it.

Is there anything productive that we could do to push for a nicer alternative to become adopted in whichever of Python/MyPy/PyPI this is relevant too? (Eg. A top-level __typed__ = True variable in packages? A setting in PyPI package info?)

[kylebebak]

I don't like that this file is required either (that said, I'm not very familiar with how mypy searches for types, and I'm betting there's a good reason they decided on using a marker file for packages).

py.typed was required by PEP-561, for Python 3.7.

I don't think they'll be changing this any time soon...

[florimondmanca]

I'm betting there's a good reason they decided on using a marker file for packages

I’d bet that too, but I can’t seem to find where this was discussed. PEP 561 doesn’t mention alternatives nor does it have a « Rejected ideas » section that we’d typically find in PEPs about public APIs.

I think scraping the web (or just GitHub) for pages mentioning « Add py.types » or « fix type annotations distribution » would be an interesting experiment to try and gauge how unintuitive the py.typed file is.

[florimondmanca]

Out of curiosity, I compared the usage of py.typed between mypy and pyre (an alternative Python static type checker implemented in OCaml).

It seems Pyre doesn't use py.typed at all. Instead, users must to specify a --search-path to specify extra paths where typed packages should be used, and that's how pyre looks for type annotations in external packages. For example:

# Can also be passed via a private config file.
pyre --search-path=venv/lib/python3.7/site-packages/

So instead of requiring package authors mark the package as typing-ready and have the type checker auto-discover those, Pyre switches the responsability on the user to tell Pyre where it should look for type annotations.

Proof of concept: Pyre was able to use typing data from asgi-lifespan==0.4.1, even though Mypy couldn't because only 0.4.2+ is PEP 561-compliant.

My main theory of why a special marker file was used is to increase performance. Pyre builds are very slow (30s-ish for a single-module project only with asgi-lifespan as a main dependency), and how little it has to build depends on how specific the user's --search-path is. On the other hand, mypy "just works", provided that imported packages comply with PEP 561.

In terms of productivity, we could push for something like __typed__ = True in packages' __init__.py (it sounds rather cheap to check for as well), but unfortunately that'd mean going through another round of PEP, risking a split in the ecosystem, and making mypy more complex at it'd have to support both APIs indefinitely. The upside would be an arguably simpler package type distrubution setup, but I'm not sure I'd personally want to be involved in this endeavour.

[andrewmwhite]

Just following up -- this looks to be fully resolved as of 0.7.5. Thanks!

[maartenbreddels]

This is about the only discussion I found on this topic, but it seems true that adding that file is needed for mypy.

In terms of productivity, we could push for something like __typed__ = True in packages' __init__.py (it sounds rather cheap to check for as well),

Importing packages can be quite expensive, so I guess it is about performance, feels strange indeed.