Skip to content

Commit

Permalink
Revert "Fixed format alias in Flytekit docs (#844)"
Browse files Browse the repository at this point in the history
This reverts commit 0d034c1.
  • Loading branch information
SmritiSatyanV committed Feb 23, 2022
1 parent 0d034c1 commit ac64434
Show file tree
Hide file tree
Showing 15 changed files with 100 additions and 120 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/pythonbuild.yml
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ jobs:
- name: ShellCheck
uses: ludeeus/action-shellcheck@master
with:
ignore_paths:
ignore:
boilerplate

docs:
Expand Down
2 changes: 1 addition & 1 deletion codecov.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ignore_paths:
ignore:
- "flytekit/bin"
- "test_*.py"
- "flytekit/__init__.py"
Expand Down
39 changes: 0 additions & 39 deletions docs/source/_templates/file_types.rst

This file was deleted.

16 changes: 8 additions & 8 deletions docs/source/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Please also take some time to read through the :std:ref:`design guides <design>`
📜 Quick Background
*******************

The first version of the Flytekit library was written circa 2017, before mypy typing was mainstream, and
targeted Python 2. That legacy code will be fully deprecated and removed in 2022. Since there are still
users of Flytekit that rely on that legacy API, you'll see 2 separate and distinct code paths within this repo.
The first version of the flytekit library was written circa 2017, before mypy typing was mainstream, and
targeted Python 2. That legacy code will be fully deprecated and removed in 2022 but because there are still
users of flytekit that rely on that legacy api, you'll see 2 separate and distinct code paths within this repo.
Users and contributors should ignore the legacy sections. Below is a listing of the most important packages that
comprise the new API:

Expand All @@ -25,7 +25,7 @@ comprise the new API:
- ``flytekit/extend``
This is the future home of extension points, and currently serves as the raw documentation for extensions.
- ``flytekit/extras``
This contains code that we want bundled with Flytekit but not everyone may find useful (for example AWS and GCP
This contains code that we want bundled with flytekit but not everyone may find useful (for example AWS and GCP
specific logic).
- ``flytekit/remote``
This implements the interface to interact with the Flyte service. Think of the code here as the Python-object version of Console.
Expand All @@ -39,7 +39,7 @@ comprise the new API:
- ``flytekit/bin/entrypoint.py``
The run time entrypoint for flytekit. When a task kicks off, this is where the click command goes.
- ``flytekit/clis``
This is the home for the CLIs.
This is the home for the clis.
- ``flytekit/configuration``
This holds all the configuration objects, but dependency on configuration should be carefully considered as it
makes compiled Flyte tasks and workflows less portable (i.e. if you run ``pyflyte package`` can someone else use
Expand Down Expand Up @@ -77,16 +77,16 @@ We recommend using a virtual environment to develop Flytekit. Inside the top lev
Install `shellcheck <https://github.com/koalaman/shellcheck>`__ for linting shell scripts.

.. note::
It's important to maintain separate virtualenvs for flytekit *development* and Flytekit *use*. The reason is that installing a Python
It's important to maintain separate virtualenvs for flytekit *development* and flytekit *use*. The reason is that installing a Python
library in editable mode will link it to your source code. That is, the behavior will change as you work on the code,
check out different branches, etc.

This will install Flytekit dependencies and also install Flytekit itself in editable mode. This basically links your virtual Python's ``site-packages`` with your local repo folder, allowing your local changes to take effect when the same Python interpreter runs ``import flytekit``.
This will install flytekit dependencies and also install flytekit itself in editable mode. This basically links your virtual Python's ``site-packages`` with your local repo folder, allowing your local changes to take effect when the same Python interpreter runs ``import flytekit``.

Plugin Development
==================

As discussed in the design component, Flytekit plugins currently live in this Flytekit repo, but under a different top level folder ``plugins``.
As discussed in the design component, Flytekit plugins currently live in this flytekit repo, but under a different top level folder ``plugins``.
In the future, this will be separated out into a different repo. These plugins follow a `microlib <https://medium.com/@jherreras/python-microlibs-5be9461ad979>`__ structure, which will persist even if we move repos. ::

source ~/.virtualenvs/flytekit/bin/activate
Expand Down
34 changes: 17 additions & 17 deletions docs/source/design/authoring.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,26 @@
Authoring Structure
############################

Enabling users to write tasks and workflows is the core feature of Flytekit, it is why it exists. This document goes over how some of the internals work.
Enabling users to write tasks and workflows is the core feature of flytekit, it is why it exists. This document goes over how some of the internals work.

*************
Background
*************
Please refer `design doc <https://docs.google.com/document/d/17rNKg6Uvow8CrECaPff96Tarr87P2fn4ilf_Tv2lYd4/edit#>`__.
Please see the `design doc <https://docs.google.com/document/d/17rNKg6Uvow8CrECaPff96Tarr87P2fn4ilf_Tv2lYd4/edit#>`__.

*********************
Types and Type Engine
*********************
Flyte has its own type system, which is codified `in the IDL <https://github.com/flyteorg/flyteidl>`__. Python, of course, has its own typing system, even though it's a dynamic language, and is mostly explained in `PEP 484 <https://www.python.org/dev/peps/pep-0484/>`_. In order to work properly, Flytekit needs to be able to convert between the two.
Flyte has its own type system, which is codified `in the IDL <https://github.com/flyteorg/flyteidl>`__. Python of course has its own typing system, even though it's a dynamic language, and is mostly explained in `PEP 484 <https://www.python.org/dev/peps/pep-0484/>`_. In order to work properly, flytekit needs to be able to convert between the two.

Type Engine
=============
This happens primarily through the :py:class:`flytekit.extend.TypeEngine`. This engine works by invoking a series of :py:class:`TypeTransformers <flytekit.extend.TypeTransformer>`. Each transformer is responsible for providing the functionality that the engine needs for a given native Python type.
The primary way this happens is through the :py:class:`flytekit.extend.TypeEngine`. This engine works by invoking a series of :py:class:`TypeTransformers <flytekit.extend.TypeTransformer>`. Each transformer is responsible for providing the functionality that the engine needs for a given native Python type.

*****************
Callable Entities
*****************
:ref:`Tasks <divedeep-tasks>`, :ref:`workflows <divedeep-workflows>`, and `launch plans <divedeep-launchplans>` form the core of the Flyte user experience. Each of these concepts are backed by one or more Python classes. These classes in turn, are instantiated by decorators (in the case of tasks and workflow) or a normal Python call (in the case of launch plans).
Tasks, workflows, and launch plans form the core of the Flyte user experience. Each of these concepts are backed by one or more Python classes. These classes in turn, are instantiated by decorators (in the case of tasks and workflow) or a normal Python call (in the case of launch plans).

Tasks
=====
Expand All @@ -49,7 +49,7 @@ Please see the documentation on each of the classes for details.

Workflows
=========
There are two workflow classes, and both of them inherit from the :py:class:`WorkflowBase <flytekit.core.workflow.WorkflowBase>` class.
There are two workflow classes, which both inherit from the :py:class:`WorkflowBase <flytekit.core.workflow.WorkflowBase>` class.

.. autoclass:: flytekit.core.workflow.PythonFunctionWorkflow
:noindex:
Expand All @@ -60,31 +60,31 @@ There are two workflow classes, and both of them inherit from the :py:class:`Wor

Launch Plan
===========
There is only one :py:class:`LaunchPlan <flytekit.core.launch_plan.LaunchPlan>` class.
There is also only one :py:class:`LaunchPlan <flytekit.core.launch_plan.LaunchPlan>` class.

.. autoclass:: flytekit.core.launch_plan.LaunchPlan
:noindex:

******************
Exception Handling
******************
Exception handling is done along two dimensions:
Exception handling is done along two dimensions

* System vs User: We try to differentiate between user exceptions and Flytekit/system level exceptions. For instance, if Flytekit
* System vs User: We try to differentiate between user exceptions and flytekit/system level exceptions. For instance, if flytekit
fails to upload its outputs, that's a system exception. If you the user raise a ``ValueError`` because of unexpected input
in the task code, that's a user exception.
* Recoverable vs Non-recoverable: Recoverable errors are retried and counted against your task's retry count. Non-recoverable errors just fail. System exceptions, by default, are recoverable (since there's a good chance it was just a blip).
* Recoverable vs Non-recoverable: Recoverable errors will be retried and count against your task's retry count. Non-recoverable errors will just fail. System exceptions are by default recoverable (since there's a good chance it was just a blip).

This is the user exception tree. Feel free to raise any of these exception classes. Note that the ``FlyteRecoverableException`` is the only recoverable one. All others, along with all non-Flytekit defined exceptions, are non-recoverable.
This is the user exception tree. Feel free to raise any of these exception classes. Note that the ``FlyteRecoverableException`` is the only recoverable one. All others, along with all non-flytekit defined exceptions, are non-recoverable.

.. inheritance-diagram:: flytekit.common.exceptions.user.FlyteValidationException flytekit.common.exceptions.user.FlyteEntityAlreadyExistsException flytekit.common.exceptions.user.FlyteValueException flytekit.common.exceptions.user.FlyteTimeout flytekit.common.exceptions.user.FlyteAuthenticationException flytekit.common.exceptions.user.FlyteRecoverableException
:parts: 1
:top-classes: Exception

Implementation
==============
For those who want to dig a bit deeper, take a look at the :py:class:`flytekit.common.exceptions.scopes.FlyteScopedException` classes.
There are two decorators which you'll find interspersed throughout the codebase.
For those that want to dig a bit deeper, take a look at the :py:class:`flytekit.common.exceptions.scopes.FlyteScopedException` classes.
There are also two decorators which you'll find interspersed throughout the codebase.

.. autofunction:: flytekit.common.exceptions.scopes.system_entry_point

Expand All @@ -93,8 +93,8 @@ There are two decorators which you'll find interspersed throughout the codebase.
**************
Call Patterns
**************
The three entities above are all callable. In Flyte terms, this means they can be invoked to yield a unit (or units) of work.
In Python terms, this means you can add ``()`` to the end of one of it which invokes the ``__call__`` method on the object.
The three entities above are all callable. In Flyte terms that means they can be invoked to yield a unit (or units) of work.
In Python terms that means you can add ``()`` to the end of one of it which invokes the ``__call__`` method on the object.

What happens when a callable entity is called depends on the current context, specifically the current :py:class:`flytekit.FlyteContext`

Expand Down Expand Up @@ -133,11 +133,11 @@ When the next task is called, logic is triggered to unwrap these promises.

Compilation
===========
When a workflow is compiled, instead of producing ``Promise`` objects that wrap literal values, they wrap a :py:class:`flytekit.core.promise.NodeOutput` instead. This is how data dependency is tracked between tasks.
When a workflow is compiled, instead of producing promise objects that wrap literal values, they wrap a :py:class:`flytekit.core.promise.NodeOutput` instead. This is how data dependency is tracked between tasks.

Branch Skip
===========
If it's been determined that a conditional is not true, then Flytekit will skip calling the task. This way, any side-effects in the task logic will not be run.
If it's been determined that a conditional is not true, then flytekit will skip actually calling the task which means that any side-effects in the task logic will not be run.


.. note::
Expand Down
2 changes: 1 addition & 1 deletion docs/source/design/clis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Command Line Interfaces

Flyte CLI
===========
``flyte-cli`` is the general CLI that can be used to talk to the Flyte control plane (Flyte Admin). It ships with Flytekit as part of the Pypi package. Think of this as the ``kubectl`` for Flyte. In fact, we're working on ``flytectl`` which is under active development the completion of which will deprecate this CLI.
``flyte-cli`` is the general CLI that can be used to talk to the Flyte control plane (Flyte Admin). It ships with flytekit as part of the Pypi package. Think of this as the ``kubectl`` for Flyte. In fact, we're working on ``flytectl`` which is under active development the completion of which will deprecate this CLI.

Think of this CLI as a network-aware (i.e. can talk to Admin) but not code-aware (doesn't need to have user code checked out) CLI. In the registration flow, this CLI is responsible for shipping the compiled Protobuf files off to Flyte Admin.

Expand Down
14 changes: 7 additions & 7 deletions docs/source/design/control_plane.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ The video below features a demo of FlyteRemote, followed by all relevant informa
Create a FlyteRemote Object
***************************

The :class:`~flytekit.remote.remote.FlyteRemote` class is the entrypoint for programmatically performing operations in a Python
runtime. There are two ways of creating a remote object:
The :class:`~flytekit.remote.remote.FlyteRemote` class is the entrypoint for programmatically performing operations in a python
runtime. There are two ways of creating a remote object.

**Initialize directly**

Expand All @@ -35,7 +35,7 @@ runtime. There are two ways of creating a remote object:
insecure=True,
)
**Initialize from Flyte config**
**Initialize from flyte config**

.. TODO: link documentation to flyte config and environment variables
Expand All @@ -60,8 +60,8 @@ Fetching Flyte Admin Entities
Executing Entities
******************

You can execute all of these Flyte entities, which returns a :class:`~flytekit.remote.workflow_execution.FlyteWorkflowExecution` object.
For more information on Flyte entities, see the See the :ref:`remote flyte entities <remote-flyte-execution-objects>`
You can execute all of these flyte entities, which returns a :class:`~flytekit.remote.workflow_execution.FlyteWorkflowExecution` object.
For more information on flyte entities, see the See the :ref:`remote flyte entities <remote-flyte-execution-objects>`
reference.

.. code-block:: python
Expand Down Expand Up @@ -89,7 +89,7 @@ You can also pass in ``wait=True`` to the :meth:`~flytekit.remote.remote.FlyteRe
Syncing Remote State
********************

Use the :meth:`~flytekit.remote.remote.FlyteRemote.sync` method to sync the entity object's state with the remote state:
Use the :meth:`~flytekit.remote.remote.FlyteRemote.sync` method to sync the entity object's state with the remote state

.. code-block:: python
Expand All @@ -100,6 +100,6 @@ Use the :meth:`~flytekit.remote.remote.FlyteRemote.sync` method to sync the enti
Inspecting Execution Objects
****************************

At any time, you can inspect the inputs, outputs, completion status, error status, and other aspects of a workflow
At any time you can inspect the inputs, outputs, completion status, error status, and other aspects of a workflow
execution object. See the :ref:`remote execution objects <remote-flyte-execution-objects>` reference for a list
of all the available attributes.
10 changes: 5 additions & 5 deletions docs/source/design/execution.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,18 @@
#######################
Execution Time Support
#######################
Most of the tasks that are written in Flytekit will be Python functions decorated with ``@task`` which turns the body of the function into a Flyte task, capable of being run independently, or included in any number of workflows. The interaction between Flytekit and these tasks do not end once they have been serialized and registered onto the Flyte control plane however. When compiled, the command that will be executed when the task is run is hardcoded into the task definition itself.
Most of the tasks that are written in flytekit will be Python functions decorated with ``@task`` which turns the body of the function into a Flyte Task, capable of being run independently, or included in any number of workflows. The interaction between flytekit and these tasks do not end once they have been serialized and registered onto the Flyte control plane however. When compiled, the command that will be executed when the Task is run is hardcoded into the task definition itself.

In the basic ``@task`` decorated function scenario, the command to be run will be something containing ``pyflyte-execute``, which is one of the CLIs discussed in that section.

That command, if you were to inspect a serialized task, might look something like ::

flytekit_venv pyflyte-execute --task-module app.workflows.failing_workflows --task-name divider --inputs {{.input}} --output-prefix {{.outputPrefix}} --raw-output-data-prefix {{.rawOutputDataPrefix}}

The point of running this script, or rather the reason for having any Flyte-related logic at execution time, is purely to codify and streamline the interaction between Flyte the platform, and the function body comprising user code. The Flyte CLI is responsible for:
The point of running this script, or rather the reason for having any Flyte-related logic at execution time, is purely to codify and streamline the interaction between Flyte the platform, and the function body comprising user code. That is, the Flyte CLI is responsible for

* I/O: The templated ``--inputs`` and ``--output-prefix`` arguments in the example command above will be filled in by the Flyte execution engine with S3 path (in the case of an AWS deployment). The ``pyflyte`` script will download the inputs to the right location in the container, and upload the results to the ``output-prefix`` location.
* I/O. The templated ``--inputs`` and ``--output-prefix`` arguments in the example command above will be filled in by the Flyte execution engine with S3 path (in the case of an AWS deployment). The ``pyflyte`` script will download the inputs to the right location in the container, and upload the results to the ``output-prefix`` location.
* Ensure that raw output data prefix configuration option, which is again filled in by the Flyte engine, is respected so that ``FlyteFile``, ``FlyteDirectory``, and ``FlyteSchema`` objects offload their data to the correct place.
* Capture and handle error reporting: Exceptions thrown in the course of task execution are captured and uploaded to the Flyte control plane for display on the Console.
* Set up helper utilities like the ``statsd`` handle, logging and logging levels, etc.
* Capture and handle error reporting. Exceptions thrown in the course of task execution are captured and uploaded to the Flyte control plane for display in the Console.
* Set up of helper utilities like the ``statsd`` handle, logging and logging levels, etc.
* Ensure configuration options about the Flyte backend, which are passed through by the Flyte engine, are properly loaded in Python memory.
Loading

0 comments on commit ac64434

Please sign in to comment.