Skip to content

Commit be2ecd2

Browse files
richardvonmoosPseudomanifold
authored andcommitted
Created Wiki page with previous Github wiki pages and links. Added wiki entry on how to create new entries.
1 parent afd0c2f commit be2ecd2

20 files changed

+706
-0
lines changed

config.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ ignoreFiles = ['publications.md$']
4747
pageRef = "/contact"
4848
weight = -70
4949

50+
[[menu.main]]
51+
name = "Wiki"
52+
url = "/wiki/"
53+
weight = -70
54+
5055
# Ditto (see above), but for the footer.
5156
[params.footer.logos]
5257
text = """

content/wiki/Adding-wiki-entries.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
title: Adding wiki entries
3+
---
4+
### Adding new sections
5+
6+
To add a new section of wiki entries, add a new folder <topic name> to content/wiki/. Then add a file to the folder
7+
with the name _index.md and add the lines:
8+
```
9+
---
10+
title: <Topic name>
11+
---
12+
```
13+
to the top of the file. This will create a new section in the wiki with the title <Topic name>.
14+
15+
### Adding a new entry
16+
17+
To add a wiki entry, drop a markdown file into content/wiki/<topic-name>. Add the lines
18+
```
19+
---
20+
title: <Content title>
21+
---
22+
```
23+
To the top of the file. Now you can fill the markdown file with content.
24+
25+
### Using figures
26+
27+
If you are referencing figures in your markdown file, add a folder <content-title> to static/wiki/.
28+
Then, add your figures to the folder static/wiki/<content-title> that you just created.
29+
You can then reference these figures in your markdown file with the following command:
30+
31+
```HTML
32+
<img src="/wiki/<content-title>/<figure-x>"
33+
alt="<figure x> not displayed correctly"
34+
style="width: 100%; height: auto;">
35+
```
36+
37+
### Adding external links
38+
39+
Add a file <external-link-name>.md to content/wiki/external-guides/ with the following content:
40+
```
41+
---
42+
title: <Content title>
43+
external_url: "<url>"
44+
---
45+
```
46+
.
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
title: Computing-Infrastructure:-Tips-and-Tricks
3+
---
4+
(**Draft**)
5+
6+
## Virtual Environments
7+
8+
Whether you are working _locally_ or on a server, it is a good idea to use virtual environments
9+
for your Python projects.
10+
11+
### Rationale
12+
13+
My personal suggestion for using the cluster involves setting up `Miniconda` with a basic
14+
environment, which you can use to maintain compilers and support libraries. Nested within
15+
this environment, we use [`poetry`](https://python-poetry.org/), a great way for managing
16+
and maintaining virtual environments with Python.
17+
18+
#### Advantages
19+
20+
- In many projects, this setup provides a 'fire-and-forget' type solution that you may use
21+
in other cluster environments as well.
22+
- Dependencies between packages are respected and reproducible builds may be produced.
23+
- `poetry` makes _publishing_ a proper Python package almost trivial.
24+
25+
#### Drawbacks
26+
27+
The integration with `pytorch` and other GPU-based packages is not optimal. However, the
28+
environment set up by `poetry` is a fully-fledged virtual environment. We can always use
29+
`pip` to install specific GPU-based packages here. While this is as little more complex,
30+
we still benefit from having different virtual environments for each project, which will
31+
prevent interference.
32+
33+
#### Alternatives
34+
35+
You can also set up and maintain virtual environments yourself, installing all packages
36+
via `pip`. In more recent versions of `pip`, the `pyproject.toml` used by `poetry` will
37+
be parsed as well.
38+
39+
### Setting up `Miniconda`
40+
41+
```bash
42+
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
43+
# Use a local temporary directory to ensure that this can be installed in
44+
# all cases. This is probably not necessary in your case.
45+
$ mkdir tmp
46+
$ TMPDIR=$HOME/tmp bash Miniconda3-latest-Linux-x86_64.sh
47+
$ bash Miniconda3-latest-Linux-x86_64.sh
48+
# Follow the installer afterwards. My personal recommendation is to
49+
# install Miniconda in $HOME/.miniconda since it will then not show
50+
# up all the time when you issue an ls command.
51+
#
52+
# After the installation, log out and in again. Start bash if it is
53+
# not already your default shell.
54+
$ conda install -c conda-forge poetry
55+
```
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
---
2+
title: Loading configs in Python
3+
---
4+
5+
## Motivation
6+
- The loading nested configurations can be difficult.
7+
- Pythons yaml loader does not deal with it nicely.
8+
- loading as dict -> key access, dot access would be nicer.
9+
- many dependencies otherwise (Hydra, OmegaConf, ...)
10+
11+
12+
Simple solution for loading.
13+
- No external dependencies.
14+
- Allows dot access.
15+
- Allows the use of `safe_load`.
16+
- `yaml.load` poses security risks -> Loads arbitrary code.
17+
18+
19+
Possible extensions:
20+
- Add Pydantic for convenient validation of configs.
21+
- Add typing.
22+
23+
```python
24+
import yaml
25+
import json
26+
from types import SimpleNamespace
27+
28+
def load_object(dct):
29+
    return SimpleNamespace(**dct)
30+
31+
def load_config(path):
32+
    """
33+
    Loads the configuration yaml and parses it into an object with dot access.
34+
    """
35+
    with open(path, encoding="utf-8") as stream:
36+
        # Load dict
37+
        config_dict = yaml.safe_load(stream)
38+
39+
        # Convert to namespace (access via config.data etc)
40+
        config = json.loads(json.dumps(config_dict), object_hook=load_object)
41+
    return config, config_dict
42+
```
43+
44+
45+
For typing one can use the following construction:
46+
47+
```python
48+
from dataclasses import dataclass
49+
50+
@dataclass
51+
class MyModelConfig:
52+
module: models.mymodel
53+
hidden_size: int
54+
55+
56+
57+
class MyModel:
58+
def __init__(self, config: MyModelConfig):
59+
# Model def goes here.
60+
# Typing allows autocomplete.
61+
pass
62+
63+
...
64+
65+
```
66+
67+
68+
# References in yaml
69+
70+
- Yaml allows for references
71+
- Avoids duplicates.
72+
- Great when configs need to be passed to multiple modules
73+
- Data and model for instance.
74+
75+
```yaml
76+
experimentconfig: &id-exp
77+
name: MyCoolExperiment
78+
79+
model:
80+
module: models.mymodel
81+
hidden_size: 10
82+
expconfig: *id-exp
83+
84+
data:
85+
module: data.mydata
86+
root: ./data
87+
expconfig: *id-exp
88+
```
89+
90+
91+
92+
Saving configs
93+
- tbd
94+
95+

content/wiki/Code/_index.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
------
2+
title: Code
3+
---

0 commit comments

Comments
 (0)