Why does the Relation Extraction with own ColumnCorpus achieves bad results?

[geheim01]

Hello,

We want to use Flair to extract relations in sentences. Since we want to use our own German corpus for this, we created it following the comment in #2726 in the ColumnCorpus format. We store the relations in column format as in this example:

# global.columns = id form ner
# text = Anna Kreuzer wuchs zusammen mit drei Brüdern in ärmlichen Verhältnissen bei ihren Eltern auf , die ab 1914 in Kiel lebten .
# relations = 1;2;20;20;wohnt_in
1 Anna PER
2 Kreuzer PER
3 wuchs O
4 zusammen O
5 mit O
6 drei O
7 Brüdern O
8 in O
9 ärmlichen O
10 Verhältnissen O
11 bei O
12 ihren O
13 Eltern O
14 auf O
15 , O
16 die O
17 ab O
18 1914 O
19 in O
20 Kiel LOC
21 lebten O
22 . O

# global.columns = id form ner
# text = Dort besuchte sie die Volksschule .
# relations = 3;3;1;1;Ausbildung_in
1 Dort Coref_LOC
2 besuchte O
3 sie Coref_PER
4 die O
5 Volksschule O
6 . O

and the code to train it is analogous to the tutorial thus:

import torch
import flair
from flair.data import Sentence
from flair.datasets import RE_ENGLISH_CONLL04
from flair.embeddings import TransformerWordEmbeddings
from flair.models import RelationExtractor
from flair.trainers import ModelTrainer
from flair.data import Corpus
from flair.datasets import ColumnCorpus
from pathlib import Path

flair.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

out_dir: Path = Path("conll04")


def train(
    transformer: str = "distilbert-base-uncased",#bert-base-german-cased", # distilbert-base-uncased
    max_epochs: int = 100,
    learning_rate: float = 5e-5,
    mini_batch_size: int = 32,
    seed: int = 42,
) -> None:
    flair.set_seed(seed)

    # Step 1: Create the training data
    # The relation extractor is *not* trained end-to-end.
    # A corpus for training the relation extractor requires annotated entities and relations.
    columns = {0:'id', 1: 'form', 2: 'ner'}

    data_folder = "/content"

    # initializing the corpus
    corpus: Corpus = ColumnCorpus(data_folder, columns,
                                  train_file="train_clean.txt")

    # Step 2: Make the label dictionary from the corpus
    label_dictionary = corpus.make_label_dictionary("relation")
    label_dictionary.add_item("O")

    # Step 3: Initialize fine-tunable transformer embeddings
    embeddings = TransformerWordEmbeddings(model=transformer, layers="-1", fine_tune=True)

    # Step 4: Initialize relation classifier
    model: RelationExtractor = RelationExtractor(
        embeddings=embeddings,
        label_dictionary=label_dictionary,
        label_type="relation",
        entity_label_type="ner",
        entity_pair_filters=[  # Define valid entity pair combinations, used as relation candidates
          ("LOC", "LOC"),
          ("PER", "LOC"),
          ("PER", "ORG"),
          ("ORG", "LOC"),
          ("PER", "PER"),
          ("LOC", "LOC")
        ],
    )

    # Step 5: Initialize trainer
    trainer: ModelTrainer = ModelTrainer(model, corpus)

    # Step 6: Run fine-tuning
    trainer.fine_tune(
        out_dir,
        max_epochs=max_epochs,
        learning_rate=learning_rate,
        mini_batch_size=mini_batch_size,
        main_evaluation_metric=("macro avg", "f1-score"),
    )

However, we do not understand why our Loss Rate is constantly at 0. And the results (Precision, Recall) for all relations at 0.

y class:
                     precision    recall  f1-score   support

           liegt_in     0.0000    0.0000    0.0000       6.0
      inhaftiert_in     0.0000    0.0000    0.0000       5.0
       Mitglied_bei     0.0000    0.0000    0.0000       5.0
Führungsposition_in     0.0000    0.0000    0.0000       4.0
           wohnt_in     0.0000    0.0000    0.0000       4.0
      Aufenthalt_in     0.0000    0.0000    0.0000       3.0
       arbeitet_bei     0.0000    0.0000    0.0000       2.0
     Teilgruppe_von     0.0000    0.0000    0.0000       2.0
            Ehe_mit     0.0000    0.0000    0.0000       2.0
         geboren_in     0.0000    0.0000    0.0000       2.0
       gestorben_in     0.0000    0.0000    0.0000       2.0
        arbeitet_in     0.0000    0.0000    0.0000       2.0
        ermordet_in     0.0000    0.0000    0.0000       2.0
         Gegner_von     0.0000    0.0000    0.0000       2.0
          liegt_bei     0.0000    0.0000    0.0000       1.0
           tätig_in     0.0000    0.0000    0.0000       1.0
           Kurzform     0.0000    0.0000    0.0000       1.0
           Standort     0.0000    0.0000    0.0000       1.0
         neuer_Name     0.0000    0.0000    0.0000       1.0
        Geschwister     0.0000    0.0000    0.0000       1.0
 ausgeschlossen_von     0.0000    0.0000    0.0000       1.0
    Gruppenmitglied     0.0000    0.0000    0.0000       1.0
         Kontakt_zu     0.0000    0.0000    0.0000       1.0
       verfolgt_von     0.0000    0.0000    0.0000       1.0

          micro avg     0.0000    0.0000    0.0000      53.0
          macro avg     0.0000    0.0000    0.0000      53.0
       weighted avg     0.0000    0.0000    0.0000      53.0

Are we currently preparing our ColumnCorpus incorrectly?
Do you have any other tips we should pay attention to?

[alanakbik]

Hello @geheim01 a few thoughts:

• distilbert-base-uncased is for English only, so you need to use a transformer that can handle German text
• if the learning rate is too high, fine-tuning will learn nothing. Depending on the transformer model you should try out at least 5e-5 and 5e-6
• have you printed the label_dictionary? does it look ok?
• can you share the training log?

[geheim01]

• We have already tried bert-base-german-cased instead of distilbert-base-uncased with the same results.

• Label_dictionary looks fine consisting of the relations we have annotated but printing one sentence from corpus.train[0] looks kinda strange: Example Sentence: "1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18" → ["1"/Ende, "2"/1932, "3"/war, "4"/sie/Coref_PER, "4 -> 9"/arbeitet_bei, "5"/an, "6"/der, "7"/Herstellung, "8"/der, "9"/Arbeiterwelt/ORG, "9 -> 17"/Teilgruppe_von, "10"/beteiligt, "11"/,, "12"/der, "13"/illegalen, "14"/Zeitung, "15"/der, "16"/Kieler, "17"/KPD/ORG, "18"/.]

• This is our training log with the adjusted learning rate and bert-base-german-cased model:

2022-08-02 15:44:44,152 Reading data from /content
2022-08-02 15:44:44,157 Train: /content/train_new delete_empty_rel.txt
2022-08-02 15:44:44,160 Dev: None
2022-08-02 15:44:44,163 Test: None
2022-08-02 15:44:45,355 Computing label dictionary. Progress:

745it [00:00, 28984.19it/s]

2022-08-02 15:44:45,395 Dictionary created for label 'relation' with 40 values: Gruppenmitglied (seen 65 times), Mitglied_bei (seen 43 times), Aufenthalt_in (seen 32 times), Bekanntschaft (seen 31 times), arbeitet_in (seen 30 times), liegt_in (seen 27 times), Führungsposition_in (seen 24 times), Gegner_von (seen 24 times), Kurzform (seen 21 times), wohnt_in (seen 18 times), inhaftiert_in (seen 17 times), verfolgt_von (seen 16 times), arbeitet_bei (seen 15 times), tätig_in (seen 13 times), Kontakt_zu (seen 12 times), Ehe_mit (seen 11 times), Unterstützung_von (seen 11 times), neuer_Name (seen 10 times), geboren_in (seen 9 times), gestorben_in (seen 9 times)


Downloading tokenizer_config.json: 100%
29.0/29.0 [00:00<00:00, 638B/s]
Downloading config.json: 100%
433/433 [00:00<00:00, 9.65kB/s]
Downloading vocab.txt: 100%
249k/249k [00:00<00:00, 4.00MB/s]
Downloading tokenizer.json: 100%
474k/474k [00:00<00:00, 7.11MB/s]
Downloading pytorch_model.bin: 100%
419M/419M [00:10<00:00, 45.8MB/s]

2022-08-02 15:44:58,516 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,521 Model: "RelationExtractor(
  (decoder): Linear(in_features=3072, out_features=41, bias=True)
  (dropout): Dropout(p=0.0, inplace=False)
  (locked_dropout): LockedDropout(p=0.0)
  (word_dropout): WordDropout(p=0.0)
  (loss_function): CrossEntropyLoss()
  (embeddings): TransformerWordEmbeddings(
    (model): BertModel(
      (embeddings): BertEmbeddings(
        (word_embeddings): Embedding(30000, 768, padding_idx=0)
        (position_embeddings): Embedding(512, 768)
        (token_type_embeddings): Embedding(2, 768)
        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        (dropout): Dropout(p=0.1, inplace=False)
      )
      (encoder): BertEncoder(
        (layer): ModuleList(
          (0): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (1): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (2): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (3): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (4): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (5): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (6): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (7): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (8): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (9): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (10): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (11): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
              (intermediate_act_fn): GELUActivation()
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
        )
      )
      (pooler): BertPooler(
        (dense): Linear(in_features=768, out_features=768, bias=True)
        (activation): Tanh()
      )
    )
  )
  (weights): None
  (weight_tensor) None
)"
2022-08-02 15:44:58,525 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,528 Corpus: "Corpus: 745 train + 83 dev + 92 test sentences"
2022-08-02 15:44:58,541 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,543 Parameters:
2022-08-02 15:44:58,545  - learning_rate: "0.000005"
2022-08-02 15:44:58,547  - mini_batch_size: "32"
2022-08-02 15:44:58,549  - patience: "3"
2022-08-02 15:44:58,552  - anneal_factor: "0.5"
2022-08-02 15:44:58,553  - max_epochs: "10"
2022-08-02 15:44:58,555  - shuffle: "True"
2022-08-02 15:44:58,557  - train_with_dev: "False"
2022-08-02 15:44:58,560  - batch_growth_annealing: "False"
2022-08-02 15:44:58,562 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,564 Model training base path: "conll04"
2022-08-02 15:44:58,566 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,568 Device: cpu
2022-08-02 15:44:58,570 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,572 Embeddings storage mode: none
2022-08-02 15:44:58,575 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,612 epoch 1 - iter 2/24 - loss 0.00000000 - samples/sec: 2436.78 - lr: 0.000000
2022-08-02 15:44:58,635 epoch 1 - iter 4/24 - loss 0.00000000 - samples/sec: 3152.69 - lr: 0.000001
2022-08-02 15:44:58,648 epoch 1 - iter 6/24 - loss 0.00000000 - samples/sec: 6105.94 - lr: 0.000001
2022-08-02 15:44:58,660 epoch 1 - iter 8/24 - loss 0.00000000 - samples/sec: 6321.19 - lr: 0.000002
2022-08-02 15:44:58,674 epoch 1 - iter 10/24 - loss 0.00000000 - samples/sec: 8418.07 - lr: 0.000002
2022-08-02 15:44:58,688 epoch 1 - iter 12/24 - loss 0.00000000 - samples/sec: 5418.67 - lr: 0.000003
2022-08-02 15:44:58,703 epoch 1 - iter 14/24 - loss 0.00000000 - samples/sec: 5434.14 - lr: 0.000003
2022-08-02 15:44:58,716 epoch 1 - iter 16/24 - loss 0.00000000 - samples/sec: 6097.48 - lr: 0.000003
2022-08-02 15:44:58,730 epoch 1 - iter 18/24 - loss 0.00000000 - samples/sec: 9664.99 - lr: 0.000004
2022-08-02 15:44:58,742 epoch 1 - iter 20/24 - loss 0.00000000 - samples/sec: 8975.67 - lr: 0.000004
2022-08-02 15:44:58,756 epoch 1 - iter 22/24 - loss 0.00000000 - samples/sec: 5221.16 - lr: 0.000005
2022-08-02 15:44:58,769 epoch 1 - iter 24/24 - loss 0.00000000 - samples/sec: 6305.59 - lr: 0.000005
2022-08-02 15:44:58,773 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,776 EPOCH 1 done: loss 0.0000 - lr 0.000005

100%|██████████| 3/3 [00:00<00:00, 133.96it/s]

2022-08-02 15:44:58,858 Evaluating as a multi-label problem: True
2022-08-02 15:44:58,884 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:44:58,894 BAD EPOCHS (no improvement): 4
2022-08-02 15:44:58,896 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:58,916 epoch 2 - iter 2/24 - loss 0.00000000 - samples/sec: 8031.46 - lr: 0.000005
2022-08-02 15:44:58,929 epoch 2 - iter 4/24 - loss 0.00000000 - samples/sec: 6845.57 - lr: 0.000005
2022-08-02 15:44:58,942 epoch 2 - iter 6/24 - loss 0.00000000 - samples/sec: 7789.09 - lr: 0.000005
2022-08-02 15:44:58,953 epoch 2 - iter 8/24 - loss 0.00000000 - samples/sec: 8656.97 - lr: 0.000005
2022-08-02 15:44:58,965 epoch 2 - iter 10/24 - loss 0.00000000 - samples/sec: 7167.26 - lr: 0.000005
2022-08-02 15:44:58,978 epoch 2 - iter 12/24 - loss 0.00000000 - samples/sec: 5802.88 - lr: 0.000005


2022-08-02 15:44:58,995 epoch 2 - iter 14/24 - loss 0.00000000 - samples/sec: 6918.97 - lr: 0.000005
2022-08-02 15:44:59,007 epoch 2 - iter 16/24 - loss 0.00000000 - samples/sec: 8371.08 - lr: 0.000005
2022-08-02 15:44:59,019 epoch 2 - iter 18/24 - loss 0.00000000 - samples/sec: 8919.90 - lr: 0.000005
2022-08-02 15:44:59,030 epoch 2 - iter 20/24 - loss 0.00000000 - samples/sec: 9930.65 - lr: 0.000005
2022-08-02 15:44:59,041 epoch 2 - iter 22/24 - loss 0.00000000 - samples/sec: 8267.69 - lr: 0.000004
2022-08-02 15:44:59,053 epoch 2 - iter 24/24 - loss 0.00000000 - samples/sec: 7192.42 - lr: 0.000004
2022-08-02 15:44:59,060 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,062 EPOCH 2 done: loss 0.0000 - lr 0.000004

100%|██████████| 3/3 [00:00<00:00, 203.03it/s]

2022-08-02 15:44:59,088 Evaluating as a multi-label problem: True
2022-08-02 15:44:59,118 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:44:59,126 BAD EPOCHS (no improvement): 4
2022-08-02 15:44:59,129 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,151 epoch 3 - iter 2/24 - loss 0.00000000 - samples/sec: 5020.30 - lr: 0.000004
2022-08-02 15:44:59,168 epoch 3 - iter 4/24 - loss 0.00000000 - samples/sec: 5773.43 - lr: 0.000004
2022-08-02 15:44:59,189 epoch 3 - iter 6/24 - loss 0.00000000 - samples/sec: 3604.95 - lr: 0.000004
2022-08-02 15:44:59,203 epoch 3 - iter 8/24 - loss 0.00000000 - samples/sec: 7950.11 - lr: 0.000004
2022-08-02 15:44:59,214 epoch 3 - iter 10/24 - loss 0.00000000 - samples/sec: 8426.26 - lr: 0.000004
2022-08-02 15:44:59,227 epoch 3 - iter 12/24 - loss 0.00000000 - samples/sec: 8601.77 - lr: 0.000004
2022-08-02 15:44:59,237 epoch 3 - iter 14/24 - loss 0.00000000 - samples/sec: 8546.18 - lr: 0.000004
2022-08-02 15:44:59,250 epoch 3 - iter 16/24 - loss 0.00000000 - samples/sec: 6008.36 - lr: 0.000004
2022-08-02 15:44:59,262 epoch 3 - iter 18/24 - loss 0.00000000 - samples/sec: 7436.71 - lr: 0.000004


2022-08-02 15:44:59,274 epoch 3 - iter 20/24 - loss 0.00000000 - samples/sec: 6677.33 - lr: 0.000004
2022-08-02 15:44:59,285 epoch 3 - iter 22/24 - loss 0.00000000 - samples/sec: 8822.86 - lr: 0.000004
2022-08-02 15:44:59,298 epoch 3 - iter 24/24 - loss 0.00000000 - samples/sec: 7183.76 - lr: 0.000004
2022-08-02 15:44:59,304 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,306 EPOCH 3 done: loss 0.0000 - lr 0.000004

100%|██████████| 3/3 [00:00<00:00, 151.99it/s]

2022-08-02 15:44:59,339 Evaluating as a multi-label problem: True
2022-08-02 15:44:59,367 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:44:59,376 BAD EPOCHS (no improvement): 4
2022-08-02 15:44:59,379 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,394 epoch 4 - iter 2/24 - loss 0.00000000 - samples/sec: 7554.11 - lr: 0.000004
2022-08-02 15:44:59,405 epoch 4 - iter 4/24 - loss 0.00000000 - samples/sec: 7329.30 - lr: 0.000004
2022-08-02 15:44:59,417 epoch 4 - iter 6/24 - loss 0.00000000 - samples/sec: 7194.54 - lr: 0.000004
2022-08-02 15:44:59,430 epoch 4 - iter 8/24 - loss 0.00000000 - samples/sec: 6463.34 - lr: 0.000004
2022-08-02 15:44:59,444 epoch 4 - iter 10/24 - loss 0.00000000 - samples/sec: 6795.32 - lr: 0.000004
2022-08-02 15:44:59,462 epoch 4 - iter 12/24 - loss 0.00000000 - samples/sec: 4303.09 - lr: 0.000004
2022-08-02 15:44:59,477 epoch 4 - iter 14/24 - loss 0.00000000 - samples/sec: 7280.40 - lr: 0.000004
2022-08-02 15:44:59,491 epoch 4 - iter 16/24 - loss 0.00000000 - samples/sec: 5780.26 - lr: 0.000004
2022-08-02 15:44:59,503 epoch 4 - iter 18/24 - loss 0.00000000 - samples/sec: 7715.21 - lr: 0.000003


2022-08-02 15:44:59,518 epoch 4 - iter 20/24 - loss 0.00000000 - samples/sec: 6675.01 - lr: 0.000003
2022-08-02 15:44:59,538 epoch 4 - iter 22/24 - loss 0.00000000 - samples/sec: 6610.90 - lr: 0.000003
2022-08-02 15:44:59,552 epoch 4 - iter 24/24 - loss 0.00000000 - samples/sec: 9762.35 - lr: 0.000003
2022-08-02 15:44:59,557 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,561 EPOCH 4 done: loss 0.0000 - lr 0.000003

100%|██████████| 3/3 [00:00<00:00, 177.40it/s]

2022-08-02 15:44:59,590 Evaluating as a multi-label problem: True
2022-08-02 15:44:59,623 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:44:59,631 BAD EPOCHS (no improvement): 4
2022-08-02 15:44:59,634 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,658 epoch 5 - iter 2/24 - loss 0.00000000 - samples/sec: 4636.03 - lr: 0.000003
2022-08-02 15:44:59,674 epoch 5 - iter 4/24 - loss 0.00000000 - samples/sec: 5851.45 - lr: 0.000003
2022-08-02 15:44:59,690 epoch 5 - iter 6/24 - loss 0.00000000 - samples/sec: 6999.26 - lr: 0.000003
2022-08-02 15:44:59,703 epoch 5 - iter 8/24 - loss 0.00000000 - samples/sec: 9025.47 - lr: 0.000003
2022-08-02 15:44:59,714 epoch 5 - iter 10/24 - loss 0.00000000 - samples/sec: 9485.02 - lr: 0.000003
2022-08-02 15:44:59,730 epoch 5 - iter 12/24 - loss 0.00000000 - samples/sec: 9124.87 - lr: 0.000003
2022-08-02 15:44:59,744 epoch 5 - iter 14/24 - loss 0.00000000 - samples/sec: 5456.45 - lr: 0.000003
2022-08-02 15:44:59,758 epoch 5 - iter 16/24 - loss 0.00000000 - samples/sec: 5944.89 - lr: 0.000003


2022-08-02 15:44:59,772 epoch 5 - iter 18/24 - loss 0.00000000 - samples/sec: 5271.92 - lr: 0.000003
2022-08-02 15:44:59,789 epoch 5 - iter 20/24 - loss 0.00000000 - samples/sec: 5745.87 - lr: 0.000003
2022-08-02 15:44:59,801 epoch 5 - iter 22/24 - loss 0.00000000 - samples/sec: 6588.67 - lr: 0.000003
2022-08-02 15:44:59,814 epoch 5 - iter 24/24 - loss 0.00000000 - samples/sec: 7846.24 - lr: 0.000003
2022-08-02 15:44:59,818 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,821 EPOCH 5 done: loss 0.0000 - lr 0.000003

100%|██████████| 3/3 [00:00<00:00, 163.14it/s]

2022-08-02 15:44:59,851 Evaluating as a multi-label problem: True
2022-08-02 15:44:59,887 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:44:59,895 BAD EPOCHS (no improvement): 4
2022-08-02 15:44:59,898 ----------------------------------------------------------------------------------------------------
2022-08-02 15:44:59,919 epoch 6 - iter 2/24 - loss 0.00000000 - samples/sec: 4690.55 - lr: 0.000003
2022-08-02 15:44:59,933 epoch 6 - iter 4/24 - loss 0.00000000 - samples/sec: 5510.32 - lr: 0.000003
2022-08-02 15:44:59,946 epoch 6 - iter 6/24 - loss 0.00000000 - samples/sec: 6193.85 - lr: 0.000003
2022-08-02 15:44:59,958 epoch 6 - iter 8/24 - loss 0.00000000 - samples/sec: 9040.97 - lr: 0.000003
2022-08-02 15:44:59,969 epoch 6 - iter 10/24 - loss 0.00000000 - samples/sec: 9276.23 - lr: 0.000003
2022-08-02 15:44:59,982 epoch 6 - iter 12/24 - loss 0.00000000 - samples/sec: 6670.03 - lr: 0.000003
2022-08-02 15:44:59,994 epoch 6 - iter 14/24 - loss 0.00000000 - samples/sec: 7562.84 - lr: 0.000002
2022-08-02 15:45:00,007 epoch 6 - iter 16/24 - loss 0.00000000 - samples/sec: 6116.79 - lr: 0.000002
2022-08-02 15:45:00,019 epoch 6 - iter 18/24 - loss 0.00000000 - samples/sec: 8211.30 - lr: 0.000002


2022-08-02 15:45:00,031 epoch 6 - iter 20/24 - loss 0.00000000 - samples/sec: 6116.65 - lr: 0.000002
2022-08-02 15:45:00,052 epoch 6 - iter 22/24 - loss 0.00000000 - samples/sec: 5474.81 - lr: 0.000002
2022-08-02 15:45:00,065 epoch 6 - iter 24/24 - loss 0.00000000 - samples/sec: 7560.28 - lr: 0.000002
2022-08-02 15:45:00,069 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,071 EPOCH 6 done: loss 0.0000 - lr 0.000002

100%|██████████| 3/3 [00:00<00:00, 184.33it/s]

2022-08-02 15:45:00,102 Evaluating as a multi-label problem: True
2022-08-02 15:45:00,138 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:45:00,149 BAD EPOCHS (no improvement): 4
2022-08-02 15:45:00,153 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,171 epoch 7 - iter 2/24 - loss 0.00000000 - samples/sec: 5880.68 - lr: 0.000002
2022-08-02 15:45:00,186 epoch 7 - iter 4/24 - loss 0.00000000 - samples/sec: 5087.96 - lr: 0.000002
2022-08-02 15:45:00,198 epoch 7 - iter 6/24 - loss 0.00000000 - samples/sec: 6383.27 - lr: 0.000002
2022-08-02 15:45:00,215 epoch 7 - iter 8/24 - loss 0.00000000 - samples/sec: 5643.19 - lr: 0.000002
2022-08-02 15:45:00,228 epoch 7 - iter 10/24 - loss 0.00000000 - samples/sec: 7493.38 - lr: 0.000002
2022-08-02 15:45:00,241 epoch 7 - iter 12/24 - loss 0.00000000 - samples/sec: 6402.60 - lr: 0.000002
2022-08-02 15:45:00,252 epoch 7 - iter 14/24 - loss 0.00000000 - samples/sec: 7803.13 - lr: 0.000002
2022-08-02 15:45:00,265 epoch 7 - iter 16/24 - loss 0.00000000 - samples/sec: 6036.46 - lr: 0.000002
2022-08-02 15:45:00,278 epoch 7 - iter 18/24 - loss 0.00000000 - samples/sec: 6980.15 - lr: 0.000002


2022-08-02 15:45:00,295 epoch 7 - iter 20/24 - loss 0.00000000 - samples/sec: 4429.12 - lr: 0.000002
2022-08-02 15:45:00,309 epoch 7 - iter 22/24 - loss 0.00000000 - samples/sec: 6041.49 - lr: 0.000002
2022-08-02 15:45:00,324 epoch 7 - iter 24/24 - loss 0.00000000 - samples/sec: 5165.20 - lr: 0.000002
2022-08-02 15:45:00,329 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,332 EPOCH 7 done: loss 0.0000 - lr 0.000002

100%|██████████| 3/3 [00:00<00:00, 151.38it/s]

2022-08-02 15:45:00,367 Evaluating as a multi-label problem: True
2022-08-02 15:45:00,406 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:45:00,414 BAD EPOCHS (no improvement): 4
2022-08-02 15:45:00,417 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,436 epoch 8 - iter 2/24 - loss 0.00000000 - samples/sec: 4247.46 - lr: 0.000002
2022-08-02 15:45:00,450 epoch 8 - iter 4/24 - loss 0.00000000 - samples/sec: 5924.68 - lr: 0.000002
2022-08-02 15:45:00,472 epoch 8 - iter 6/24 - loss 0.00000000 - samples/sec: 3261.99 - lr: 0.000002
2022-08-02 15:45:00,485 epoch 8 - iter 8/24 - loss 0.00000000 - samples/sec: 6183.01 - lr: 0.000002
2022-08-02 15:45:00,499 epoch 8 - iter 10/24 - loss 0.00000000 - samples/sec: 6028.87 - lr: 0.000001
2022-08-02 15:45:00,515 epoch 8 - iter 12/24 - loss 0.00000000 - samples/sec: 5302.11 - lr: 0.000001
2022-08-02 15:45:00,527 epoch 8 - iter 14/24 - loss 0.00000000 - samples/sec: 6382.66 - lr: 0.000001


2022-08-02 15:45:00,543 epoch 8 - iter 16/24 - loss 0.00000000 - samples/sec: 4627.72 - lr: 0.000001
2022-08-02 15:45:00,560 epoch 8 - iter 18/24 - loss 0.00000000 - samples/sec: 4662.93 - lr: 0.000001
2022-08-02 15:45:00,574 epoch 8 - iter 20/24 - loss 0.00000000 - samples/sec: 6403.52 - lr: 0.000001
2022-08-02 15:45:00,586 epoch 8 - iter 22/24 - loss 0.00000000 - samples/sec: 6457.28 - lr: 0.000001
2022-08-02 15:45:00,598 epoch 8 - iter 24/24 - loss 0.00000000 - samples/sec: 6788.96 - lr: 0.000001
2022-08-02 15:45:00,602 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,609 EPOCH 8 done: loss 0.0000 - lr 0.000001

100%|██████████| 3/3 [00:00<00:00, 152.95it/s]

2022-08-02 15:45:00,648 Evaluating as a multi-label problem: True
2022-08-02 15:45:00,677 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:45:00,684 BAD EPOCHS (no improvement): 4
2022-08-02 15:45:00,687 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,709 epoch 9 - iter 2/24 - loss 0.00000000 - samples/sec: 9146.64 - lr: 0.000001
2022-08-02 15:45:00,721 epoch 9 - iter 4/24 - loss 0.00000000 - samples/sec: 6131.74 - lr: 0.000001
2022-08-02 15:45:00,740 epoch 9 - iter 6/24 - loss 0.00000000 - samples/sec: 5021.99 - lr: 0.000001
2022-08-02 15:45:00,752 epoch 9 - iter 8/24 - loss 0.00000000 - samples/sec: 6440.70 - lr: 0.000001
2022-08-02 15:45:00,765 epoch 9 - iter 10/24 - loss 0.00000000 - samples/sec: 6190.57 - lr: 0.000001
2022-08-02 15:45:00,780 epoch 9 - iter 12/24 - loss 0.00000000 - samples/sec: 5380.55 - lr: 0.000001
2022-08-02 15:45:00,795 epoch 9 - iter 14/24 - loss 0.00000000 - samples/sec: 5126.04 - lr: 0.000001
2022-08-02 15:45:00,813 epoch 9 - iter 16/24 - loss 0.00000000 - samples/sec: 4292.84 - lr: 0.000001


2022-08-02 15:45:00,834 epoch 9 - iter 18/24 - loss 0.00000000 - samples/sec: 3808.03 - lr: 0.000001
2022-08-02 15:45:00,854 epoch 9 - iter 20/24 - loss 0.00000000 - samples/sec: 4334.57 - lr: 0.000001
2022-08-02 15:45:00,872 epoch 9 - iter 22/24 - loss 0.00000000 - samples/sec: 5155.48 - lr: 0.000001
2022-08-02 15:45:00,888 epoch 9 - iter 24/24 - loss 0.00000000 - samples/sec: 5380.65 - lr: 0.000001
2022-08-02 15:45:00,895 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,898 EPOCH 9 done: loss 0.0000 - lr 0.000001

100%|██████████| 3/3 [00:00<00:00, 117.61it/s]

2022-08-02 15:45:00,934 Evaluating as a multi-label problem: True
2022-08-02 15:45:00,963 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:45:00,971 BAD EPOCHS (no improvement): 4
2022-08-02 15:45:00,974 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:00,996 epoch 10 - iter 2/24 - loss 0.00000000 - samples/sec: 5000.01 - lr: 0.000001
2022-08-02 15:45:01,012 epoch 10 - iter 4/24 - loss 0.00000000 - samples/sec: 8982.28 - lr: 0.000001
2022-08-02 15:45:01,026 epoch 10 - iter 6/24 - loss 0.00000000 - samples/sec: 5724.79 - lr: 0.000000
2022-08-02 15:45:01,039 epoch 10 - iter 8/24 - loss 0.00000000 - samples/sec: 7890.06 - lr: 0.000000
2022-08-02 15:45:01,050 epoch 10 - iter 10/24 - loss 0.00000000 - samples/sec: 7842.57 - lr: 0.000000
2022-08-02 15:45:01,065 epoch 10 - iter 12/24 - loss 0.00000000 - samples/sec: 5463.22 - lr: 0.000000
2022-08-02 15:45:01,077 epoch 10 - iter 14/24 - loss 0.00000000 - samples/sec: 7037.79 - lr: 0.000000
2022-08-02 15:45:01,089 epoch 10 - iter 16/24 - loss 0.00000000 - samples/sec: 7229.81 - lr: 0.000000
2022-08-02 15:45:01,101 epoch 10 - iter 18/24 - loss 0.00000000 - samples/sec: 8023.78 - lr: 0.000000


2022-08-02 15:45:01,117 epoch 10 - iter 20/24 - loss 0.00000000 - samples/sec: 5787.12 - lr: 0.000000
2022-08-02 15:45:01,129 epoch 10 - iter 22/24 - loss 0.00000000 - samples/sec: 6961.68 - lr: 0.000000
2022-08-02 15:45:01,144 epoch 10 - iter 24/24 - loss 0.00000000 - samples/sec: 5271.30 - lr: 0.000000
2022-08-02 15:45:01,150 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:01,153 EPOCH 10 done: loss 0.0000 - lr 0.000000

100%|██████████| 3/3 [00:00<00:00, 128.73it/s]

2022-08-02 15:45:01,185 Evaluating as a multi-label problem: True
2022-08-02 15:45:01,219 DEV : loss 0.0 - f1-score (macro avg)  0.0
2022-08-02 15:45:01,232 BAD EPOCHS (no improvement): 4


2022-08-02 15:45:02,722 ----------------------------------------------------------------------------------------------------
2022-08-02 15:45:02,730 Testing using last state of model ...

100%|██████████| 3/3 [00:00<00:00, 171.32it/s]

2022-08-02 15:45:02,761 Evaluating as a multi-label problem: False
2022-08-02 15:45:02,778 0.0	0.0	0.0	0.0
2022-08-02 15:45:02,781 
Results:
- F-score (micro) 0.0
- F-score (macro) 0.0
- Accuracy 0.0

By class:
                     precision    recall  f1-score   support

           liegt_in     0.0000    0.0000    0.0000       6.0
      inhaftiert_in     0.0000    0.0000    0.0000       5.0
       Mitglied_bei     0.0000    0.0000    0.0000       5.0
Führungsposition_in     0.0000    0.0000    0.0000       4.0
           wohnt_in     0.0000    0.0000    0.0000       4.0
      Aufenthalt_in     0.0000    0.0000    0.0000       3.0
       arbeitet_bei     0.0000    0.0000    0.0000       2.0
     Teilgruppe_von     0.0000    0.0000    0.0000       2.0
            Ehe_mit     0.0000    0.0000    0.0000       2.0
         geboren_in     0.0000    0.0000    0.0000       2.0
       gestorben_in     0.0000    0.0000    0.0000       2.0
        arbeitet_in     0.0000    0.0000    0.0000       2.0
        ermordet_in     0.0000    0.0000    0.0000       2.0
         Gegner_von     0.0000    0.0000    0.0000       2.0
          liegt_bei     0.0000    0.0000    0.0000       1.0
           tätig_in     0.0000    0.0000    0.0000       1.0
           Kurzform     0.0000    0.0000    0.0000       1.0
           Standort     0.0000    0.0000    0.0000       1.0
         neuer_Name     0.0000    0.0000    0.0000       1.0
        Geschwister     0.0000    0.0000    0.0000       1.0
 ausgeschlossen_von     0.0000    0.0000    0.0000       1.0
    Gruppenmitglied     0.0000    0.0000    0.0000       1.0
         Kontakt_zu     0.0000    0.0000    0.0000       1.0
       verfolgt_von     0.0000    0.0000    0.0000       1.0

          micro avg     0.0000    0.0000    0.0000      53.0
          macro avg     0.0000    0.0000    0.0000      53.0
       weighted avg     0.0000    0.0000    0.0000      53.0

2022-08-02 15:45:02,785 ----------------------------------------------------------------------------------------------------

[alanakbik]

Yes, the weird sentence printout points to the problem. Try:

columns = {1: 'text', 2: 'ner'}

instead of

columns = {0:'id', 1: 'form', 2: 'ner'}

and see if the sentences print out better. We actually use "text" instead of "form" but I think we should probably add support for calling it "form" as well.

[geheim01]

The sentences are now printed properly!

However, the results are still the same as in the training log.

[alanakbik]

Strange, @dobbersc can you take a look?

[dobbersc]

Hey @geheim01,

1. In your training script, you have used the same values for the entity_pair_filters in the RelationExtractor as in the example script:

   # Step 4: Initialize relation classifier
   model: RelationExtractor = RelationExtractor(
       embeddings=embeddings,
       label_dictionary=label_dictionary,
       label_type="relation",
       entity_label_type="ner",
       entity_pair_filters=[  # Define valid entity pair combinations, used as relation candidates
         ("LOC", "LOC"),
         ("PER", "LOC"),
         ("PER", "ORG"),
         ("ORG", "LOC"),
         ("PER", "PER"),
         ("LOC", "LOC")
       ],
   )

   These entity_pair_filters need to be adjusted to conform with the structure of your relations. In the dataset snipped, you have entity labels, such as PER, ORG, Coref_PER and Coref_LOC. The entity_pair_filters are a list of valid relation entity pair combinations used as relation candidates. Here, specify entity pairs in a list of tuples of labels (<HEAD>, <TAIL>). E.g. for the wohnt_in relation, only relations from PER to LOC and Coref_PER to Coref_LOC make sense:

   entity_pair_filters = [('PER', 'LOC'), ('Coref_PER', 'Coref_LOC')]

   It is recommended to populate the entity pair filters for improved model performance. If you don't want to use them, you may also use None as a value.

2. It looks very weird that the loss is constantly zero, even at the beginning, same for the f1 score. If my first suggestion with the entity pair filters does not work, it would be best if you could provide the training script and the dataset file.

[geheim01]

Hey @dobbersc ,

Unfortunately also the adjustment of the entity_pair_filters to all possible combinations did not improve the situation and the loss remains at 0 as before from the beginning. That's why I have sent you our training script and dataset file via email

[geheim01]

Ok Update: Now that I have once renamed all NER tags in our conll file from, for example, PER's to I-PER's/LOC to I-LOC, I am now not directly getting Loss values of 0 when training the relation extraction model.

[dobbersc]

Ok Update: Now that I have once renamed all NER tags in our conll file from, for example, PER's to I-PER's/LOC to I-LOC, I am now not directly getting Loss values of 0 when training the relation extraction model.

Good catch! In Flair, we use the BIO or BIOES annotation format for spans. For an example see this comment. So

1 Anna PER
2 Kreuzer PER
3 wuchs O

should actually be

1 Anna B-PER
2 Kreuzer I-PER
3 wuchs O

I hope you succeed in training your model, now :).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.