DaCy: An efficient NLP Pipeline for Danish

DaCy is a Danish natural language preprocessing framework made with SpaCy. Its largest pipeline has achieved State-of-the-Art performance on Named entity recognition, part-of-speech tagging and dependency parsing for Danish. Feel free to try out the demo. This repository contains material for using the DaCy, reproducing the results and guides on usage of the package. Furthermore, it also contains behavioural tests for biases and robustness of Danish NLP pipelines.

📰 News

• 1.2.0 (04/11/21)
  • Removed DaNLP dependency, now DaNLP models is downloaded directly from Huggingface's model hub which is faster and more stable 🌟.
  • Removed the readability module, we recommend using the more extensive textdescriptives package developed by HLasse and I for extracting readability and other text metrics.
  • Added support for configuring the default save path with the environmental variable DACY_CACHE_DIR thanks to a PR by dhpullack 🙏.
• 1.1.0 (23/07/21)
  • DaCy is now available on the Huggingface model hub 🤗 . Including detailed performance descriptions of biases and robustness.
  • It also got a brand new online demo - try it out!
  • And more, including documentation update and prettier prints.
• 1.0.0 (09/07/21)
  • DaCy version 1.0.0 releases as the first version to pypi! 📦
    • Including a series of augmenters with a few specifically designed for Danish
    • Code for behavioural tests of NLP pipelines
    • A new tutorial for both 📖
    • The first paper on DaCy; check it out as a preprint and code for reproducing it here! 🌟
  • A new beautiful hand-drawn logo 🤩
  • A test for biases and robustness in Danish NLP pipelines 🧐
  • DaCy is now officially supported by the Centre for Humanities Computing at Aarhus University
• 0.4.1 (03/06/21)
  • DaCy now has a stunningly looking documentation site 🌟. It even has a dark mode!

To see older news items check out the website.

🔧 Installation

To get started using DaCy simply install it using pip by running the following line in your terminal:

pip install dacy

The large version of DaCy uses the sentencepiece tokenizer and protobuf for serialization, to install both of these as well, simply run:

pip install dacy[large]

pip install dacy[danlp]

pip install dacy[all]

git clone https://github.com/centre-for-humanities-computing/DaCy.git
cd DaCy
pip install .

👩‍💻 Usage

To use the model you first have to download either the small, medium, or large model. To see a list of all available models:

import dacy
for model in dacy.models():
    print(model)
# da_dacy_small_tft-0.0.0
# da_dacy_medium_tft-0.0.0
# da_dacy_large_tft-0.0.0

To download and load a model simply execute:

nlp = dacy.load("da_dacy_medium_tft-0.0.0")
# or equivalently
nlp = dacy.load("medium")

Which will download the model to the .dacy directory in your home directory.

To download the model to a specific directory:

dacy.download_model("da_dacy_medium_tft-0.0.0", your_save_path)
nlp = dacy.load_model("da_dacy_medium_tft-0.0.0", your_save_path)

For more on how to use DaCy please check out our documentation

👩‍🏫 Tutorials and documentation

DaCy includes detailed documentation as well as a series of Jupyter notebook tutorial. If you do not have Jupyter Notebook installed, instructions for installing and running it can be found here. All the tutorials are located in the tutorials folder.

	Content	Google Colab
🌟 Getting Started	An introduction on how to use DaCy
📖 Documentation	The Documentation of DaCy
😡😂 Sentiment	A simple introduction to the new sentiment features in DaCy.
🍒 Augmentation	A guide on how to augment text using the DaCy and SpaCy augmenters.
⛑ Fairness and Robustness	A guide on how to use augmenters to measure model robustness and biases using DaCy.
😎 wrapping a fine-tuned Tranformer	A guide on how to wrap an already fine-tuned transformer to and add it to your SpaCy pipeline using DaCy helper functions.

🦾 Performance and Training

The following table shows the performance on the DaNE test set when compared to other models. Highest scores are highlighted with bold and second highest is underlined.

Stanza uses the spacy-stanza implementation. The speed on the DaNLP model is as reported by the framework, which does not utilize batch input. However, given the model size, it can be expected to reach speeds comparable to DaCy medium. Empty cells indicate that the framework does not include the specific model.

Robustness and Biases

DaCy compares the performance of Danish language processing pipeline under a large variaty of augmentations to test the robustness and biases hereof. To find out more please check the website.

🤔 Issues and Usage Q&A

To ask report issues or request features, please use the GitHub Issue Tracker. Questions related to SpaCy are kindly referred to the SpaCy GitHub or forum. Otherwise, please use the discussion Forums.

FAQ

where_is_my_dacy()

pip install -r requirements.txt
pip install pytest

python -m pytest

python -m pytest dacy/tests/test_utils.py

pip install pytest-cov

python -m pytest--cov=.

# install sphinx, themes and extensions
pip install sphinx furo sphinx-copybutton sphinxext-opengraph

# generate html from documentations

make -C docs html

Acknowledgements

DaCy is a result of great open-source software and contributors. It wouldn't have been possible without the work by the SpaCy team which developed and integrated the software. Huggingface for developing Transformers and making model sharing convenient. BotXO for training and sharing the Danish BERT model and Malte Hojmark-Bertelsen for making it easily available and developing Ælæctra. A huge compliment also goes out to DaNLP which have made it easy to get access to Danish resources and even supplied some of the tagged data themselves.

References

If you use this library in your research, please cite the paper:

@misc{enevoldsen2021dacy,
      title={DaCy: A Unified Framework for Danish NLP}, 
      author={Kenneth Enevoldsen and Lasse Hansen and Kristoffer Nielbo},
      year={2021},
      eprint={2107.05295},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

To read more on this paper or to see the code for reproducing the results please check out the associated readme.

License

DaCy is released under the Apache License, Version 2.0. See the LICENSE file for more details.

Contact

For feature request issues and bugs please use the GitHub Issue Tracker. Otherwise, please use the Discussion Forums.