WebLINX

🤗Dataset	📄Paper	🌐Website	📓Colab
🤖Models	💻Explorer	🐦Tweets	🏆Leaderboard

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Xing Han Lù*, Zdeněk Kasner*, Siva Reddy
*Equal contribution
ICML 2024 (Spotlight)

Intro

Welcome to WebLINX's official repository! In addition to providing code used to train the models reported in our WebLINX paper, we also provide a comprehensive Python library (aka API) to help you work with the WebLINX dataset.

If you want to get started with weblinx, please check out the following places:

If you want to learn more about the codebase itself, please keep on reading!

Installation

# Install the base package
pip install weblinx

# Install all dependencies
pip install weblinx[all]

# Install specific dependencies for...
# ...processing HTML 🖥️
pip install weblinx[processing]
# ...video processing 📽️
pip install weblinx[video]
# ...evaluating models 🔬
pip install weblinx[eval]
# ...development of this library 🛠️
pip install weblinx[dev]

Structure

This repository is structured in the following way:

Modeling

Our modeling/ repo-level directory has code for processing, training and evaluating the models reported in the paper (DMR, LLaMA, MindAct, Pix2Act, Flan-T5). It is separate from the weblinx library, which focuses on data processing and evaluation. You can use it by cloning this repository, and it is recommended to edit the files in modeling/ directly for your own needs. Our modeling code is separate from the weblinx library, but requires it as a dependency. You can install the modeling code by running:

# First, install the base package
pip install weblinx

# Then, clone this repo
git clone https://github.com/McGill-NLP/weblinx
cd weblinx/modeling

For the rest of the instructions, please take a look at the modeling README.

Evaluation

To install packages necessary for evaluation, run:

pip install weblinx[eval]

You can now access the evaluation module by importing in Python:

import weblinx.eval

Use weblinx.eval.metrics for evaluation metrics, weblinx.eval.__init__ for useful evaluation-related functions. You may also find it useful to take a look at weblinx.processing.outputs to get an idea of how to use the outputs of the model for evaluation.

To run the automatic evaluation, you can use the following command:

python -m weblinx.eval --help

For more examples on how to use weblinx.eval, take a look at the modeling README.

Note: We are still working on the code for weblinx.eval and weblinx.processing.outputs. If you have any questions or would like to contribute docs, please feel free to open an issue or a pull request.

Citations

If you use this library, please cite our work using the following:

@misc{lù2024weblinx,
      title={WebLINX: Real-World Website Navigation with Multi-Turn Dialogue}, 
      author={Xing Han Lù and Zdeněk Kasner and Siva Reddy},
      year={2024},
      eprint={2402.05930},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

This project's license can be found at LICENSE. Please note that the license of the data in tests/data follow the license from the official dataset, not the license of this repository. The official dataset's license can be found in the official dataset page. The license of the models trained using this repo might also differ - please find them in the respective model cards.