Skip to main content

Rater scoring modeling tool

Project description

Rater Scoring Modeling Tool

Gitlab CI status Azure Pipeline status Coverage status Conda package for SKLL Docs DOI for citing RSMTool Supported python versions for RSMTool Latest version on PyPI License

Introduction

Automated scoring of written and spoken test responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include Project Essay Grade for written responses and SpeechRater for spoken responses.

Rater Scoring Modeling Tool (RSMTool) is a python package which automates and combines in a single pipeline multiple analyses that are commonly conducted when building and evaluating such scoring models. The output of RSMTool is a comprehensive, customizable HTML statistical report that contains the output of these multiple analyses. While RSMTool does make it really simple to run a set of standard analyses using a single command, it is also fully customizable and allows users to easily exclude unneeded analyses, modify the default analyses, and even include custom analyses in the report.

We expect the primary users of RSMTool to be researchers working on developing new automated scoring engines or on improving existing ones. Note that RSMTool is not a scoring engine by itself but rather a tool for building and evaluating machine learning models that may be used in such engines.

RSMTool is driven by a configuration file that users have to supply. Given the large number of available options, this can get complicated especially for new users. That’s why RSMTool can help users generate configuration files interactively via guided prompts! The video below demonstrates this feature.

Interactive Generation Demo Video

Getting Started

To get started with RSMTool, please see the extensive official documentation. If you use the Dash app on macOS, you can also download the complete RSMTool documentation for offline use. Go to the Dash preferences, click on “Downloads”, then “User Contributed”, and search for “RSMTool”.

Requirements

  • ipython

  • jupyter

  • notebook

  • numpy<2

  • openpyxl

  • pandas

  • python>=3.8

  • scikit-learn

  • scipy

  • seaborn

  • shap

  • skll==4.0.1

  • statsmodels

  • wandb

  • xlrd

  • xlwt

Contributing

Contributions to RSMTool are very welcome. Please refer to the documentation for how to get started on developing new features or functionality for RSMTool.

Citing

If you are using RSMTool in your work, you can cite it as follows:

MLA

Madnani, Nitin and Loukina, Anastassia. “RSMTool: A Collection of Tools for Building and Evaluating Automated Scoring Models”. Journal of Open Source Software 1(3), 2016.

BibTex

@article{MadnaniLoukina2016,
  doi = {10.21105/joss.00033},
  url = {http://dx.doi.org/10.21105/joss.00033},
  year  = {2016},
  month = {jul},
  publisher = {The Open Journal},
  volume = {1},
  number = {3},
  author = {Nitin Madnani and Anastassia Loukina},
  title = {{RSMTool}: A Collection of Tools for Building and Evaluating Automated Scoring Models},
  journal = {{Journal of Open Source Software}}
}

Changelog

See GitHub Releases.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rsmtool-12.0.0.tar.gz (212.8 kB view details)

Uploaded Source

File details

Details for the file rsmtool-12.0.0.tar.gz.

File metadata

  • Download URL: rsmtool-12.0.0.tar.gz
  • Upload date:
  • Size: 212.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.13

File hashes

Hashes for rsmtool-12.0.0.tar.gz
Algorithm Hash digest
SHA256 dfcbd0d1fc3c3bdb9ad97377917a91b885fa8a140774e6befa5ce90acfefc093
MD5 b17e745d4e74a045d9737d926d0e0325
BLAKE2b-256 79c28992752dd5662c5edc270adf3715d83f596749e825ab86c7970d6942635f

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page