srcopsmetrics 2.6.3

Data extraction request

To request data extraction for repository or organization, create Data Extraction Issue in MI-Scheduler repository. Use this link TODO

Data extraction Pipeline (diagram)

MI pipeline is simple to understand, see diagram below

                  +---------+
                  |ConfigMap|
                  +----+----+
                       |
            +--+-------+--------+--+
            |  |                |  |
            |  |  mi-scheduler  |  |
            |  |                |  |
            +------+---+---+-------+
                |   |   |   |    |
                |   |   |   |    |
                |   |   |   |    |
                | Argo Workflows |
                |   |   |   |    |
                |   |   |   |    |
+---------------v---v---v---v----v------------------+                                          +--------------------        +--------------------+
|                                                   |                                          |   Visualization   |        |   Recommendation   |
|  +---------+  +---------+            +---------+  |                                          +-------------------+        +--------------------+
|  |thoth/   |  |  AICoE  |            | your    |  |                                          |   Project Health  |        |   thoth            |
|  |  station|  |         |            |     org |  |                                          |    (dashboard)    |        |                    |
|  +---------+  +---------+            +---------+  |                                          |                   |        |                    |
|  |solver   |  |...      |            |your     |  |                                          +---------+---------+        +----------+---------+
|  |         |  |         |            |   repos |  |           thoth-station/mi                         ^                             ^
|  |amun     |  |...      | X X X X X  |         |  |     (Meta-information Indicators)                  |                             |
|  |         |  |         |            |         |  |                                                    +-------------+---------------+
|  |adviser  |  |...      |            |         |  |                                                                  |
|  |         |  |         |            |         |  |                                                                  |
|  |....     |  |...      |            |         |  |                                                +-----------------+-------------------+
|  |         |  |         |            |         |  |                                                |                                     |
|  +---------+  +---------+            +---------+  |                                                |       Knowledge Processsing         |
|                                                   |                                                |                                     |
+-----------------------+---------------------------+                                                +-----------------+-------------------+
GitHub repositories   |                                                                                              ^
                        |                 +--------------------------------------------------------+                   |
                        |                 |                                                        |                   |
                        |                 |      Entities Analysis   +------->      Knowledge      |                   |
                        +---------------->-+                                                      +--------------------+
                                          +---------+----------------+----------+------------------+
                                          |  Issues |  Pull Requests |  Readmes |  etc...........  |
                                          |         |                |          |                  |
                                          +---------+----------------+----------+------------------+

What can MI extract from GitHub?

MI analyses entities specified on the srcopsmetrics/entities page Entity is essentialy a repository metadata that is being inspected (e.g. Issue or Pull Request), from which specified features are extracted and are stored to dataframe.

MI is essentialy wrapped around PyGitHub module to provide careless data extraction with API rate limit handling and data updating.

pip

MI is available through PyPI, so you can do

pip install srcopsmetrics

git

Alternatively, you can install srcopsmetrics by cloning repository

git clone https://github.com/thoth-station/mi.git

cd mi

pipenv install --dev

Setup

To store data locally, use -l when calling CLI or set is_local=True when using MI as a module.

By default MI will try to store the data on Ceph. In order to store on Ceph you need to provide the following env variables:

• S3_ENDPOINT_URL Ceph Host name

• CEPH_BUCKET Ceph Bucket name

• CEPH_BUCKET_PREFIX Ceph Prefix

• CEPH_KEY_ID Ceph Key ID

• CEPH_SECRET_KEY Ceph Secret Key

For more information about Ceph storing look here

CLI

See –help for all available options

See some of the examples below

srcopsmetrics --create --is-local --repository foo_repo --entities PullRequest

srcopsmetrics -clr foo_repo -e PullRequest

srcopsmetrics -clo foo_org -e PullRequest

srcopsmetrics -clr foo_repo,bar_repo -e PullRequest

srcopsmetrics -clr foo_repo -e PullRequest,Issue,Commit

Implementation

Look at Template entity to get an idea for requirements that need to be satisfied for custom entity implementation.