msticpy 0.2.1

MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE Project-URL: Documentation, http://msticpy.readthedocs.io Project-URL: Code, https://github.com/microsoft/msticpy Description: # MSTIC Jupyter and Python Security Tools

Microsoft Threat Intelligence Python Security Tools.

The msticpy package was initially developed to support [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/What%20is%20the%20Jupyter%20Notebook.html) authoring for [Azure Sentinel](https://azure.microsoft.com/en-us/services/azure-sentinel/). Many of the included tools can be used in other security scenarios for threat hunting and threat investigation. There are three main sub-packages:

• sectools - python security tools to help with data analysis or investigation

• nbtools - Jupyter-specific UI tools such as widgets and data display

• data - data interfaces specific to Sentinel/Log Analytics

The package is in an early preview mode so there are likely to be bugs and there are several areas that are not yet optimized for performance. We welcome feedback, bug reports and suggestions for new features.

## Installing

pip install msticpy

or for the latest dev build

pip install git+https://github.com/microsoft/msticpy

## Documentation

Full documentation is at [ReadTheDocs](https://msticpy.readthedocs.io/en/latest/overview.html)

Sample notebooks for many of the modules are in the [docs/notebooks](./docs/notebooks) folder and accompanying notebooks.

You can also browse through the sample notebooks referenced at the end of this document (especially the Windows Alert Investigation notebook) to see some of the functionality used in context.

—

## Security Tools Sub-package - sectools

This subpackage contains several modules helpful for working on security investigations and hunting:

### base64unpack

Base64 and archive (gz, zip, tar) extractor. Input can either be a single string or a specified column of a pandas dataframe. It will try to identify any base64 encoded strings and decode them. If the result looks like one of the supported archive types it will unpack the contents. The results of each decode/unpack are rechecked for further base64 content and will recurse down up to 20 levels (default can be overridden). Output is to a decoded string (for single string input) or a DataFrame (for dataframe input).

[Base64Unpack Notebook](./docs/notebooks/Base64Unpack.ipynb)

### iocextract

Uses a set of builtin regular expressions to look for Indicator of Compromise (IoC) patterns. Input can be a single string or a pandas dataframe with one or more columns specified as input.

The following types are built-in:

• IPv4 and IPv6

• URL

• DNS domain

• Hashes (MD5, SHA1, SHA256)

• Windows file paths

• Linux file paths (this is kind of noisy because a legal linux file path can have almost any character)

You can modify or add to the regular expressions used at runtime.

Output is a dictionary of matches (for single string input) or a DataFrame (for dataframe input).

[Base64Unpack Notebook](./docs/notebooks/IoCExtract.ipynb)

### vtlookup

Wrapper class around [Virus Total API](https://www.virustotal.com/en/documentation/public-api/). Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing requires a Virus Total account and API key and processing performance is limited to the number of requests per minute for the account type that you have. Support IoC Types:

• Filehash

• URL

• DNS Domain

• IPv4 Address

[VTLookup Notebook](./docs/notebooks/VirusTotalLookup.ipynb)

### geoip

Geographic location lookup for IP addresses. This module has two classes for different services:

• GeoLiteLookup - Maxmind Geolite (see <https://www.maxmind.com>)

• IPStackLookup - IPStack (see <https://ipstack.com>)

Both services offer a free tier for non-commercial use. However, a paid tier will normally get you more accuracy, more detail and a higher throughput rate. Maxmind geolite uses a downloadable database, while IPStack is an online lookup (API key required).

[GeoIP Lookup Notebook](./docs/notebooks/GeoIPLookups.ipynb)

### eventcluster

This module is intended to be used to summarize large numbers of events into clusters of different patterns. High volume repeating events can often make it difficult to see unique and interesting items.

This is an unsupervised learning module implemented using SciKit Learn DBScan.

The module contains functions to generate clusterable features from string data. For example, an administration command that does some maintenance on thousands of servers with a commandline can be collapsed into a single cluster pattern by ignoring the character values in the string and using delimiters or tokens to group the values.

`bash install-update -hostname {host.fqdn} -tmp:/tmp/{GUID}/rollback `

### outliers

Similar to the eventcluster module, but a little bit more experimental (read ‘less tested’). It uses SkLearn Isolation Forest to identify outlier events in a single data set or using one data set as training data and another on which to predict outliers.

### auditdextract

Module to load and decode Linux audit logs. It collapses messages sharing the same message ID into single events, decodes hex-encoded data fields and performs some event-specific formatting and normalization (e.g. for process start events it will re-assemble the process command line arguments into a single string).

This is still a work-in-progress.

## Notebook tools sub-package - nbtools

This is a collection of display and utility modules designed to make working with security data in Jupyter notebooks quicker and easier.

• nbwidgets - groups common functionality such as list pickers, time boundary settings, saving and retrieving environment variables into a single line callable command.

• nbdisplay - functions that implement common display of things like alerts, events in a slightly more consumable way than print()

• entityschema - implements entity classes (e.g. Host, Account, IPAddress) used in Log Analytics alerts and in many of these modules. Each entity encaspulates one or more properties related to the entity.

[Notebook Tools Notebook](./docs/notebooks/NotebookWidgets.ipynb)

## Data sub-package - data

These components are currently still part of the nbtools sub-package but will be refactored to separate them into their own package.

• query manager - collection of modules that implement common kql/Log Analytics queries using KqlMagic

• security_alert and security_event - encapsulation classes for alerts and events.

Each has a standard ‘entities’ property reflecting the entities found in the alert or event. These can also be used as meta-parameters for many of the queries. For example, the following query will extract the value for the hostname query parameter from the alert:

qry.list_host_logons(provs==[query_times, alert])

—

## Clone the notebooks in this repo to Azure Notebooks

Requires sign-in to Azure Notebooks <a href=”https://notebooks.azure.com/import/gh/Microsoft/msticpy”> <img src=”https://notebooks.azure.com/launch.png” /> </a>

## More Notebook Examples

See the following notebooks for more examples of the use of this package in practice:

• Windows Alert Investigation in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb)

• Windows Host Explorer in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb)

• Office 365 Exploration in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb)

• Cross-Network Hunting in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb)

## To-Do Items

• Refactor data modules into separate package(s).

• Replace custom data schema with [Intake](https://intake.readthedocs.io/en/latest/).

• Add additional notebooks to document use of the tools.

## Supported Platforms and Packages

• msticpy is OS-independent

• Requires [Python 3.6 or later](https://www.python.org/dev/peps/pep-0494/)

• Requires the following python packages: pandas, bokeh, matplotlib, seaborn, setuptools, urllib3, ipywidgets, numpy, attrs, requests, networkx, ipython, scikit_learn, typing

• The following packages are recommended and needed for some specific functionality: Kqlmagic, maxminddb_geolite2, folium, dnspython, ipwhois

See [requirements.txt](requirements.txt) for more details and version requirements.

—

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit <https://cla.microsoft.com>.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

Keywords: security,azure,sentinel Platform: UNKNOWN Classifier: Programming Language :: Python :: 3.6 Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Development Status :: 4 - Beta Requires-Python: >=3.6 Description-Content-Type: text/markdown