機械学習でGitHubタスクを自動化して楽しみと利益を得る方法

An archive of data science, data analytics, data engineering, machine learning, and artificial intelligence writing from the former Towards Data Science Medium publication.

How to Automate Tasks on GitHub With Machine Learning for Fun and Profit

A tutorial on how to build a GitHub App that predicts and applies issue labels using Tensorflow and public datasets.

By Hamel Husain, Michal Jastrzębski and Jeremy Lewi

Teaser: Build a model that labels issues and launch it as a product!

Link to this issue. More examples can be viewed here.Press enter or click to view image in full sizeLink to this issue. More examples can be viewed hereInstall this app on your GitHub repositories from this page.

Motivation: The elusive, perfect machine learning problem

Finding the ideal machine learning problem can feel like searching for treasure or chasing dragons. Image credit.Our friends and colleagues who are data scientists would describe the ideal predictive modeling project as a situation where:

There is an abundance of data, which is already labeled or where labels can be inferred.

The data can be used to solve real problems.

The problem relates to a domain you are passionate about or the problem you want to solve is your own and you can be your first customer.

There is a platform where your data product can reach a massive audience, with mechanisms to gather feedback and improve.

You can create this with minimal expense and time, hopefully using languages and tools you are familiar with.

There is a way to monetize your product if it becomes successful.

The above list is aspirational and data scientists are lucky to encounter a problem that meets all of these (the authors feel lucky if we can find a problem that satisfies even half of these!).

Enter GH-Archive & GitHub Apps: Where Data Meets Opportunity

Today, we present a dataset, platform, and domain that we believe satisfies the criteria set forth above!

GH-Archive logs a tremendous amount of data from GitHub by ingesting most of these events from the GitHub REST API. These events are sent from GitHub to GH-Archive in JSON format, referred to as a payload. Below is an example of a payload that is received when an issue is edited:

A truncated version of this example.As you can imagine, there is a large number of payloads given the number of event types and users on GitHub. Thankfully, this data is stored in BigQuery which allows for fast retrieval through a SQL interface! It is very economical to acquire this data, as Google gives you $300 when you first sign up for an account, and if you already have one, the costs are very reasonable.

Since the data is in a JSON format, the syntax for un-nesting this data may be a bit unfamiliar. We can use the JSON_EXTRACT function to get the data we need. Below is a toy example of how you might extract data from issue payloads:

Example query syntax you might use for GH-Archive data stored on BigQuery. Pay close attention to the syntax used.A step-by-step explanation on how to extract GitHub issues from BigQuery can be found in the appendix section of this article, however, it is important to note that more than issue data is available — you can retrieve data for almost anything that happens on GitHub! You can even retrieve large corpus of code from public repos in BigQuery.

The platform: GitHub Apps & The GitHub Marketplace

The GitHub platform allows you to build apps that can perform many actions, such as interacting with issues, creating repositories or fixing code in pull requests. Since all that is required from your app is to receive payloads from GitHub and make calls to the REST API, you can write the app in any language of your choice, including python.

Most importantly, the GitHub marketplace gives you a way to list your app on a searchable platform and charge users a monthly subscription. This is a great way to monetize your ideas. You can even host unverified free apps as a way to collect feedback and iterate.

Surprisingly, there are not many GitHub apps that use machine learning, despite the availability of these public datasets! Raising awareness of this is one of the motivations for this blog post.

An End-to-End Example: Automatically Label GitHub Issues With Machine Learning

Automatically labeling issues can help with organizing and finding information. Photo by Susan YinIn order to show you how to create your own apps, we will walk you through the process of creating a GitHub app that can automatically label issues. Note that all of the code for this app, including the model training steps are located in this GitHub repository.

Step 1: Register your app & complete pre-requisites.

First, you will need to set up your development environment. Complete steps 1–4 of this article. You do not need to read the section on “The Ruby Programming Language”, or any steps beyond step 4. Make sure you set up a Webhook secret even though that part is optional.

Note that there is a difference between GitHu