Cargo Crates

An easy way to build data extractors in Docker.

Get it ... it's data containers ... cargo crates.

Anyway.

Usage

Convention over configuration(?).

Cargo crates share a common set of patterns that make them easy to use.

Each crate is associated with a specific API.
Authentication is provided via environment variables.
Parameters are provided to the container at runtime.

For example, to extract sleep data from the Oura API you would run a container like this:

 docker run -e OURA_PAT=<TOKEN> ghcr.io/dacort/crates-oura

You can be more specific or select different APIs, too:

# Returns data beginning 2021-01-01
docker run -e OURA_PAT=<TOKEN> -e start=2021-01-01 ghcr.io/dacort/crates-oura

# Returns activity data instead of sleep data (the default)
docker run -e OURA_PAT=<TOKEN> ghcr.io/dacort/crates-oura activity

Other APIs might need more flexibility - for example, with Twitter there are numerous endpoints.

In this case, the .env file contains the environment variables needed for Twitter authentication.

# Extract current user info
docker run --env-file .env ghcr.io/dacort/crates-twitter users/show dacort

# Or fetch a user's followers
docker run --env-file .env ghcr.io/dacort/crates-twitter followers dacort

The idea is that the Docker container provides an abstraction on top of the underlying APIs and can be implemented however you want.

Perhaps someday down the road there could be a set of base images that provide a nice set of abstractions for generic authentication and error handling. But for now, it's just a bunch of python.

Caveats

Right now I'm building this for my own purposes. So the set of supported APIs or endpoints is rather limited.

Supported Services

GitHub - Get traffic stats and release information, including downloads
YouTube - Get stats for specific video IDs and channel IDs
Oura - Get your sleep, activity, and readiness stats
Reddit - Get your saved posts and search all of Reddit or a subreddit
Slack - List channels in a Slack team and search for a keyword
Stack Overflow - Search for questions on Stack Overflow
Twitter - List the last 10,000 followers for a screen name and show a user's profile

GitHub

Supported Commands:

traffic - returns results from GitHub traffic stats including clones, popular/paths, popular/referrers, and views. Each type of traffic stat is a subcommand.
releases - returns information about releases for a specific GitHub repo.

Environment Variables:

GITHUB_PAT - Your GitHub Personal Access Token

Examples:

Return all traffic stats for dacort/cargo-crates

docker run -e GITHUB_PAT \
    ghcr.io/dacort/crates-github \
    traffic dacort/cargo-crates

{"repo": "dacort/cargo-crates", "path": "clones", "stats": {"count": 77, "uniques": 8, "clones": [{"timestamp": "2021-03-18T00:00:00Z", "count": 77, "uniques": 8}]}}
{"repo": "dacort/cargo-crates", "path": "popular/paths", "stats": [{"path": "/dacort/cargo-crates/actions", "title": "Actions \u00b7 dacort/cargo-crates", "count": 33, "uniques": 1}, {"path": "/dacort/cargo-crates", "title": "dacort/cargo-crates", "count": 11, "uniques": 2}, {"path": "/dacort/cargo-crates/actions/workflows/crates.yaml", "title": "Actions \u00b7 dacort/cargo-crates", "count": 7, "uniques": 1}, {"path": "/dacort/cargo-crates/actions/runs/666130915", "title": "Remove useless job0 \u00b7 dacort/cargo-crates@6b54337", "count": 4, "uniques": 1}, {"path": "/dacort/cargo-crates/actions/runs/666151009", "title": "Trying something else \u00b7 dacort/cargo-crates@ed3d226", "count": 3, "uniques": 1}, {"path": "/dacort/cargo-crates/actions/runs/666165537", "title": "Hmm \u00b7 dacort/cargo-crates@64c59e7", "count": 3, "uniques": 1}, {"path": "/dacort/cargo-crates/actions/runs/666215936", "title": "Add requirements \u00b7 dacort/cargo-crates@6161093", "count": 3, "uniques": 1}, {"path": "/dacort/cargo-crates/actions/runs/666227882", "title": "Does this actually work now?? \u00b7 dacort/cargo-crates@37d31ea", "count": 3, "uniques": 1}, {"path": "/dacort/cargo-crates/pulls", "title": "Pull requests \u00b7 dacort/cargo-crates", "count": 3, "uniques": 1}, {"path": "/dacort/cargo-crates/actions/workflows/test_matrix.yaml", "title": "Actions \u00b7 dacort/cargo-crates", "count": 2, "uniques": 1}]}
{"repo": "dacort/cargo-crates", "path": "popular/referrers", "stats": [{"referrer": "github.com", "count": 3, "uniques": 2}]}
{"repo": "dacort/cargo-crates", "path": "views", "stats": {"count": 108, "uniques": 2, "views": [{"timestamp": "2021-03-18T00:00:00Z", "count": 108, "uniques": 2}]}}

Return only views for dacort/cargo-crates

docker run -e GITHUB_PAT ghcr.io/dacort/crates-github traffic dacort/cargo-crates views

{
  "repo": "dacort/cargo-crates",
  "path": "views",
  "stats": {
    "count": 108,
    "uniques": 2,
    "views": [
      { "timestamp": "2021-03-18T00:00:00Z", "count": 108, "uniques": 2 }
    ]
  }
}

YouTube

Supported Commands:

videos - return video stats for up to 50 video IDs.
channel_videos - return video stats for 50 videos on the provided channel ID.

Environment Variables:

YOUTUBE_API_KEY - An API key for the YouTube Data API

Examples:

Return video stats for Intro to EMR Studio
```
docker run -e YOUTUBE_API_KEY \
    ghcr.io/dacort/crates-youtube \
    videos rZ3zeJ6WKPY
```
Click to view JSON
```json { "kind": "youtube#video", "etag": "tbhtHqbvEMe3jNGfFGK36HpowJk", "id": "rZ3zeJ6WKPY", "snippet": { "publishedAt": "2021-04-21T22:43:26Z", "channelId": "UCd6MoB9NC6uYN2grvUNT-Zg", "title": "Intro to Amazon EMR Studio", "description": "Introduction to Amazon EMR Studio. In this video, we show how to:\n- Create a new workspace\n- Utilize cluster templates for EMR clusters\n- Connect EMR Studio to a GitHub repository\n- Execute parameterized notebooks\n\nLearn more about Amazon EMR Studio - https://amzn.to/3xhbuEj\n\nSubscribe: \nMore AWS videos - http://bit.ly/2O3zS75 \nMore AWS events videos - http://bit.ly/316g9t4\n\n#AWS #AWSDemo #AmazonEMRStudio", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "Amazon Web Services", "tags": [ "AWS", "Amazon Web Services", "Cloud", "AWS Cloud", "Cloud Computing", "Amazon AWS", "Amazon EMR", "Amazon EMR Studio", "Jupyter" ], "categoryId": "28", "liveBroadcastContent": "none", "localized": { "title": "Intro to Amazon EMR Studio", "description": "Introduction to Amazon EMR Studio. In this video, we show how to:\n- Create a new workspace\n- Utilize cluster templates for EMR clusters\n- Connect EMR Studio to a GitHub repository\n- Execute parameterized notebooks\n\nLearn more about Amazon EMR Studio - https://amzn.to/3xhbuEj\n\nSubscribe: \nMore AWS videos - http://bit.ly/2O3zS75 \nMore AWS events videos - http://bit.ly/316g9t4\n\n#AWS #AWSDemo #AmazonEMRStudio" }, "defaultAudioLanguage": "en" }, "contentDetails": { "duration": "PT12M34S", "dimension": "2d", "definition": "hd", "caption": "false", "licensedContent": false, "contentRating": {}, "projection": "rectangular" }, "statistics": { "viewCount": "537", "likeCount": "9", "dislikeCount": "0", "favoriteCount": "0", "commentCount": "0" } } ```

Oura

Retrieve one of three different daily summaries from the Oura Ring API.

Supported Commands:

sleep - return sleep periods
activity - return activity summary
readiness - return readiness data

Environment Variables:

OURA_PAT - Your Oura personal access token

Examples:

Return sleep data for the past 7 days
```
docker run -e OURA_PAT \
    ghcr.io/dacort/crates-oura \
    sleep | head -n 1
```
Click to view JSON
```json { "awake": 2010, "bedtime_end": "2021-04-20T06:55:43-07:00", "bedtime_end_delta": 24943, "bedtime_start": "2021-04-19T23:24:43-07:00", "bedtime_start_delta": -2117, "breath_average": 16.25, "deep": 4290, "duration": 27060, "efficiency": 93, "hr_5min": [ 63, 63, 64, 65, 66, 66, 66, 67, 66, 66, 67, 70, 71, 70, 69, 69, 70, 69, 68, 68, 68, 67, 67, 66, 66, 66, 66, 66, 62, 64, 70, 68, 67, 66, 66, 67, 66, 66, 65, 66, 66, 67, 67, 67, 67, 70, 69, 71, 71, 69, 68, 66, 63, 65, 67, 66, 65, 64, 63, 62, 63, 65, 66, 65, 69, 72, 74, 72, 68, 68, 69, 68, 66, 62, 63, 62, 61, 60, 60, 58, 60, 57, 58, 57, 57, 58, 61, 64, 66, 61, 0 ], "hr_average": 65.68, "hr_lowest": 57, "hypnogram_5min": "4221111111222322221111222122223332222222223344333332222212222112333334233332222242222222444", "is_longest": 1, "light": 14730, "midpoint_at_delta": 11083, "midpoint_time": 13200, "onset_latency": 270, "period_id": 0, "rem": 6030, "restless": 29, "rmssd": 27, "rmssd_5min": [ 33, 27, 20, 19, 17, 17, 17, 17, 19, 22, 31, 28, 18, 19, 20, 19, 14, 15, 17, 14, 14, 15, 15, 18, 17, 17, 16, 40, 44, 32, 21, 26, 30, 27, 20, 20, 17, 27, 19, 16, 18, 22, 23, 24, 33, 23, 27, 19, 19, 19, 36, 29, 38, 29, 27, 27, 36, 31, 29, 34, 29, 28, 24, 36, 26, 23, 19, 17, 31, 30, 29, 30, 32, 47, 39, 30, 44, 53, 48, 41, 42, 65, 47, 62, 48, 39, 36, 22, 26, 36, 0 ], "score": 83, "score_alignment": 97, "score_deep": 88, "score_disturbances": 77, "score_efficiency": 98, "score_latency": 72, "score_rem": 91, "score_total": 77, "summary_date": "2021-04-19", "temperature_delta": 0.06, "temperature_deviation": 0.06, "temperature_trend_deviation": 0.29, "timezone": -420, "total": 25050, "activity_type": "sleep" } ```

Reddit API

I wanted to get my saved posts out of the Reddit API. It's a little tough to understand, and the saved posts aren't actually documented there...but after some hunting, I found that they can be accessed at /user/dacort/saved.json.

This functionality just fetches the most recent 100 saved posts every time. I considered adding a start_date filter, but don't need it right now. 😁

Supported Commands:

saved - list saved posts for <username>
search [subreddit] <search_term> - search all of Reddit or a specific subreddit (r/subreddit_name) for a search term
- Note that terms in quotes need to be surrounded in single quotes, e.g. '"cargo crates"'

Environment Variables:

CLIENT_ID - Client ID of a "script" type app
CLIENT_SECRET - App secret
USERNAME - Your Reddit username
PASSWORD - Your Reddit password

Slack Web API

The Slack Web API is intended for use with ad-hoc queries and I use it to query basic info about some of the channels I'm in.

Supported Commands:

channels - list all channels in a Slack team
search - search for a keyword in your Slack team

Environment Variables:

SLACK_TOKEN - Some sort of Slack token that starts with xox...

If using a browser-based token (starts with xoxc) you need the d and (sometimes) d-s cookies from dev tools

var localConfig = JSON.parse(localStorage.localConfig_v2)
localConfig.teams[localConfig.lastActiveTeamId].token

SLACK_COOKIE_TOKEN - d cookie
SLACK_DS_COOKIE - d-s cookie

Examples:

Get a list of channels for the workspace the token is associated with

docker run -e SLACK_TOKEN -e SLACK_COOKIE_TOKEN \
    ghcr.io/dacort/crates-slack channels

Perform a search in the workspace the token is associated with

docker run -e SLACK_TOKEN -e SLACK_COOKIE_TOKEN \
    ghcr.io/dacort/crates-slack search dacort

Stack Overflow

The Stack Exchange API can be used to query sites across Stack Exchange. I'm using it to query for specific tags to keep up-to-date with conversations.

Supported Commands:

search - search for questions with one or more tags on Stack Overflow

Environment Variables:

n/a - The API doesn't require authentication, but you may get limited by your IP address

Examples:

Get a list of questions tagged amazon-emr

docker run \
    ghcr.io/dacort/crates-stackoverflow search amazon-emr

Note: Tags are an AND constraint, so if you search for multiple it will only return posts that contain up to 5 tags.

Perform a search for questions tagged amazon-emr and emr-serverless

docker run \
    ghcr.io/dacort/crates-stackoverflow search "amazon-emr;emr-serverless"

Twitter

Only a couple Twitter endpoints are defined at this point.

Supported Commands:

followers - return list of followers for a given screen name
users/show - return user profile data for a given screen name

Environment Variables - create a Twitter app and define the 4 different variables.

CONSUMER_KEY
CONSUMER_SECRET
ACCESS_TOKEN_KEY
ACCESS_TOKEN_SECRET

You can store all these environment variables in a file and reference that file with Docker.

Examples:

Return @dacort's profile

docker run --env-file .env \
    ghcr.io/dacort/crates-twitter \
    users/show dacort

{ "id": "99723", "name": "Damon Cortesi", "username": "dacort" }

Writing output to Amazon S3

The original idea behind this repo was that I could run these containers and easily output the resulting data to templated paths on S3.

So I built another tool, called Forklift, that comes bundled with each Cargo Crate.

If you run the container normally, the output gets printed to stdout. However, if you provide a FORKLIFT_URI environment variable, the data will get written to the S3 path provided. You can templatize parts of the path with JSON keys or a couple helper functions.

For example, the command below will upload my profile info to s3://<BUCKET>/forklift/twitter/dt=2021-05-15/dacort.json

docker run \
    -e FORKLIFT_URI='s3://<BUCKET>/forklift/twitter/dt={{ today }}/{{json "username"}}.json' \
    -e AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY \
    --env-file .env \
    ghcr.io/dacort/crates-twitter users/show dacort

Note that we passed AWS credentials as environment variables. You will either need to do that or run the container in AWS using instance profiles.

Some higher level thoughts after a few implementations

The data is intended to be streamed
- Output is always a JSON stream
Each implementation is intended to be completely independent
- They're all Python...they could be anything else!
Each implementation is intended to be as lightweight as possible
- Don't force unnecessary requirements into a crate

Next steps

Formalize the executor API and base class
- I want a way to be able to register commands, their parameters, and required environment variables
Formalize the output API and stdout/S3 classes
- Right now it's pretty hardcoded into the GitHub crate

Maybe someday

A generic "auth" command for crates

Resources

Python project structure
Minimal Docker containers for Python
(ended up not using, but) IO Streams with Python
For building matrix builds

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
.github/workflows		.github/workflows
crates		crates
.dockerignore		.dockerignore
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Cargo Crates

Usage

Caveats

Supported Services

GitHub

YouTube

Oura

Reddit API

Slack Web API

Stack Overflow

Twitter

Writing output to Amazon S3

Some higher level thoughts after a few implementations

Next steps

Maybe someday

Resources

About

Uh oh!

Releases

Packages

Uh oh!

Uh oh!

Languages

License

dacort/cargo-crates

Folders and files

Latest commit

History

Repository files navigation

Cargo Crates

Usage

Caveats

Supported Services

GitHub

YouTube

Oura

Reddit API

Slack Web API

Stack Overflow

Twitter

Writing output to Amazon S3

Some higher level thoughts after a few implementations

Next steps

Maybe someday

Resources

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Languages

Packages