Google's officially supported Node.js client library for accessing Google APIs. Support for authorization and authentication with OAuth 2.0, API Keys and JWT (Service Tokens) is included.
The maintner and samplr workers both make a significant number of calls to GitHub, and measuring and observing the applications's access is critical to long term success.
Metrics that need to be tracked at a minimum:
Engineering Protip: Make state someone else's problem. -James Ward
It is probably time to introduce some sort of state (read that as "database") to our solution.
Reasons to introduce a database:
samplrdis the only "source of truth" for both reads and writes for Snippets. Should the pod go down (for whatever reason) requests for snippets will result in either errors or 0 snippets returned.
There are some good arguments for using a Relational database:
There are two main arguments for using a Graph DB:
Some Issues tracked in maintner have so many comments in them, it is impossible to return them in a single RPC. To that end, we need to implement a new "GetComments" rpc for maintner that supports pagination
The Makefiles are getting better but need improvement
export-ing variables in a shell doesn't scale well)
As the number of keys in the cluster increases, observability to which keys are available and what their capacity is will be.... key. This necessitates the creation of an application to monitor these activities.
Note: This will be dependent on the work done in #14
The application should:
sprvsr module only updates the list of tracked repositories when told to do so via an HTTP request.
This should be moved to a periodic check every N minutes, with the added feature of being able to be told to sync "now"
The sprvsr module has been useful for working across both
samplr, but currently as the module defines it's service and exposes an api, it is not as reusable or as portable as it needs to be moving forward.
Migrate the sprvsr module to a 'corpus like' model which is stateful and updates itself on an interval, removing the api component from it entirely.
samplr-rtr does not implement the
ListRepositories rpc specified in the proto.
To do so we need to
samplr-rtrpods can run as so that they may have
viewpermissions on services
TrackedRepositoriesbased on the labels (this should be guarded by a
RWMutex). Have the
ListRepositoriesmethod read from that list.
samplr-rtr should expose an internal-only gRPC service to signal the router to update it's list of available repositories. This rpc is called from
samplr-sprvsr when it updates the set of repositories.
Currently within the cluster, our services are communicating over the wire without any encryption or security. While this is fine, adding mTLS (possibly via Istio) would add an extra layer of security to the cluster.
The samplr service should expose a gRPC service to describe its status. Things the service should report:
As a part of the gRPC unification, the health checks across all services/deployments need to be done via gRPC health probe, and each application should register for the gRPC health service
The v0 api needs to be removed and the current v1 api needs to be renamed to v1beta1
samplr only supports tracking snippets on a branch named
There are plenty of repositories that do not use that naming convention which would benefit from snippet tracking.
To that end, I propose we add an additional GitHub API call when first Tracking a GitHub repository which queries the GitHub API for the "default_branch" of a given repository' and stores it in the
watchedRepository struct. This allows for repository owners to
samplrwill have stale data and requires a refresh.
Adds new RPC to ListGitHubComments (and associated message contracts).
Allows for filtering and pagination
Adds Terraform config files to automate GCP project and resource creation.
This will (hopefully) obviate the need for awkward make file scripts and allow for continuous delivery and project collaboration.
In the GraphQL API for GitHub, it is possible to encounter repositories that are considered "not existing" as we don't have api access to them (they could be private).
As opposed to erroring out and finishing the sweep job, simply log a warning and continue
Scales number of replicas to 5 and adds a pod anti affinity policy that requires pods for maintner-rtr to be scheduled on different Nodes in the pool. This is to help reduce single point of failure modes in the pods where either a single pod is unhealthy and cannot serve or where a single Node is unhealthy and cannot serve.
Add utility to individually mark issues as tombstoned