Home | History | Annotate | Download | only in repodiff 
      1 # To Setup Application:
      2 
      3 1.  It is assumed that the Go runtime has been installed with a properly set
      4     $GOPATH
      5 2.  Run `make bootstrap`
      6 
      7 # To Provision a Database
      8 
      9 **Use Google Cloud**. This is one assumption made based on the idea that Google
     10 Data Studio will visualize the data.
     11 
     12 1.  Log in to [the Google Cloud Platform](https://pantheon.corp.google.com)
     13 2.  Under the **Storage** section, navigate to **SQL**
     14 3.  Click **Create Instance**, select **MySQL** and hit **Next**
     15 4.  Choose **MySQL Second Generation** (the default)
     16 5.  Set Instance ID and root password, then click **Create**
     17 6.  Wait for the instance to initialize, then navigate to **databases** and then
     18     click **create database**; choose the defaults and make note of the chosen
     19     database name.
     20 7.  Navigate to **Users** and **Create user account** for the purpose of
     21     creating a non-root user to log into the database; Make note of the username
     22     and password chosen.
     23 
     24 The application assumes that both a development and production environment
     25 exist, therefore the above steps will need to be completed a second time. Now
     26 the following environment variables will need to be set in your application
     27 environment:
     28 
     29 *   GCP_DB_INSTANCE_CONNECTION_NAME_DEV: The instance ID of the provisioned
     30     database server; this is listed on the SQL homepage in Google Cloud
     31     alongside the respective database instance
     32 *   GCP_DB_NAME_DEV: the database name created from step 6
     33 *   GCP_DB_USER_DEV: the username created from step 7
     34 *   GCP_DB_PASSWORD_DEV: the password created from step 7
     35 *   GCP_DB_PROXY_PORT_DEV: an arbitrary, unique port used for the local secure
     36     MySQL proxy
     37 
     38 The same environment variables should be set for production. The names are the
     39 same but replace **"DEV"** with **"PROD"**
     40 
     41 One the environment variables are set, the database can be readied by running:
     42 
     43 `make db_upgrade`
     44 
     45 To run the same set of upgrades for production, run:
     46 
     47 `ROLE="prod" make db_upgrade`
     48 
     49 # Running the Application
     50 
     51 Executing the application will by default:
     52 
     53 1.  Clone all of the repositories specified in config.json and write the
     54     observed differentials to the database
     55 2.  Run application-specified analytics on the persisted data and write the
     56     output to designated denormalized tables; those tables are tightly coupled
     57     to Datastudio data sources in the [AOSP Branch
     58     Differentials](https://datastudio.google.com/c/u/0/org/UTgoe29uR0C3F1FBAYBSww/reporting/1lTzMXHBWiVvc0Dnb2DJvRQeTVCAIuvGF/page/9rqP)
     59 3.  Generate CSV reports based on persisted data
     60 
     61 Applications steps can be optionally disabled for presumably one-off runs:
     62 `ROLE="prod" ./repodiff
     63 [--execute-diff=false][--denormalize-data=false][--generate-report=false]`
     64 
     65 `config.json` will specify input and output parameters for the application;
     66 update to reflect your own environment.
     67 
     68 To run:
     69 
     70 `make run`
     71 
     72 For production: `ROLE="prod" make run`
     73 
     74 # Tooling
     75 
     76 To connect to the MySQL server used by the application, run:
     77 
     78 `make db_shell`
     79 
     80 To create new, canonical migration scripts to update the database schema, run:
     81 
     82 `make sql_script`
     83 
     84 To undo the latest database migration, run:
     85 
     86 `make db_downgrade`
     87 
     88 To upgrade to the latest database version, run:
     89 
     90 `make db_upgrade`
     91 
     92 To run tests, run:
     93 
     94 `make test`
     95 
     96 # Consuming the Application Output
     97 
     98 TLDR: Use [Google Data Studio](https://datastudio.google.com); Your datasource
     99 will be tables from the provisioned database set up in the provisioning
    100 instructions. All intended consumable tables are prefixed with
    101 **denormalized_view_**
    102 
    103 The rationale behind the application is that setup in Data Studio should require
    104 little to no learning curve (for developers especially), and views are simple
    105 projections of the underlying model. The application, then, should run whatever
    106 necessary logic to produce desired analytics that can be written to a
    107 denormalized database table for straightforward consumption.
    108