NOTES

NOTES
Similar services: iron.io ironworker, aws lambda, stdlib.com, webtask.io, google functions, 
azure functions, ibm openwhisk, graph.cool, openFaas, fx, now, hook.io

APACHE CUSTOM PACKAGE
Deprecated, but interesting:
To make package of custom apache24: build port, then run "make package".  Package tar will be in ./work/pkg

MONGODB
create a backup in dump/ in pwd: mongodump -u '<userwithbackuprole>' -p 'xxxxxx'
restore a backup from dump/ in pwd: 
Dev mongo shell: mongo --username bigcgi --password bigcgi --authenticationDatabase bigcgi-main bigcgi-main

OTHER TECH TIDBITS
bottle-cork has been added to a vendor/ folder.  Its customized for bigcgi.

PKG ON FREEBSD
Sometimes pkg gets corrupted by ansible.  When it does this, it cant find or install
packages.  to fix:
/usr/local/sbin/pkg-static update -f

POTENTIAL DEMO APPS
formspree.io clone
blog software (search/filter on static)
scientific method project management

ARCHITECTURE 2

Architecture 2 is a planned overhaul of the internal bigcgi structure.  Basically, it replaces user
directories and permissions as the sandboxing method and replaces it with jails.

Currently, when a user is provisioned to bigcgi, they get a user on the freebsd system and a home
directory, where their scripts live.  A celery task populates it with files that are stored in the
mongo database.  When a user runs a app, bigcgi goes through a bunch of python subprocess calls to
drop the permissions to the user and run the CGI app.

This was good for the POC but it has several security issues.  First, having users on the system
directly is problematic.  Second, it requires the webapp to have sudo access to the scripts that
add users, their home directories, and moving files around.

Architecture 2 breaks this up by instead using FreeBSD jails as the sandboxes.  On user creation,
they are provisioned a jail on a host, and their files are moved there.  This provisioning doesn't
happen from the webapp itself, but via a celery worker.  This requires the redis queue to be
locked down and secured.  After provisioning, files are moved in a similar manner.  When a
user runs an app, it is dropped on the celery queue, and a worker jexec's a bigCGI script that
lives inside the jail to usher the request to the app via the CGI standard.

In addition to changing the user management piece of bigcgi, the application itself will also
change.  There will no longer be synchronous calls, only async.  Also, it will need the concept
of an "app run" that will store payloads, running statistics, outputs/logs, and debug information.
This should be availabe in the UI.

I'm not sure when I'll get around to this, so I wanted to write it down here so I wouldn't forget.
ezjail / jailme

DEVLOG: 

-- 9-28-2017 --
Today I spent multiple hours working on a strange bug.  I was getting 403 Forbidden responses
from bigCGI.  This was only affecting the development vagrant box; production was unaffected.
I spent hours looking at the file permissions, rebuilding the apache package, playing with different
Vagrant boxes, disabling suExec, disabling userdir, reconfiguring both to no avail.  I compared config
on production and in the dev environment and almost everything was the same.  I spent a lot of time
looking into why the suexec log didn't exist, hence the recompiling of the package with different
options.

The problem turned out to be a difference in home directory links.  In production, /home is a link to
/usr/home, on this vagrant box, it was not.  UserDir was configured to go to /usr/home/*/public_html,
which, if the /home directory is a link to that location, works fine.  Otherwise, it is trying to serve
CGI from a directory it doesn't have permission to, aka 403.

It should be noted that this was not an issue before, as I had to change my Vagrant box from
10.3-STABLE to 10.3-RELEASE due to a recent update that broke the build.  **LESSONS LEARNED** -
FreeBSD continues to bite me, due to their crappy support for containerization/virtualization outside
of their jails.  Props to me for sticking with it, even though I knew it would boil down to something
ridiculous.  Dev logs are cathartic.

-- 12-17-2017 --
Today I pushed a commit to production that eliminates the Apache server for running CGI programs.  This
is a really big step, and I'm very proud of it.  When I set out to do logging for apps, I struggled for
a long time with Apache trying to get user/app specific logging.  I felt like I tried everything to
no avail; I floundered for a long time.  In the end I did the "unthinkable"; got rid of Apache and replaced
it with a custom CGI server implemented in Python.  I want to make note of this day because I took a
seemingly unsurmountable problem and solved it by completely abandoning what I thought was set in stone.
It gives me confidence that I can solve any problem thrown my way so long as I keep a creative open mind.

-- 05-10-2018 --
Today I pushed a change to prod that allows users to upload read-only files for library and config files.
It was a massive effort months in the making, and deployment was a bit of a pain, but I'm happy the way
it turned out.  I'm sure I'll be doing debugging for a little, this feels like a big relief already.
This deployment taught me a few things; 1, bottle-cork is in a bad state and I need to fork it.  2,
I need to start utilizing jails so I can build my environment from scratch each time and set up true
staging environments.  I will work on that next, after I do branding and marketing work and before any
new big features.

-- 03-14-2021 --
Today I'm finishing up adding Ansible to improve the time it takes to provision a bigcgi node.  This
is super nice; I can now install mongodb, pound (replacing apache), and bigcgi in a single command.
I plan on using ansible as a generic tool runner as well.  This is apart of my bigger plan of what I'm
calling "architecture 2", which will be improving on the overall architecture.  In short, instead of
using users and homedirs as the sandboxing method, which was nice for POC but not realistic, I will be
using FreeBSD jails.  I'm also switching bigCGI to entirely async job calls; no more "app hosting".
Adding ansible is a big step forward to have a better development environment, and eventually reduce
risk if/when I deploy it again.