farza and tim cook eat lasagna together in bikini bottom
You know, it’s been a long time since I’ve written a tech focused weekly email. Some of the oldest subscribers of my email remember all the heavy tech stuff I used to write about back in January 2019 - deep learning, GPU endpoints, synthetic datasets, setting up optimal local environments, AWS deployments, etc. I was actually reading my old emails and damnnnnn I sounded so smart I really fooled you guys.
So why did all the tech heavy stuff stop? Well. Honestly, the tech stuff just stopped becoming a problem really fast. It’s funny. Usually the hard part about a tech company isn’t the tech. The hard part is getting people to actually use that stuff you made.
In any case, you need to have a really good tech foundation if you want to move fast at a tech company (surprise surprise). If you all do is keep hacking shit together without thinking coding becomes miserable and the ability to move fast is rekt because every time one person on the team makes a code change the code breaks for the other 4 engineers LOL.
It took us a couple of tries, but I think the tech stack at Kanga is really good at this point. It’s nothing crazy. In fact, it’s all pretty basic. That’s what makes it dope.
It’s especially good for small teams (5-20 people) who just want to bang out code, ship product, and iterate.
Here are some of our strategies!
Local Environments
Having a really clean and simple developer experience is a massive reason we move so fast. By making it insanely easy to get code running on an engineers machine the engineer can focus on actually fixing bugs or adding features instead of all the BS related to just getting the code running.
To get your local Kanga environment running, you run one script called “local.sh”. (here is the full file if you’re curious). This is something Evan came up with and it works super well. Here’s what it looks like when you run it:
That’s it.
One bash script. No BS around installing libraries, the right versions of languages, or anything like that. The script handles database migrations, building/starting local Docker containers, and installing any new modules the codebase may have. Did someone make changes to the codebase? Pull the changes down via git. Run local.sh again. Done. Now you have the latest code running on your machine.
Docker is 90% of the reason why working locally is so good at Kanga.
Docker makes it so ezpz to ensure that every engineer on the team is working in the same environment. If it works on one engineer’s machine, it will work on every engineer’s machine.
All I need to is write a Dockerfile (example of a Dockerfile we use to build our Go API). Then, I hook it all up to a docker-compose file that looks like this:
version: "3"
services:
redis:
image: redis
container_name: redis-k3
ports:
- "6379:6379"
postgres:
image: postgres:10
container_name: postgres-k3
ports:
- "5432:5432"
api:
build:
context: services
dockerfile: api-Dockerfile
container_name: api
ports:
- "80:80"
depends_on:
- postgres
- redis
And that’s it. I tell Docker: “Yo. Give me Redis, give me Postgres 10, build my API, and set up all the ports”. Easy. Each developer doesn’t need download the right versions of Go, Redis, and Postgres separately and maintain each one separately. It’s all just setup.
Everyone at Kanga really values a good developer experience. This includes everything from setting up your local environment to deploying to production. If the developer experience is trash you won’t be as productive and you probably won’t have as much fun. So, that means it’s really on every person to make small optimizations as the codebase changes.
The Backend
Our API is in Go hosted on EC2. I’ve only ever written APIs w/ Go, NodeJS, Django, Flask, and Rails. Go is by far my favorite. Initially we chose Go because Evan was super comfy with it. I trusted him and I’m glad I did because holy fuck this language is insanely good in so many ways. It’s good because:
Go is statically typed. Dynamically typed languages are great in some places, for example where you want to quickly code up a neural network in Python and don’t want to spend forever figuring out what the type of each variable needs to be. But I don’t think dynamically typed languages belong in a production API where the type of the variables you’re including in your payload or inserting in to your DB matters a lot. Also, by being statically type you can save yourself from a lot of stupid mistakes related to typing because, well, the code won’t even compile until you fix it.
Building an endpoint is ezpz. With Go, all you have to do is specify the structure of the payload, the types, and the names of the fields. Extremely transparent. Easy to read. No questions on what the payload for an endpoint is. It’s clear af. The best part is, you can also use this same struct to insert data to your DB if you decide to use an ORM. Inserting to your DB can be as easy as “User.save()”. We use GORM as our ORM and it’s been lovely :).
type User struct {
ID uint `json:"id"`
Name string `json:"name"`
Age uint `json: "age"`
Email string `json: "email"`
EncryptedPass []byte `json: "-"`
}
// The above struct will return a payload that looks like:
// {
// "id": 69,
// "name": "Farza",
// "age": 7,
// "email": "elon@musk.com"
// }
Go has an extremely powerful standard library. You don’t need to install a bunch of random open source packages to do basic shit like: working with “time”, making API requests, or even creating cryptographic hashes of passwords. Go has it all built in. It’s pretty insane. Your project doesn’t need to become a Frankenstein looking thing consisting of like 25 packages all with different dependencies.
It’s really fun to write. I know that sounds weird but it’s true. Go lets you spend time solving real problems and helps you avoid dumb problems caused by your own stupid mistakes or because of random issues with libraries.
So, basically, if you want to make an API. Use Go. You can move fast and rest assured that Go will scale really well if your product ends up getting huge. Note: I also like Rails, but I like Go more.
Database and Cache
For our DB we use PostgresSQL + AWS RDS and for our cache we use Redis + AWS ElastiCache. Postgres has been super flexible and worked really well for us. Zero complaints and I don’t think people should use any other database when starting out. It even lets you store JSON right in the table.
Zero complaints with AWS RDS and AWS ElastiCache as well. 10/10 services. Why host your DB and cache yourself when Amazon can just do it for you?
Don’t solve problems other people have already solved. No shame! The less code you gotta write the better tbh. This isn’t always the best strategy but in the early stages it almost always is.
For our migrations, we use Flyway. Honestly, I don’t know a lot about migrations and still don’t. But, Flyway makes it such that all I have to do is write the migration - and it will take care of all the edge cases.
BigData
We didn’t have time to learn what the fuck an Apache Hadoop was. We had a lot of data, needed to store it somewhere easily, and needed to be able to query it easily.
Why? For the original version of Kanga, we were storing the different things our computer vision algorithms detected. For example, at one point we had about 300 streams being processed concurrently. Each stream produced about 60 new data points per minute. So, that’s 25,920,000 data points per day across all 300 streams.
No one the team was familiar with the world of BigData and no one really wanted to figure the nitty gritty aspects of it. There were more important problems to solve than creating our own big data pipeline. So, we used Google BigQuery :). It’s worked great, has a fantastic API to insert/retrieve data, and has covered all out data warehousing needs.
This definitely illustrates one of our core philosophies: Don’t recreate the wheel unless you absolutely need to. No shame in using stuff other people made.
Deployments
To deploy at Kanga you all you do is run a script by the name of “prod.sh”. That’s it. Everything else is taken care of. Everyone has the power to deploy at anytime.
All our deployments are done by using Ansible which is a tool that makes it really easy to configure your servers without having to SSH into each one and manually run commands.
For example, lets say you have an API you want to deploy to three instances. Well, you’d need to SSH into each one of the three instances, install packages by hand, and start the API manually on each one. Seems extremely annoying and very error prone. Deploying to prod and getting the latest code to users should be easy and work properly every time without error.
This is where Ansible comes in.
Basically, I can write some YAML like so:
- name: Start.
hosts: AllInOne
tasks:
- name: Copy over generated server config
synchronize:
recursive: yes
delete: yes
src: ../server_data_folder/
dest: /home/ec2-user/server_data_folder
- name: Install the basics
yum:
name: "{{ packages }}"
state: present
vars:
packages:
- docker
- htop
- gcc
- python3
- python-pip
- jq
- awscli
- gcc
- name: Pull docker images
shell: docker-compose pull
- name: Bring up Docker containers
docker_compose:
recreate: "{{ 'always' }}"
project_src: server_data_folder
project_name: kanga3
What does this do? Well, first I specify the “hosts”. This is a list of IP addresses of the EC2 instances I want to deploy to. “AllInOne” is the name of a command line variable specifying these IP addresses. Using this, Ansible attempts to connect to each instance.
If that works out - it will start running “tasks” on each instance one-by-one. The first thing I do here is copy a file from my local machine called “server_data_folder” to the instances. Then, the next task is to install a bunch of packages on to the instances using “yum” which is the preferred package manager for Amazon Linux. The last thing I do is pull the latest Docker containers on to the instances, start them up, and we’re done!
The really cool thing here is imagine if I create a new server that I need to provision. All I need to do is run this Ansible file on it, and I’m done. The cooler thing is that Ansible doesn’t repeat itself. If you try to run the same Ansible file on an instance twice, it will recognize that everything is already done and won’t repeat the installation process. This is wildly useful because it ensures that all our instances will maintain the same state. That is a blessing.
Database and Cache
We don’t host these ourselves on the instance.
For our DB we use PostgresSQL + AWS RDS and for our cache we use Redis + AWS ElastiCache. Postgres has been super flexible and worked really well for us. Zero complaints and I don’t think people should use any other database when starting out. It even lets you store JSON right in the table. You sorta get the power of NoSQL which is nice.
Zero complaints with AWS RDS and AWS ElastiCache as well. 10/10 services. Why host your DB and cache yourself when Amazon can just do it for you? They solve all the hard problems of replication, backups, and scaling up.
Don’t solve problems other people have already solved. No shame! The less code you gotta write the better tbh. This isn’t always the best strategy but in the early stages it almost always is. The only goal should be getting it to users.
User Analytics
Segment + Mixpanel. That’s it. Mixpanel is low-key garbage and expensive as fuck, but, I mean it does the job. Segment is high-key amazing. Only use Mixpanel to actually do analytics on users. MP’s API is trash. Use Segment’s libraries to forward data to MP. I know it sounds weird but trust. Segment is really good at forwarding data. MP is really good at doing analytics.
Tracking users and seeing if people are actually using our shit is probably the most important thing to us, so we really invested in getting to know these two services and getting stuff working.
Other Stuff
We use Heroku a lot randomly when we want to quickly spin up an admin panel or some other one off service that users aren’t using directly.
Frontend is all React. It works great. Has a lot of support. Just use React.
Mobile is all done in React Native. It so easy. For like 95% of early products (that are mobile apps) I feel like React Native is perfect.
For GPUs use Google Cloud. They make it so easy to do everything from setting up CUDA to getting TFServing working ASAP.
We used DataDog for all things logging. Looking back, this was kinda overkill and we probably should have just used Amazon’s logging service. DataDog is really good but we just needed kinda good.