Motivation and Contents​

If you play fantasy football on Sleeper and have ever wanted to perform data analysis without orchestrating spreadsheets and navigating the Sleeper API, this is a completely free solution for you. If you found this blog post, I assume you are comfortable writing a bit of code and enjoy exploring data. That said, I don’t assume that you’re familiar with Databricks. The first section walks through deployment steps, and the rest is a step-by-step guide to building the solution yourself, from software installation to data ingestion to building an ETL pipeline to visualizing data in a dashboard.

The Rise of Speculative Wealth Traps​

Before diving into this post, I want to clarify my specific beliefs regarding sports betting and prediction markets. In a vacuum, I have no moral qualms with the principle of gambling. I’ve certainly laid down some sports bets while in Las Vegas. That said, I do not play daily fantasy sports (DFS) or engage in any sort of online betting or prediction markets. Mirroring many of my thoughts regarding social media, I personally feel that unfettered online access to the dopamine hits of gambling is deeply unhealthy for the human psyche. In addition, I believe that the potential for sudden personal wealth via highly speculative risks (much like cryptocurrencies) attracts so many willing participants due to the rapidly vanishing middle class and the increasing scarcity of quality job opportunities. As the already massive global wealth inequality continues to widen, I view these as yet another transfer of capital from the middle and lower classes back to corporations and executives. As a diehard sports fan, I am also profoundly disappointed at how intertwined sports media and betting have become. People today simply cannot watch any sports coverage without being bombarded with online betting ads. In fact, I fear that new sports fans are now unable to distinguish between betting and the sports themselves. Kids and teenagers today only know betting to be woven into the very fabric of sports.

A Hill Worth Dying On​

I am living proof of the American Dream, though I fear that my story is becoming less and less common. I was born and raised in Victoria, Texas, a city of 65,534 as of the 2020 census. In this blue-collar town, I grew up in a working-class home. My father is one of 14 children to immigrant parents and spent the vast majority of his life working in the brutal oilfield elements to provide for his family. My mother, also the child of an immigrant, worked as a substitute teacher. They often reminded me to pursue what my father called an ‘office job,’ which, in hindsight, represents socioeconomic betterment. Although I have now worked in the technology industry for 13 years, being a software engineer wasn’t a vocation that I even knew existed when I was growing up. I was fortunate enough to take a Java programming class in 2004, which radically altered the path of my life. So much so that I spent four years volunteering by teaching AP computer science at various high schools in Houston from 2018 to 2022 to pay this forward. Education is core to my beliefs and identity, and in many ways, that started at the University of Texas at Austin.

I’m tired. Existentially.

As a senior engineer with nearly 13 years of professional experience, I’ve reinvented myself countless times. I’ve seen more technologies than I can count or remember come and go. I’ve seen bubbles form and burst. I remember when the latest flop was supposed to be the next big thing. I’ve spent countless hours learning new skills and honing my craft in a knowingly futile effort to stay up-to-date with technology. I’ve seen layoffs and downturns. Trust me, I understand that the macroeconomics of the tech industry has peaks and valleys. After all, I wrote my first Hello, world! in high school shortly after the .com crash. However, something feels different this time. I’m starting to feel deeply disillusioned about the future of AI.

Motivation and Background​

While I’ve used Kubernetes professionally in a few capacities (particularly in customer engagements while working at AWS), I wanted to cement my knowledge and increase my mastery with a systematic approach. I decided to prepare for the Certified Kubernetes Application Developer (CKAD) exam. I’ve taken and passed more than a dozen technology certification exams spanning AWS, Azure, HashiCorp, and more. This exam is unique in several ways. Namely, it’s all hands-on in a lab environment. Azure exams often have a coding, configuration, or CLI command component, but even these are typically multiple-choice questions. The CKAD presents you with a virtual desktop and several Kubernetes clusters, making you tackle 15-20 tasks with a strict two-hour time limit. I put together this repository and post for a few reasons:

• I wanted to document all of my hands-on preparation for when I have to recertify in two years
• I wanted to share my knowledge with others and offer a supplemental guide to a CKAD course
• Since the CKAD exam focuses on Kubernetes from a cloud-agnostic perspective, I wanted to fill in the gaps in my own knowledge of running Kubernetes in the AWS ecosystem (e.g., Karpenter, Container Insights, etc.)
• Many courses and guides leverage Microk8s or minikube to run Kubernetes locally, but I wanted to focus on cloud-based infrastructure, especially for things like EBS volumes created via PVCs, ELBs created via a Service, etc.

In summary, this material focuses on hands-on exercises for preparing for the exam and other tools in the cloud-agnostic and AWS ecosystems.

About the Book​

Recently, I started reading a fantastic book called The Nature of Code by Daniel Shiffman. From the description:

How can we capture the unpredictable evolutionary and emergent properties of nature in software? How can understanding the mathematical principles behind our physical world help us to create digital worlds? This book focuses on a range of programming strategies and techniques behind computer simulations of natural systems, from elementary concepts in mathematics and physics to more advanced algorithms that enable sophisticated visual results. Readers will progress from building a basic physics engine to creating intelligent moving objects and complex systems, setting the foundation for further experiments in generative design.

Daniel implements numerous examples using a programming language called Processing. Instead, I decided to write my own versions using JavaScript, React, Three.js, and D3. For this blog series, I intend to implement my learnings from each chapter.

About​

This is a tool that I developed and open sourced at AWS. Find the latest in the GitHub repository here. It’s released under MIT-0.

AWS Billing Conductor (ABC) Overview​

AWS Billing Conductor is a priced service in the AWS billing suite designed to support showback and chargeback workflows for any AWS customer who needs to enforce visibility boundaries within their Organization or add custom rates unique to their business. This alternative version of the monthly bill is called a pro forma bill.

Solution Overview​

As I wrote about previously, AWS users are shifting left on costs using DevOps and automation. While tools like Infracost are powerful for estimating costs for Lambda and other services, they alone do not provide optimization or tuning feedback during the development lifecycle. This is where a tool like AWS Lambda Power Tuning assists:

AWS Lambda Power Tuning is an open-source tool that can help you visualize and fine-tune the memory and power configuration of Lambda functions. It runs in your own AWS account, powered by AWS Step Functions, and it supports three optimization strategies: cost, speed, and balanced.

Lambda pricing is determined by the number of invocations and the execution duration. There are several strategies for decreasing duration costs including using Graviton for 20% savings (which this solution does for both Lambda and CodeBuild), leveraging the latest runtime versions, taking advantage of execution reuse, etc. In addition to these, optimizing memory allocation is a key mechanism for efficiency. From the documentation:

The [duration] price depends on the amount of memory you allocate to your function. In the AWS Lambda resource model, you choose the amount of memory you want for your function and are allocated proportional CPU power and other resources. An increase in memory size triggers an equivalent increase in CPU available to your function.

Without running the Lambda function using different configurations, it is unclear what is the most optimal memory amount for cost and/or performance. This solution demonstrates AWS Lambda Tuning Tools integration with a CodeSuite CI/CD pipeline to bring Lambda tuning information to the pull request process and code review discussion. The source code is hosted on GitHub.

Overview​

I learn best by doing, so with every release cycle, I take the time to build fully functional examples and digest the blog posts and video content. Below are some of my favorite releases from re:Invent 2022. You can find all source code in this GitHub repository.

About the Book​

Recently, I started reading a fantastic book called The Nature of Code by Daniel Shiffman. From the description:

How can we capture the unpredictable evolutionary and emergent properties of nature in software? How can understanding the mathematical principles behind our physical world help us to create digital worlds? This book focuses on a range of programming strategies and techniques behind computer simulations of natural systems, from elementary concepts in mathematics and physics to more advanced algorithms that enable sophisticated visual results. Readers will progress from building a basic physics engine to creating intelligent moving objects and complex systems, setting the foundation for further experiments in generative design.

Daniel implements numerous examples using a programming language called Processing. Instead, I decided to write my own versions using JavaScript, React, Three.js, and D3. For this blog series, I intend to implement my learnings from each chapter. This first post covers the introduction section of the book.