31 posts tagged with "Technology and Programming"

Motivation and Contents​

If you play fantasy football on Sleeper and have ever wanted to perform data analysis without orchestrating spreadsheets and navigating the Sleeper API, this is a completely free solution for you. If you found this blog post, I assume you are comfortable writing a bit of code and enjoy exploring data. That said, I don’t assume that you’re familiar with Databricks. The first section walks through deployment steps, and the rest is a step-by-step guide to building the solution yourself, from software installation to data ingestion to building an ETL pipeline to visualizing data in a dashboard.

The Rise of Speculative Wealth Traps​

Before diving into this post, I want to clarify my specific beliefs regarding sports betting and prediction markets. In a vacuum, I have no moral qualms with the principle of gambling. I’ve certainly laid down some sports bets while in Las Vegas. That said, I do not play daily fantasy sports (DFS) or engage in any sort of online betting or prediction markets. Mirroring many of my thoughts regarding social media, I personally feel that unfettered online access to the dopamine hits of gambling is deeply unhealthy for the human psyche. In addition, I believe that the potential for sudden personal wealth via highly speculative risks (much like cryptocurrencies) attracts so many willing participants due to the rapidly vanishing middle class and the increasing scarcity of quality job opportunities. As the already massive global wealth inequality continues to widen, I view these as yet another transfer of capital from the middle and lower classes back to corporations and executives. As a diehard sports fan, I am also profoundly disappointed at how intertwined sports media and betting have become. People today simply cannot watch any sports coverage without being bombarded with online betting ads. In fact, I fear that new sports fans are now unable to distinguish between betting and the sports themselves. Kids and teenagers today only know betting to be woven into the very fabric of sports.

I’m tired. Existentially.

As a senior engineer with nearly 13 years of professional experience, I’ve reinvented myself countless times. I’ve seen more technologies than I can count or remember come and go. I’ve seen bubbles form and burst. I remember when the latest flop was supposed to be the next big thing. I’ve spent countless hours learning new skills and honing my craft in a knowingly futile effort to stay up-to-date with technology. I’ve seen layoffs and downturns. Trust me, I understand that the macroeconomics of the tech industry has peaks and valleys. After all, I wrote my first Hello, world! in high school shortly after the .com crash. However, something feels different this time. I’m starting to feel deeply disillusioned about the future of AI.

Motivation and Background​

While I’ve used Kubernetes professionally in a few capacities (particularly in customer engagements while working at AWS), I wanted to cement my knowledge and increase my mastery with a systematic approach. I decided to prepare for the Certified Kubernetes Application Developer (CKAD) exam. I’ve taken and passed more than a dozen technology certification exams spanning AWS, Azure, HashiCorp, and more. This exam is unique in several ways. Namely, it’s all hands-on in a lab environment. Azure exams often have a coding, configuration, or CLI command component, but even these are typically multiple-choice questions. The CKAD presents you with a virtual desktop and several Kubernetes clusters, making you tackle 15-20 tasks with a strict two-hour time limit. I put together this repository and post for a few reasons:

• I wanted to document all of my hands-on preparation for when I have to recertify in two years
• I wanted to share my knowledge with others and offer a supplemental guide to a CKAD course
• Since the CKAD exam focuses on Kubernetes from a cloud-agnostic perspective, I wanted to fill in the gaps in my own knowledge of running Kubernetes in the AWS ecosystem (e.g., Karpenter, Container Insights, etc.)
• Many courses and guides leverage Microk8s or minikube to run Kubernetes locally, but I wanted to focus on cloud-based infrastructure, especially for things like EBS volumes created via PVCs, ELBs created via a Service, etc.

In summary, this material focuses on hands-on exercises for preparing for the exam and other tools in the cloud-agnostic and AWS ecosystems.

About the Book​

Recently, I started reading a fantastic book called The Nature of Code by Daniel Shiffman. From the description:

How can we capture the unpredictable evolutionary and emergent properties of nature in software? How can understanding the mathematical principles behind our physical world help us to create digital worlds? This book focuses on a range of programming strategies and techniques behind computer simulations of natural systems, from elementary concepts in mathematics and physics to more advanced algorithms that enable sophisticated visual results. Readers will progress from building a basic physics engine to creating intelligent moving objects and complex systems, setting the foundation for further experiments in generative design.

Daniel implements numerous examples using a programming language called Processing. Instead, I decided to write my own versions using JavaScript, React, Three.js, and D3. For this blog series, I intend to implement my learnings from each chapter.

About​

This is a tool that I developed and open sourced at AWS. Find the latest in the GitHub repository here. It’s released under MIT-0.

AWS Billing Conductor (ABC) Overview​

AWS Billing Conductor is a priced service in the AWS billing suite designed to support showback and chargeback workflows for any AWS customer who needs to enforce visibility boundaries within their Organization or add custom rates unique to their business. This alternative version of the monthly bill is called a pro forma bill.

Solution Overview​

As I wrote about previously, AWS users are shifting left on costs using DevOps and automation. While tools like Infracost are powerful for estimating costs for Lambda and other services, they alone do not provide optimization or tuning feedback during the development lifecycle. This is where a tool like AWS Lambda Power Tuning assists:

AWS Lambda Power Tuning is an open-source tool that can help you visualize and fine-tune the memory and power configuration of Lambda functions. It runs in your own AWS account, powered by AWS Step Functions, and it supports three optimization strategies: cost, speed, and balanced.

Lambda pricing is determined by the number of invocations and the execution duration. There are several strategies for decreasing duration costs including using Graviton for 20% savings (which this solution does for both Lambda and CodeBuild), leveraging the latest runtime versions, taking advantage of execution reuse, etc. In addition to these, optimizing memory allocation is a key mechanism for efficiency. From the documentation:

The [duration] price depends on the amount of memory you allocate to your function. In the AWS Lambda resource model, you choose the amount of memory you want for your function and are allocated proportional CPU power and other resources. An increase in memory size triggers an equivalent increase in CPU available to your function.

Without running the Lambda function using different configurations, it is unclear what is the most optimal memory amount for cost and/or performance. This solution demonstrates AWS Lambda Tuning Tools integration with a CodeSuite CI/CD pipeline to bring Lambda tuning information to the pull request process and code review discussion. The source code is hosted on GitHub.

About the Book​

Recently, I started reading a fantastic book called The Nature of Code by Daniel Shiffman. From the description:

How can we capture the unpredictable evolutionary and emergent properties of nature in software? How can understanding the mathematical principles behind our physical world help us to create digital worlds? This book focuses on a range of programming strategies and techniques behind computer simulations of natural systems, from elementary concepts in mathematics and physics to more advanced algorithms that enable sophisticated visual results. Readers will progress from building a basic physics engine to creating intelligent moving objects and complex systems, setting the foundation for further experiments in generative design.

Daniel implements numerous examples using a programming language called Processing. Instead, I decided to write my own versions using JavaScript, React, Three.js, and D3. For this blog series, I intend to implement my learnings from each chapter. This first post covers the introduction section of the book.

Solution Overview​

My current role focuses on every facet of AWS cost optimization. Much of this entails helping to remediate existing infrastructure and usage. Many customers ask how they can shift left on cloud costs, like they do with security. Ultimately, cost consciousness needs to be injected into every aspect of the engineering lifecycle: from the initial architecture design to implementation and upkeep.

One such aspect is providing developers visibility into the impact of their code changes. Infrastructure as code has made it easy to deploy cloud resources faster and at larger scale than ever before, but this means that cloud bills can also scale up quickly in parallel. This solution demonstrates how to integrate Infracost into a deployment pipeline to bring cost impact to the pull request process and code review discussion. The source code is hosted on GitHub.

The Course and Companion​

Rahul Nath recently released a course called AWS Lambda for the .NET Developer on Udemy and Gumroad. I had a ton of fun going through the exercises and highly recommend purchasing a copy. While working through the material, I implemented the solutions with infrastructure as code using AWS CDK in C# and .NET 6. I also containerized most of the Lambda functions and wrote unit tests for both the functions and infrastructure. You can find all my source code on GitHub.