Code That Rocks

Private and public ingress + LetsEncrypt

Wed, 11 Nov 2020 00:39:50 -0600

My home setup is a full-blown Kubernetes cluster (Overkill some might say, but it’s what I do, so it makes sense to have one). As part of this, I have load balancing, provided by MetalLB, and routing to my applications thanks to Nginx Ingress. This is all well and good, and setting up public sites and private sites behind my router, while challenging, has been largely successful. Except for one particularly challenging bit of configuration that, for whatever reason, escaped me until just recently.

Some of the services I run on my cluster are Drone CI (For CI/CD), and a dedicated Docker Registry (thanks to Harbor). For Drone in particular, the configuration is slightly more complex than a normal service. There is an application server, which is fronted by a private-only ingress, and there is a portion of the application that responds to webhooks from GitHub to trigger events, which must be exposed via a public ingress. This means the application has both public _and_ private ingresses, and that anything inside the cluster needs to route its traffic to the private side. The trouble here is when you mix in LetsEncrypt (via Cert Manager). The certificate generated needs to be tied to your public ingress, not your private one, or the HTTP01 validation won’t work, but that’s normal and expected. To do that we can specify the cert-manager.io/cluster-issuer on the ingress that is public, and share the secret between the two. Problem is: Cert Manager needs to validate the temporary ingress it creates as part of its “propagation check”. This will check that DNS resolution is working and the HTTP01 validation will work. However, the hostname that will be resolved here is the same one on the public side as the private side, and inside the cluster it will always resolve to the IP of the internal ingress controller, not the public ingress controller hosting the validation ingress.

One fix to this (and one I used for a while to get by) is to set the DNS resolver to route to the public IP while Cert Manager does its thing. This works, but it then prevents everything on the inside network from reaching the private service at the same time. Only one thing can work at a time in this scenario (And for the last year or so, every three months I’d swap it over, renew my certs, and swap it back). The real fix is to update _only the Cert Manager’s pod_ to resolve to the public IP, and permitting everything else internally to resolve as normal.

I achieved this with the following config:

hostAliases:
  - hostnames:
    - drone.codethat.rocks
    - registry.codethat.rocks
ip: 10.0.10.2

This leverages the Kubernetes built-in functionality to effectively peg the /etc/hosts file in the pod with some overridden DNS resolution, while allowing all other DNS lookups to work as expected.

It makes me sad it took this long to finally come to a working solution, but hopefully this advice can help someone else looking to solve a similar problem. I know I couldn’t find anything out there.

Automated pet feeding

Tue, 01 Sep 2020 10:48:17 -0500

One of many projects I’ve worked on while my blog has sat here silent was an automatic solution for feeding our cats while we are away from the house. My initial research on different available products had some interesting findings: There are generally two kinds:

Well-made (often US-made) full-featured, IOT (internet connected) with cameras etc… for a LOT of money ($150+++).
- Kinda sketched out by an internet-connected device that has a camera etc… and I don’t know what it’s security posture is
- Not interested in paying a ton of money
- Really not interested in something that requires a third party service that might shut down and turn the thing into a dud (See PetNet https://arstechnica.com/information-technology/2020/07/petnet-charges-new-30-annual-fee-for-a-service-that-still-doesnt-work/ )
Cheap, questionable quality imports. $40 or less sometimes
- 100% of the reviews I read on these read something like this: “Worked great for X days, then stopped working at all, my pets starved”
- The overall vibe I got from all of the reviews I saw was that the standup designs (like what I got) were definitely superior and lasted a long while, but the electronics on them were flaky and unreliable

So, to tackle this dilemma I felt I had two options: Spend way more than I was comfortable with on something that skeeves me out, or spend what I was comfortable with to get something that’s nearly guaranteed to break after the first use. However, if I got the cheap one, I could probably replace the logic board with a Raspberry Pi and most likely the physical device (motors, sensors, etc…) would be just fine, if the thing didn’t work as expected.

I ordered it, set it up, and let it do its thing

https://github.com/ShadowZero3000/petfeeder

When watercooling goes bad, it doesn't have to be all bad

Sun, 12 Mar 2017 19:34:43 -0500

July 2016 marked the end of a less pleasant time in my life, and as a celebration of its closure, I treated myself to a completely new PC build, using the freshly released Nvidia 1080, and a lot of other high-end parts. To house this new build I wanted something aesthetically pleasing and notable that also supported a water cooling setup. After reading quite a few independent reviews, as well as sales site reviews, I settled on a white/blue Deepcool Genome.

Upon assembling this beast of a machine, I was quite satisfied with the case and setup, aside from the lack of brightness on the helix (I already expected this thanks to some great reviews).

Later, I decided to flip my graphics card vertically for the visual effect of things. After doing so, I found the case’s first flaw: There was insufficient airflow for a large graphics card in the vertical position (as they suck air from where the window is, and that left little gap for intake). This was easily remedied by carving a hole, and adding a personal touch to my machine.

With this, I ended up with something unique and functional, and things were good for a while. Unfortunately I later had difficulties with the quality of the cable I used for the vertical mount, and reverted back to a traditional horizontal mounting (this is relevant to the next part of my saga).

Fast forward to this last Friday. I work from home at my computer desk, though on a separate computer. While I was working, I noticed my desktop was making a gurgling sound (as though a water pump has insufficient water). It was relatively minor, but somewhat concerning, as the Genome should be a closed loop (no refilling ever needed). By the end of the day, the gurgling had gotten quite notable, and so I emailed Genome’s support about it, and continued on my merry way. I fired up a video game, and shortly into it my screens went black. My heart stopped, and I glanced over inside my case to find this: (That’s water on my 1080)

Little did I know, there is now a very commonly reported issue with the water cooling block failing on this case, but the reports of failures after 6-12 months only started coming into NewEgg AFTER I purchased mine, so I had no idea.

After recovering from the shock, and attempting to dry out my graphics card, I determined I’d need to put in a new watercooler. However, the original unit had the helix at the front of the case built into the loop. There was no way for me to incorporate it safely into a new system, and without it I would be left with an 8×2 gaping hole in the front of my case. This sparked an idea: Why did I need to keep what was there, or have a blank spot? Why not put something even cooler in its place? A trip to MicroCenter for a replacement cooler and some parts later, and I had my new plan: Embed a Raspberry Pi inside my case with a screen mounted in the gap where the helix was, and display anything I feel like on the Pi’s screen. ~~With everything installed I was also able to check that my graphics card was luckily not damaged (whew).~~ (Turns out it actually was damaged…it just didn’t fail under light strain)

It took a few attempts to get everything situated, but I now have a lan cable running from one of my two ports through my case and directly into the Pi, power being fed off the system’s USB 3.1 ports powering both the Pi and its screen, and a VNC/SSH accessible OS that allows me to control it however I want. The end result is rather amusing:

I’m rather pleased with the end result. Some simple linux-fu (involving gifview and xdotool) allows me to get the animations up on the screen on bootup. In the future I’d like to put in some live data (I.E. maybe temperature sensors, or “what I’m currently playing” type data) on the screen, but with a full-on Raspberry Pi in there, the sky’s the limit on what I can do when the time comes.

In summary: Sometimes something bad can be turned into something awesome.

Update:

After the system seemed to be ok for a day, it blacked out again after loading up a game. Turns out that basically the GPU, motherboard, and RAM ended up damaged from the shock. Deepcool was willing to help out, and I was able to rebuild the system, so things turned out ok.

Dear future-self, I meant well at least

Wed, 22 Jun 2016 07:04:40 -0500

The other day a co-worker of mine asked me an interesting and difficult to answer question:

How do you determine whether to take the time to fully design and implement a robust and well-architected solution to a problem as compared to utilizing a solution that will handle the immediate issue/needs of the problem at hand?

I both love and hate questions like this. It shows a true concern for balancing doing things “right” as well as an understanding that it’s often easy to get carried away with what “right” means, but answers to questions like this are so situational and cultural that it’s difficult to respond to. So, I’d like to share some of my thoughts, as highly-opinionated and potentially incorrect as some may interpret them to be.

I’ve had numerous years of experience in software development and delivery, and I’ve seen some beautifully designed and implemented solutions that flex to meet unexpected future requirements, and I’ve seen those five-minute hacks that cripple production systems and make folks reach for their resumes. However, I’ve also seen robust architecture crack under its own complexity, or require such complex refactoring that it would be easier to start over. At the end of the day, past-self (That’s probably you today) always has a thankless job. If you do things right, it’s as if you’re never there, if you take the shortcuts, odds are future-self (The jerk you’re likely to see in the mirror in a few months) will be cursing at you later. I like to think of this relationship between past, current, and future self any time I have to make a decision regarding complexity and design, and try to treat my future self as kindly as possible.

The first thing I analyze when I’m worried about whether to spend time designing vs. implementing is: why am I even considering not taking the time? Typically I’m either being lazy, I’m pressed for time, or I’m not sufficiently familiar with what I’m working on if I am asking these questions. If I’m being lazy, future-self is sure to hate whatever code I write, and I’m likely to have to re-do this effort later because of it, and it’s a pretty easy decision (if I’m only going to bother myself, sometimes I stay lazy, but if it’s for work, a client, or something I care about, I’ll take the time needed). However, the other two cases are where I feel further thought is necessary.

If you’re working on a project without sufficient experience in its technology to design a robust solution, there’s a huge opportunity for learning, but spending a ton of time searching for best practices and how-to’s to write some theoretical code will often result in lots of time spent, and little to show for it. On the flip-side, diving in and trying to write code on something you’re not familiar with is a guarantee of low-quality, inefficient, and future-self-maddening code. My personal experience has found that the best results come from accepting that fact, and building re-factoring in to the process. Instead of spending hours researching the ‘ideal’ way to do something, start with something and do your best to get it working. Then, once it’s operational, don’t accept it as done. Refine it. Find a way to make it faster, cleaner, more flexible, or more conforming to a standardized style. Examine how your code differs from what you’re seeing in searches for similar solutions or examples, and make it that little bit better. Good, working better now? Stop and do it again! Maybe tackle a bit more of the project first, but don’t consider that “working” code to be acceptable. Keep comments that say “TODO: Make this future-friendly”, and get back to it immediately (Not “when the project is done I’ll clean this up”. We all know that doesn’t happen). You’ll learn what bad code looks like sooner by writing it, and then analyzing how it fails than by trying to search for perfect code on Google. Don’t be afraid of refactoring, of starting from scratch half-way through. Accept that working on a project as a learning experience is going to require triple the effort, and don’t get attached to your code. Remember: future-self doesn’t have any attachment to current-self’s code, and he can’t remember your excuse for why it was acceptable to take that shortcut.

The other circumstance I see a lot is the heavy hand of time-constraints. “We’ve got three days to get this working”, or “how soon can you get this done?” should be red flags that future-self will be cursing at you. If you’ve got a set amount of time to get something working, you’ve got a few options: Shortcut everything necessary to get there, and be ready to spend three times as long (Heh…if you’re lucky) fixing that mess down the road and trying to make sure your superiors understand the cost. Doing some parts well, and others poorly (still going to be just as messy to clean later, but you’ll forget what parts were good). Or doing things right, and either not delivering on-time or at all. There’s even the occasional (and advised against by future-self, trust me, we talked) herculean “do everything right within the time-box” approach which means stress and mistakes. Sometimes it will even end up that you were worried about nothing to begin with. The right answer should be: Do things right and adjust your expectations accordingly, but that’s not always an option. I’ve yet to find the solution to fixing management’s lack of understanding that last statement, but I’ll write a book on it if I ever succeed.

Along these lines, here are a few thoughts and some generally biased advice around writing code that rocks:

Take ownership of your code’s quality
- Your job may be demanding, deadlines looming, bad past choices haunting, but at the end of the day the quality of any code you produce is yours alone. Take pride in what you’re putting out there in the world, it affects how you see yourself, how others see you, and quite possibly how you will be remembered. Don’t blame your circumstances for your failures, and when you see your own bad code, remember that by simply recognizing it as bad, you’ve already grown to be smarter/better than you once were, and cherish the journey.
Teach your environment to respect you and your needs
- Take care of yourself and what you need to make your code good, and focus on eliminating what detracts from that. If that means you have to take twice as long to do your job right, then perhaps that needs to become your new operating pace. If your boss is making unobtainable demands, confront them, and if that doesn’t work, consider whether you need a better employer or boss. If distractions keep you from doing your best, write code in distraction-free environments. If your keyboard is bothersome, get a new one. This stuff is easy to fix when you identify it, and refuse to accept your status-quo from keeping you from being your best.
Balance your doing vs. planning vs. research time
- It’s important to plan out work when it’s sufficiently complicated, and it’s also important to research topics you’re working on in order to do well at them, but if you’re not keeping tabs on how much time you’re spending doing vs. planning/researching, it’s very easy to let any one of the three spiral out of control. If you’re not doing any kind of planning or research, you may want to consider slowing down and making sure you aren’t burrowing down a hole that’s hard to back out of. If you’re spending more time planning out your work than doing it, perhaps you don’t understand what exactly you intend to do (or need better management in general). If you’re spending your days reading up on tech and not implementing anything it’s going to take you more effort to get rolling when the time comes. There’s no X% planning/research/doing that defines “in balance”, it’s a personal thing, but you’ll know when you’ve reached balance, because the frustrations and friction that each of these causes in imbalance disappear.
Trust yourself and embrace your mistakes
- It’s easy to fall into a trap of doubting your capabilities, or the robustness of your solutions and to over-analyze or critique your own work to the point where your fear of doing things wrong keeps you from doing anything. You have to trust that (assuming you are taking the time to build in quality) you can do things well, and allow yourself to learn from mistakes to improve continually. You don’t become great by waiting around until you just have greatness, you become great by failing (a lot) and learning from your failures what it takes to be better. How many times do you think the great sculptors made sculptures with imperfections before they were recognized as amazing artists? How many of those same artist’s most renowned works are truly perfect?
Learn about coding dynamically
- Software code is incredible in that every problem can be solved in innumerable ways, and frequently a good solution to one problem can be re-used to solve another. It’s a rare many-to-many relationship that should not be ignored. So I encourage you: When you encounter a problem, try to make your solution a bit less tied to the exact problem you’re solving, and more at the type of problem you’re solving. This doesn’t always apply, but I can’t tell you how happy you’ll make future-self when they can take some past-self solutions and use them to solve future problems. Learning how to think this way is an art, but it’s very rewarding in both time savings and satisfaction. Make your code dynamic enough to handle the problems you will solve later when you can.
Surround yourself with excellence
- Learning from your own mistakes and experiences is invaluable, but there’s another fantastic way to accelerate this: learning from your peers’ experience. This can be attending conferences where folks that use the technologies you do explain what they are doing and how they’ve conquered their own problems, it could be sitting in an IRC channel seeing the problems others encounter and participating in finding solutions, or simply spending a few evenings a month in the company of thought leaders or co-workers idly chatting about the technologies you’re passionate about. You never know when an old co-worker will encounter an issue or tackle a problem that will save you hours of frustration down the line, or when you’ll do the same for someone else.
Remove the pedestals from your heroes
- It’s very easy to treat the creators of your favorite tools/languages/etc… with the same kind of isolation and admiration that American culture places on our celebrities, setting them aside as somehow different or untouchable. (I’m guilty of this) But technology isn’t this way, and the people that wrote those tools are often just other passionate developers etc… out there sharing their own solutions (Mitchell Hashimoto‘s story is an inspiring example of this). Don’t be afraid to talk to the ‘heroes’ of technology, you’d be surprised sometimes how happy they are to interact with their communities (and worst-case, you’ll end up with an email in a ‘deleted’ box, no harm). Open pull requests, start conversations in IRC chats and Slack channels, comment on issues. You never know when you’re going to be the voice that reminds someone else why they love coding. I have recently learned this lesson, and it’s been incredibly empowering and energizing to realize how friendly and passionate the Puppet communities and thought leaders I admire are.

I hope my ramblings strike a chord and perhaps provoke some positive thoughts out there. There’s an unfathomably huge amount of code out there in the world, and I want to do my part to encourage more of it to be code that rocks, and to save us all (or maybe at least myself) from the wrath of future-self.

Helpful info for debugging rspec-puppet

Thu, 09 Jun 2016 12:19:15 -0500

So I found this awesome piece of information about debugging rspec-puppet, and particularly this block of code: File.write( 'myclass.json', PSON.pretty_generate(catalogue) ) 

This is INCREDIBLY useful for debugging.

In fact, I found, by using this, that when testing defined type parameters, they are all lowercased. Even when the parameters SHOULD have camel-casing etc…

Be aware, this annoyed the heck out of me and my team.

Let's Encrypt: The future is now

Thu, 07 Apr 2016 00:41:16 -0500

I remember several years ago looking at my site and thinking “ya know, it’s about time I got hip with this SSL thing”. Several days later, I was able to implement ssl on my primary site using https://www.startssl.com/. This took me a notable amount of manual effort to obtain the certificates from their site (Props to them, they use client certificates to validate users in addition to credentials, which is rather secure, if obtuse and inconvenient), and several hours of learning how to tweak my Apache configs to use them (I had a complex dynamic setup, and isolating what sites get what certs was not something I had built into that design). Success! Or not….I then had to go back through and fix various parts of the actual site to be consistent about loading everything from https, and to allow regular http calls where appropriate. It was a painful ordeal, but at the end of the day, I had a more secure site, and I had learned quite a bit about everything involved. (Nowadays it probably wouldn’t be nearly as painful with Puppet etc… but it’d still require manually obtaining the certs etc…)

Fast-forward one year, I had entirely forgotten about this effort, and the process involved. I left myself no documentation on what I did or how to renew the cert, and received an email letting me know that the cert was expiring in a month which I had promptly ignored, and now had a site with one of those big red X’s in the corner. I went back to StartCom to renew, and couldn’t even get logged back in (I forgot about the whole client certificate thing, and had a new laptop at that point). I put fixing the situation on the back-burner, and went several months before allocating some time to figure out where I went wrong. I was eventually able to remedy my expired cert, and removed that ugly X, but the process made me dread the idea of having to re-do this annually. However, being a bit of a cheapskate when it comes to technology, I resigned myself to my fate.

Fast-forward again to today, with a shiny new server and domain configured, Puppetized, and running Nginx instead of Apache I was feeling ready to tackle the SSL situation again. I had been reading about Let’s Encrypt for a while, and was excited to give it a try. All I can say is: wow. I added danzilio/letsencrypt to my Puppet module arsenal, dropped a few lines into my server’s hiera config, and put together an 8-line profile to wrap the ‘certonly’ defined type, and had certificates for multiple domains ready to rock. No pain, no manual effort and client certificates, and a shockingly easy setup (I hit one small snag with forgetting the ‘webroot_paths’ parameter that slowed me down all of 15 minutes). I’ll admit I struggled to tune my Nginx config to be just the way I wanted it for a day, but now that it’s all in place my domains redirect to https where appropriate, use a Let’s Encrypt cert, and will keep that cert valid for me with zero effort on my part thanks to the quality workmanship in the letsencrypt module. What a complete polar-opposite experience!

If you’re not using SSL yet: Let’s Encrypt – The future is now, and your site’s clients deserve some encryption.