Mar 12VENV — This Ain’t Your Grandpa’s Virtual EnvironmentImagine you’re sitting there, doing your job, minding your own business, and then the sky opens up and all these packages come crashing down on you. Pandas, Notebooks, Pillows… Not cool right? Well, that’s how the Global Python Environment feels when you Pip install packages directly onto it. …Python4 min readPython4 min read
May 28, 2022List and Dictionary Comprehensions Made EasyIn my day job as a Python developer, I use list and dictionary comprehensions quite regularly. They also come in handy during live coding tests for an interview. List Comprehensions The way I remember a list comprehension is simply: [x for x in list] If you can remember this, then you can…Python2 min readPython2 min read
Apr 7, 2022UUIDs in PysparkI often need UUIDs in Pyspark and Databricks. I use both deterministic(repeatable) ids and random ids. Random UUIDs For random ids, before I used a UDF: import uuid @udf def create_random_id(): return str(uuid.uuid4()) But as of Spark 3.0.0 there is a Spark SQL for random uuids. So now I use this: from pyspark.sql import functions as F df.withColumn(“uuid”, F.expr(“uuid()”)) This is nicer and is much faster since it uses native Spark SQL instead of a UDF (which runs python).Pyspark1 min readPyspark1 min read
Dec 8, 2020Should you be a data engineer or a software engineer?It depends on what you enjoy building with code. You should try both and see. Here is how I would find out. Build a small software program that does something interesting. An example could be a python program that web scrapes the top headline on a website that you follow. …Software Engineer3 min readSoftware Engineer3 min read
Published inThe Startup·Dec 6, 2020Web Scraping All the Links With PythonRecently I wanted to get all the links in an archive of newsletters. The goal was to have a text file with the links so that I didn’t have to manually go through each newsletter. This is a real world example so I wanted it to be fast and easy…Web Scraping3 min readWeb Scraping3 min read
Dec 6, 20208 Things to Check in a Python Code ReviewCode reviews are one of the most important activities in the daily workings of a software team. They are an opportunity for everyone to become familiar with the codebase. They help prevent siloing (one person becoming the only source of knowledge). …Python2 min readPython2 min read
Apr 27, 2020Zip and Reduce in DartAs a Python developer who is also writing Dart, there are a few things I was missing when developing in Dart. One of those was the mighty zip function. The zip function in Python is really handy for combining two or more lists into a list of lists. …Flutter2 min readFlutter2 min read
Apr 25, 2020List Comprehensions in DartIn my day job as a Python developer, I use list comprehensions a lot. Now that I’ve been doing some development in Dart, and more specifically Flutter, I wanted a place to reference the common list comprehension style executions in Dart. So here are a few good ones. The Map()…Flutter2 min readFlutter2 min read
