I'm a full-stack freelancer for early-stage startups--that means I compete with No-Code tools, agencies, and especially offshore.  Every day that I work, I'm expected to deliver significant progress on new features day, since many of my clients are short on funds--therefore, there's quite a bit of pressure.

I'm also "polyglot"--which means I specialize in learning common elements between multiple frameworks and programming languages, and then using that wide understanding to deliver results for my clients--it also means I'm constantly looking stuff up on Google, StackOverflow, and spending huge amounts of time on a spaced repetition system to memorize the important bits of each language/framework.

AI should be able to help me.

GitHub CoPilot: stranded offline

After many breathless software developers exclaiming "the end of programming" on Twitter, I decided to give GitHub Copilot a try, which is an AI plugin you can install in your IDE or text editor to recommend code for you.

In my case, I installed it in Vim, and was quickly disappointed when these outgoing network requests hit my firewall:

You see, GitHub Copilot doesn't ship the model to your computer--it runs the model on a server and then connects a web socket to your local machine, feeding you recommendations--so if you're offline, you're out of luck.

GitHub CoPilot: discovering its helpfulness

After getting the plugin installed and setting up my firewall, I was pleasantly surprised that it gave me useful recommendations that I would actually use in my code:

GitHub CoPilot: awkward workflow for a senior developer

Normally, my workflow looks like this:

1. read code and understand it
2. write down a plan of what I want to accomplish
3. write down the files and functions that needs to be changed to execute the goal
4. execute the code change

It's an aggressive workflow where I spend as little time as possible writing code--a lot of time is invested up-front, a lot of work is done outside of my code editor, and then the source code change is all executed at once in a final "render" phase, after I fully understand my goal--Step 4 is where GitHub Copilot could really help me.

I was planning on using Copilot as a really smart auto-complete, that could save a lot of typing time, and save time looking up the exact arguments for the different functions that I use.

The reality was, I type very fast, and there was about a half-second to two-second delay before the recommender returned with a code suggestion--often times I would have already typed out the majority of the code snippet before Copilot would return with a recommendation.

I uninstalled it, and continue to run vanilla Vim with syntax highlighting, a couple Git plugins, and that's it.

GitHub CoPilot verdict

In my own case, as a senior developer with a spaced repetition system supporting my work, where the source code is treated as just a render artifact and not the core of my work, Copilot was not useful--hardly "the end of programming".

There was an alternate workflow, where you write comments and have Copilot write the code for you, but I didn't need this, since I already knew how to program, and it was too slow to be useful anyway.

In the future, I could see an open-source version of Copilot being useful to me, that ran locally.

Our role as software developers isn't going away any time soon

If I really tried, I could make Copilot and AI useful to me--but I don't have time for that.  I'm busy solving real problems for real people.
As for junior developers, I could see AI solutions like Copilot both helping them and hurting them:

• helping them by quickly solving errors and problems in code
• hurting them by being a path-of-least-resistance around documentation and actual understanding, leaving them helpless in the long-term

I'm very impressed with the technical accomplishments with ChatGPT, but as with most things in life, my overall outlook is people-focused, not tool-focused.

So you've built out the MVP of your SaaS product, and you're ready to do a pilot for your first business customer.

There's one problem--the business has hundreds or thousands of existing customer's they'd like to import into your application, and nobody is going to type that manually.

You think "OK, we'll build a quick CSV import feature, and be on our way" -- and then it turns into a nightmare.  In fact, that's probably why you're reading this right now.

Since users are the most common form of data to migrate into a SaaS product, we'll use this as a concrete example for this article.

Why do CSV import features always explode?

Usually, the first mental model we use to think about a CSV import is wrong.  In fact, even labeling the feature "CSV Import" and speaking about it that way in your standups and JIRA boards will lead you down the wrong path.

It's scary being in an early-stage startup, with limited funding and limited time, and an overwhelming heap of responsibilities on your plate--your brain wants the CSV import to be simple, so that you can move on to other features which are badly in need of your attention.

The truth is that this simplified, happy, and optimistic mental model is a trick of your brain.  The cure is to swap your light and fluffy "CSV Import" mental model with a heavy and dense "CSV Migration" model--the reward is accurate time and cost expectations, a solid MVP architecture, and a great experience for your very first clients.

What's the difference in mindset, between an Import and a Migration?

When we talk about a "file import", the mental model is that it's one-shot data:

• importing a CSV into Excel, and then you forever work in Excel afterwards
• importing an Excel  spreadsheet in to a tax app, and then you complete your taxes in the app
• importing your pictures into a photo app, and then you work with them inside the photo app

A migration is different, because you are importing "living" entities from another database--entities that change, get added to, get removed, and get re-added.

If you focus on just the import part, you are setting yourself up for disaster with your first customer, the second they hand you a CSV file with a duplicate user, a missing user, or a user record that's changed.

How do I know if I should use a Migration mindset, versus an Import?

Look at the fields you need to import for your records, if it contains an identifier from the other business' system, then it's a migration:

• a numeric database ID that represents the record in the other database
• a unique email that identifies the user in the other system
• a UUID string that uniquely identifies the user
• a slug or specially-formatted word that identifies each records

These types of fields are called "foreign keys", and if your imported records contain them, then buckle up--you need a migration, not an import.

What is the migration mental model?

With an Import, it's just one-shot data--all we focus on is getting the data into our system, and then we're done.

With a migration, we focus on keeping our system in sync with our clients' systems via the foreign keys.

In the migration mental model, the CSV file is simply the medium used to convey the current state of our client's system--your MVP needs to look at each CSV when it comes in, look at it's own records, and then make an intelligent decision on what to do with each one.

Migrations are a much heavier feature than doing a simple import--particularly when you're importing users, which often play a central role inside your product.

Ok, I need to build a CSV migration feature, what additional concerns to I need to worry about?

Think about the customers using the client's existing system--they're changing, being added, and being removed, and sometimes there are problems which require human intervention.

Think about your own system--the same thing happens.

When these two systems are running separately, a CSV file comes to bridge the gap--now you need to handle the following scenarios:

• a row in the CSV is invalid for some reason--it's incomplete, the data is mangled, or there is a logical conflict with an existing record in your MVP
• a user was created in the client's system, and they need to be provisioned in yours
• a user's state changed in the client's system (think deactivated or reactivated), and this state change also needs to happen in your MVP, and fire additional logic if needed
• a user was deleted from the client's system, and they need to be removed from yours

In addition, if a business is migrating users to your platform, chances are those users already have a password in the client's system, and don't want to set up another one in yours--therefore, your clients will ask for Single Sign-On.

How to build robust CSV Migrations quickly

Now that we're in a CSV Migration mental model, we have a clear path forward to serve our first clients, and we can pick and choose the pieces we need in the MVP.
Here are some general technical features needed to complete a CSV Migration for users--run these by your software developer to decide which ones you can implement with the resources you have:

• You'll need to transport and store the CSV files themselves--usually done through something called SFTP
• You'll need to process the CSV file using a background job, to avoid timeouts and errors within the UI of your MVP
• You'll need a provisioning system to create users "headless"--without the UI that's currently in your MVP, and without user intervention
• The CSV Migration records need to be in their own table in your database, so that you can manually intervene to fix records with errors of any kind
• (optional) If the import conflicts need to be resolved by a non-technical, you might need to make a UI within your MVP to do so
• (optional) The client will usually ask for Single Sign-On, which you can implement with something called SAML--but this can usually be delayed until after the pilot

How does this make it quicker to build a CSV Import?

When you thought you were building only one feature (just a CSV Import), you actually need six.  Indeed--building out this migration capability will take a lot more effort, money, and time that you originally expected.

If you are able to overcome that feeling of overwhelm, and scrape together the resources necessary to execute this, these are the rewards you'll reap:

• when you mention things like SFTP and SAML (or their non-technical versions "Single Sign-On" and "Secure File Server") you'll be viewed as a subject matter expert
• when you give your pilot clients a launch date, it'll be accurate, because you already know the major pieces to implement
• you'll avoid that feeling of terror, when a client uploads their first CSV, and it errors--because you already have the architecture in place to handle all the concerns of the migration
• you'll never have to spend time and money re-writing a CSV Import feature that was built correctly as a CSV Migration in the first place

Sometimes the long road is indeed faster.

Good luck!

Yesterday, my laptop ran out of battery while I was working on an art project, and my computer shut down unexpectedly.

I plugged it in and rebooted, and continued to work on my art project.
The next day, when I began to work on my programming project, I was greeted with this error:

In this blog post, you'll learn how to troubleshoot Postgres installed with Homebrew on a Mac, and how this example fits into a larger framework that you can use to troubleshoot any software error.

UPDATE (Nov 2022): Homebrew now stores data in /opt/homebrew, not /usr/local as shown in the screenshots

UPDATE (Nov 2022): Homebrew now separates Postgres packages by version, e.g. brew info postgresql@14 or brew services start postgresql@14

The game plan: how do we troubleshoot automated systems?

For now, it's enough to mention the general model of how to troubleshoot part of an automated system:

• try shotgun solutions, in case you can skip troubleshooting complelety
• design an experiment to isolate and reproduce the error reliably
• gather log samples from your experiment and make a diagnosis
• apply the fix, and verify your experiment passes clear

A simple math problem: why we should avoid troubleshooting

I've found that troubleshooting is time-consuming, non-productive time--and contrary to my first intuition, it's actually a very poor substitute for intentional, directed learning.

Therefore, it's in our best interests to minimize troubleshooting time to the smallest possible value.

We can accomplish this by never troubleshooting at all!

For example, if you know that...

• restarting XYZ will fix your issue
• you need to restart XYZ once per day
• it takes 10 seconds to restart XYZ
• it would take about 45 minutes to find root cause and actually troubleshoot the issue

... then you won't break even on troubleshooting for 270 days.

Chance are, in 270 days, you'll upgrade the particular package, stumble across the root cause of the problem somewhere else, or the problem will mysteriously go away.

Trying some shotgun solutions

First, get the status of all services:

brew services list

I got this output, which tells me that Postgres failed to start:

Next, attempt a restart:

brew services restart postgresql@14

... Dang, that doesn't work--we need to actually troubleshoot this.

Design an experiment to isolate and reproduce the issue, reducing the iteration time for fix attempts

In software, we're generally trouble shooting pieces of automated systems.  We could make hypotheses, apply solutions, and attempt to test the entire system as a whole, but this generally takes too much time.

The reason is that these long chains of automation generally have very long iteration loops:

• rebooting computers
• making API requests
• compiling the entire project's source code

Since we know the shotgun solutions didn't work, and we're going to commit time to troubleshooting, our first action is to reduce this iteration loop to the smallest possible time.

write down some manual commands you can run to reproduce the issue

We know there is a problem starting Postgres through Homebrew, therefore we look at the source code for the automated system that's executing that command:

vim ~/Library/LaunchAgents/homebrew.mxcl.postgresql@14.plist

• often times, the call site will point you to special options or config used to run the command
• find the call site first, and then move on to configuration second, if there's nothing interesting

Under the ProgramArguments key, we can see the exact command and arguments used:

Step 2 of the troubleshooting model is done--since we have the command that Homebrew is using to execute the command, we can just run the command ourselves:

postgres -D /usr/local/var/postgres

This is our "experiment"--we can apply potential fixes, manually run this command, and make very fast rapid-fire iteration until the error is fixed.

No need to reboot our computer, or wonder if the Homebrew commands are hiding command output, or wonder if some other part of the system is interfering with starting Postgres--we have isolated and reproduced the error in a little test tube, on command.

Gather log samples from your experiment

In this case, the Postgres command logs additional output to the console:

And look at that--running the command manually produces a verbose error message, which basically points to our solution.

Aha!  That must be a stale postmaster.pid file, that was never deleted when my computer shut down yesterday

Apply the fix and retry the experiment

Remove the offending file:

rm -f /usr/local/var/postgres/postmaster.pid

Retry the experiment:

postgres -D /usr/local/var/postgres	

It starts up normally--we're done!

Conclusion

I hope this blog post has helped those of you searching for an answer to your Postgres start-up problem, and given you the seeds of a troubleshooting system you can use for rest of your career.

Resources

• https://apple.stackexchange.com/questions/437618/why-is-homebrew-installed-in-opt-homebrew-on-apple-silicon-macs Discussion of why Homebrew moved its storage from /usr/local to /opt/homebrew

Especially in early-stage startups, expectations are high, time is short, and funding is limited--that's why we create software estimates: to help steer the project.

The problem comes when our time estimates are way off--sometimes by a factor of 3 or 4!  Deadlines are missed, frustrations are inflamed, and our own sense of worth is called into question.

In this blog post, I'll give you a simple mathematical formula you can use to create accurate software estimates, even when the full scope of work is not known ahead of time.

The napkin estimate: a system for harsh consequences and broken promises

The most common type of estimate that I see in the wild is called "The Napkin Estimate"--named so because it's done at random, in front of other people at an important meeting, and under pressure.

It's said to be "written on a napkin" because it should be regarded as crude, disposable, and inaccurate--the problem is, I rarely see it being offered that way, and it's often presented as if it were a valid estimate.

The silence at the dinner party: how a Napkin Estimate happens

Picture this: you're a software engineer at an important planning meeting with engineers, product stakeholders, and sales people, who all want to align on a particular date.  A cool idea for a feature is floated, and then the Head Honcho turns to you and asks the Forbidden Question:

"How long would it take to do The Feature?"

Everyone turns to look at you--some of the eyes are pleading, some of them hopeful, others judging, but your fate for the next 3 months is determined by what comes out of your mouth next.

-- Phase 1 --

Your eyes look up and to the right, you say "Mmmmmm.....", and the following thoughts race through your mind:

• "That's a trivial feature, which should be easy"
• "We can just connect X to Y, and then just insert step Z into the workflow"
• "That's kinda like another feature I've made, which I could implement now in X weeks"
• "... I'm gonna quote X weeks!"

You feel hopeful, like a ray of sunshine!

-- Phase 2 --

But then the hope quickly disappears, as you become aware of a sinking feeling in your gut:

• "..... wait, the description of the feature is missing specific details"
• "... the interaction points between other features haven't been mapped out, and probably aren't known"
• "... I don't even know which questions I'd need to ask, and it's likely that nobody in this room could answer them"

-- Phase 3 --

Time passes, and the silence becomes uncomfortable.  You say "Ummmmm....", and stare off into space:

• "everyone is looking at me"
• "I have no idea what to quote"
• "If I quote too high, then I'll disappoint them and ruin their plans"
• "If I quote too high, I'll look stupid in front of my other engineers and managers"
• "... I'm gonna quote  weeks, and deal with the consequences later"

You quote X weeks, and the uneasy silence fades away, as the rest of the team happily goes on planning, satisfied with your quick napkin estimate.

The bill comes due: the consequences of the napkin estimate

X weeks have happily passes, except The Feature is not done--it's not even close to being done:

• you do overtime, and force your team to do overtime
• you constantly quote "one more week" every time the stakeholders ask you for a status update
• in order to meet the agreed-upon time deadline, you release a half-finished feature, which is unusable, hated by the users, and creates lots of technical clean-up work and backwards-compatibility concerns

When you do a post-mortem on this feature, after you add up all the hours of planning, overtime, and the fact that the feature wasn't complete, and still needed work after it's release, you discover that it was off by a factor of 3:

• in other words, if you quoted three weeks to finish The Feature, it would have taken you over two months
• There are a couple alternatives out of this self-inflicted misery, which I'll cover in later blog posts.  But for now, we'll focus on the napkin estimate itself, and how to turn it into something good.
  
  Short-circuiting the napkin estimate: catching it is solving it
  The critical moment of a napkin estimate comes at the beginning of Phase 2, which will be signaled by your body:
• eyes moving down
• feeling a sinking feeling in your stomach
• feeling a sick feeling at the back of your throat, like it's difficult to speak
• automatically vocalizing "Ummmm....." or another non-word sound

Any of these tells are your body's way of communicating to you, "I'm being forced into an uncomfortable situation, and I feel afraid".

Right now, if you're thinking "Yes, I've felt that before!", you can move on to the next section.

On the other hand, if you can't recall ever feeling any of these things, I recommend you pause right now and Google "The Science of Enlightenment", "contemplative prayer", or whatever you use for inner work--once you become aware of the napkin estimate, you can fix it.

Thwarting the napkin estimate with math, and accepting difficult realities

Alright--you've been waiting for it, and now I'm revealing the grand plan of how to defeat the napkin estimate, and make it useful:

• multiply it by three, and quote a range

That's it.

I know that's unbelievable, but I've seen that 3x figure be accurate across companies, dev teams, and tech stacks--the human brain will consistently under-estimate the time it takes to complete a software project by a factor of three.
For example, if you think it'll take 3 weeks to complete, say "best-case three weeks, worst-case 9 weeks".

However, after you give such a wide-ranging estimate, you'll trade glances of shock, annoyance, and sadness with your peers--now is the time for you to shine as an engineer, by revealing the limits of what is, and is not known about The Feature:

• you don't know the full scope of work, in every detail
• you don't know how these details will interact with existing features
• you don't know what parts of the work will require new infrastructure
• you don't know what will happen with your labor, while the feature is being built
• you don't know what will happen with business logic changes, as new details are discovered
• the actual time-to-complete is wildly variable
• ... and you don't know what you don't know

If the group is reluctant to accept my assessment of the situation, I'll often respond with calibrated questions to point out specific areas of uncertainty

• "What should happen if there is an error in Process A of The Feature?"
• "What should we do, if we discover new infrastructure is needed for The Feature?  This will push back the estimate."
• "What if Assumption X turns out not to be true?"

These questions are an art in themselves, which you can refine over time, but the main technique is really just quoting the 3x variance and acknowledging that you can't see into the future.

When a 3x range isn't good enough

Usually, an estimate with a 3x variance won't make anybody happy, so stand your ground, and explain the above points.

On the other, certain features do require a more accurate estimate, and this is simply a fact of life.

Luckily, there are many alternatives to the Napkin Estimate, which I'll explain in future blog posts.

Summary

• train yourself to listen to cues from your body, when you've been put on the spot
• quote a range that is three time larger than what you think it'll take
• respond to disappointment by asking calibrated questions or simply acknowledging uncertainty that you can't see in the future

Even if this doesn't make a great estimate, it'll make a safe estimate, and you can continue to flesh it out from here.

Resources

• The Science of Enlightenment by Shinzen Young: meditation techniques to improve your awareness of tells your body is sending you
• Never Split the Difference by Chris Voss: techniques to ask calibrated questions, which you can use to help your team accept difficult realities

A long time ago, I found myself in a vicious learning cycle, where I’d sacrifice time to work through a YouTube tutorial, turn back to my life’s responsibilities for a few weeks, and then realize with horror that I’d forgotten nearly everything.

Today, I’ll share with you some general strategies that helped me to climb out of this hole.

tldr;

I started taking notes, in very specific ways, which I’ll cover in this blog post.  These techniques won’t necessarily be right for your situation, but they will provide some building blocks you can use to build your own personalized learning system.

Step 1: the art of finding a tutorial by an author that resonates with you

Selection is a meta-skill to learning; by improving your selection of learning content, you automatically increase the effectiveness of the content itself, without having to do anything.

Here is a process and criteria that I use to select consistently high-quality tutorials:

• do a search for the programming topic you want to study
• open the first 8 results in separate browser tabs, closing any tabs by the same author
• click to about 1min 30seconds (seems to be a good time when most creators are past the intro)
• watch the video 30 seconds
• after you’ve watched a small piece of all the videos, select the author you like the best

Things to look for:

• good audio, legible text
• the level of explanation matches where you’re at in your learning journey
• beginner: look for authors that explain everything typed on the screen
• advanced: look for authors that only explain the new code their typing in
• the areas of focus
• relations are explored as part of the tutorial
• only the specific topic at hand is covered in the tutorial

Things to avoid:

• “AnNowImGonnaDoThis” and “AnNowImGonnaTypeThat” without any explanation as to what this code actually represents
• “And now I’m gonna do this
• Instead: find an author that speaks from a  proactive tone instead
• “OK, and now we’re going to set up X.  In order to do this, we will type Y into the terminal"

Browse all the videos from that author, and select 25 minutes of content.

Step 2: Brain soak unfamiliar content

The basic idea is that learning is related to memory, which poses a problem for brand-new abstract content: you have no existing memories to attach the new content to.

To mitigate this, we do a brain soak of the 25-minute content, which means watching the videos at 1.5x speed--that's it.

Just listen to the words they are saying, and in particular vocabulary terms that are repeated over and over again—these are the major components of the topic.

This step will create a weak scaffold of words that you can associate the actual learning content to.  Although it's not as good as having actual mastery and deep experience of the content, it's better than nothing.

Spend 25 minutes doing the brain soak, until you're confident that you could at least speak intelligently about it for a 30-second interaction--I imagine myself speaking to a project manager, client, or another developer about the topic.

Step 3: use a computer to quickly capture content in real-time

Video content (also called "animated content" in academia) poses two challenges for learners:

• You can’t skim the video content easily
• Mixture of audio and visual content makes transcription incomplete
• You can only refer back to timestamps, instead of page/line numbers

We can overcome these challenges by setting up systems to quickly capture spoken and visual content.

How to prepare for content capture:

• set up a hotkey to capture a region of the screen
• for example, on macOS it’s CMD+SHIFT+4
• Open a note-taking program, such as Evernote, and create a note for the video
• If the video has a table of contents in the description, type each entry as a bold heading inside your note, so that you have a place to quickly capture content for each topic

Programming tutorials are typically project-based, starting with a snippet of code that is iterated on, as the tutorial progresses.  Ideally, we'd want to follow along with the project, gaining that deep understanding--but we don't have time for that.

Therefore, we flip into a "capture" mindset, plowing through the video, and taking quick notes and actual screenshots of the code the instructor is typing:

• As soon as the author mentions something important, pause the video
• type a single sentence into your notes, under the bold headline for that section
• take a screen capture of the video content underneath your summary
• type in the exact time stamp in case you need to go back to this video later to resolve a question

After that section is complete, delete the screenshots and summaries that aren’t really important, expand your quick notes into actual sentences, and maybe look up one important concept on StackOverflow or Google.

The important thing is not to get bogged down here, and maintain forward velocity through the tutorial.

There is no step 4

One of the great things about memory is that it's automatic--we learn by doing.  At this point, you should have enough knowledge to muddle through your work, and if you get stuck, you know which vocabulary terms and concepts to Google.

As you complete your work, those areas of knowledge which are useful will automatically form memories and associations around them, and you don't have to do anything else.

Extra Credit: Cement long-term memories with physical notes

Physical notes are a great tool, to help you “re-hydrate” content that you haven’t seen in long periods of time, such as 6-24 months.

Although physical notes are their own topic, that deserve their own article, here are a few scattershot tips to help you improve their impact

Physical notes are a reflective process.

You’ve already done summaries and content capture in your electronic notes—don’t bother repeating that content here:

• write down connections you made to other topics while watching the video
• the process of physical writing is itself a memory that you will form connections to new content in the future
• the physical process of writing activates multi-modal memory triggers--memories anchored in place by sight, sound, touch, and the physical space of the room you're in, rather than just visual images on a screen

Step 5: sustain memories with spaced repetition

Sometimes you don't have a practical project to work on right away, and you need an industrial-strength method to cement that content, so that you can still recall it in 6 to 24 months.

For example, one time I went through a machine learning course, even though I didn't plan on working in the field for at least a year--I still wanted to retain all that content.

An effective tool to use in this situation is spaced practice, which basically means "Write your own flashcards, and tell a computer show them to you at decreasing intervals over time".

Spaced practice has several benefits:

• mitigating the effects of distractions
• reducing the time it takes to recall memories
• sustaining memories in the absence of a practical project
• sustaining memories in the absence of emotional associations to the content

This is beyond the scope of this blog post, but I challenge you to check out a program called Anki https://apps.ankiweb.net/ and write just one spaced practice card for every tutorial you watch.

The important thing is to make your own flashcards, make as few flashcards as possible, and to keep your spaced practice light and fun.  If anything within this practice becomes heavy and burdensome, wield that ban hammer and suspend all cards that aren't absolutely important to maintain.

Remember that you can start very small and simple, and gradually build up a spaced repetition practice over months or years.

You can do this right now.

Step 6: Accept that mastering a skill requires time and effort

In my personal experience, doing all of the note-taking and spaced practice has doubled the time it takes to consume a YouTube video.

For example, a 45-minute video will actually take around an hour and a half to capture, plus the 25-minute brain soak added on at the beginning.

The tradeoff is that I’ve completely mastered the content from the videos I do decide to watch, and I can recall specific details from tutorials that I’ve watched five years ago.

I’ve thrived in my career, in very difficult situations, by using this exact process, so I hope that it also unlocks something of value to you in your journey!

Resources for learning

The Learning Scientists: Elaborative Interrogation: https://www.learningscientists.org/learning-scientists-podcast/2017/11/1/episode-6-elaborative-interrogation
Scientific American: laptop vs hand-written notes: https://www.scientificamerican.com/article/a-learning-secret-don-t-take-notes-with-a-laptop/
The Global Scholars: Handwriting vs Typing: https://theglobalscholars.com/2018/10/19/handwriting-vs-typing-which-is-better/
Luca  Lampariello general ways to improve listening comprehension: https://www.lucalampariello.com/improve-your-listening-comprehension/

In this blog post, I'll tell you how to remap your Caps Lock key on Windows to Control when pressed with other keys, and Escape when pressed alone.

Why would you want this?

The Control key is used by many keyboard shortcuts in Windows, and the physical control key has awkward positioning on many keyboards.

If you use Linux and Vim on WSL2, it's import to also have the Escape key in close proximity, since it's constantly used to exit modes in Vim.  Escape is also handy for software of all types, which you'll start using more, once it's in a better position

History

The idea to remap Caps Lock to Control, and then Escape if tapped alone was made famous by Steve Losh, in his "Space Cadet" blog post from 2012: https://stevelosh.com/blog/2012/10/a-modern-space-cadet/

That solution made its rounds among Mac web developers, particularly in the Rails community, and then later spread to Windows via various registry and AutoHotKey scripts, such as this one: https://gist.github.com/sedm0784/4443120

There was an alternate way of doing this, for people that just needed to remap Caps Lock to Control, that only involved making a registry edit via the SharpKeys program: https://www.randyrants.com/category/sharpkeys/

Combining SharpKeys and AutoHotKey v2

Some people used only SharpKeys, and some used AutoHotKey, but the best approach was to use SharpKeys to remap CapsLock -> Escape, and then use AutoHotKey to layer on the Escape functionality.

AutoHotKey v2 is currently in beta, and much improved in many ways over V1.  A user posted this script for v2: https://www.autohotkey.com/boards/viewtopic.php?t=92738

• use in combination with SharpKeys for the best overall solution

I installed this version of SharpKeys:

And I installed the following key remap:

Reboot for that to take effect.

Run AutoHotKey v2 Executable on startup

Compile your AHK script into an executable:

• great!  Double-click and run it... test in WSL, and everything's good!

Running an executable at startup
Reference: https://support.microsoft.com/en-us/windows/add-an-app-to-run-automatically-at-startup-in-windows-10-150da165-dcd9-7230-517b-cf3c295d89dd

Drag your executable here:

• A shortcut to get there is to press WIN+R and run the shell:startup command

Note on an alternative: Dual key remap

https://github.com/ililim/dual-key-remap

Written in C to use the Windows API directly, this program can do both the Caps Lock --> Control and Caps Lock --> Escape remap.

I chose to use Sharpkeys/AutoHotKey because the registry hack will also work on the Windows login screen, and I always have AutoHotKey installed on Windows anyway.

Platforms like Wordpress, Drupal, and Ruby on Rails have ruled the web for a long time.

To many developers that have mastered these platforms, they’re looking for something more—they’re looking to grow and expand, and many look towards the JavaScript ecosystem as the next rational place to expand their skillset.

But often times, what initially seemed like a natural direction to grow can become a trap of unproductive work for work's sake.

How it starts

The story begins on a traditional server-rendered web framework, such as Rails, Django, or a multitude of others.  Then, an intermediate or senior developer starts to wonder:

“I'm so excited to use this new and exotic framework!”

“I can’t wait to use all those fancy npm libraries!”

“I can’t wait to feel like a real application developer!”

“I’m so excited to have React on my resume!”

“I’m so excited to apply my skills to mobile and desktop platforms!”

How it ends

• Fiddling with an additional build pipeline
• Booting the React app with serialized state or an API call
• Implementing caching, tree shaking, and async module loading to handle the large JS bundle size
• Finding clever ways to handle the flash of un-styled content while the app loads
• Reconciling the traditional CSS from the server with the CSS-in-JS from the React app
• Adding API routes and controllers for the React app
• Debating with your team if the React app and the server should be in one git repo or two
• Adding a GraphQL layer to your back-end
• Finding "React compatible" libraries for animations, drag and drop, and select lists that come with their own performance trade-offs
• Cycles of deciding which state management library to use, changing your mind, and re-writing parts of your app

Substitute "React" for any front-end framework, and the result is the same.

What are some alternative ways to grow?

The opportunities that exist in the world are as far-ranging as your imagination can reach.

There’s no reason to limit your growth pathways to just the latest tool laid out by a rich FAANG, or keep yourself tied to an endless treadmill of tutorials.

Here are three suggestions to engage your creative mind, and grow from the inside-out:

• learn algorithms and data structures, and model parts of your application as a graph, a stack, or any number of alternatives, and express your own unique style through the choices you make
• become a leader for your team by modeling existing code in state charts, and accidentally fix bugs in the process
• master an obscure technical topic, and become a highly-paid expert in that community

These are just a few ideas I came up with after thinking for a couple seconds--you dont need to be constrained by huge software stacks and complicated dependency trees.

The real reason for this blog post

When I started my dev career, I relied on popular frameworks to bridge the gap between where my skills were at, and the projects I needed to accomplish.  The problem is, those frameworks became a trap that limited me for far too long.

And so here I am, building products with state charts, the raw DOM, and the simplest software stack I can possibly use. I hope that an intermediate developer will one day sneer at my work, because it's not using the latest libraries, and then some time later find himself, thinking "Wait, that was actually a good idea!"

Command to transfer the file to Skyleafdesign.com

scp ./valkyrie.zip my_host_:/home/my_user_/

I got this error after doing apt-get update followed by apt-get upgrade:

• didn’t cause any problems

Commands to upgrade the theme

sudo chown -R ghost:ghost ./my_theme
sudo find ./my_theme -type d | sudo xargs chmod 755
sudo find ./my_theme -type d | sudo xargs chmod 644

ghost start gave the following error:

Command to fix file permissions

sudo find ./ ! -path "./versions/*" -type f -exec chmod 664 {} \;

Good to go after that:

Upgrading to Ghost 4.0

Reference: https://ghost.org/docs/update/

Command to upgrade Ghost

ghost update

Oh no! I need to upgrade Node to get Ghost:

Recommended version:

Commands to install Node

curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -
sudo apt-get install nodejs

• the above Nodesource script will create a file in /etc/apt/sources.list.d/nodesource that should contain the version you specified

But there’s a compatibility issue, since my Ghost version is too old to work with the new NodeJS version:

You have to upgrade Node to 12 first, and then upgrade to Ghost V4, and then to Node 14 if you still want it.

After installing Node v14, apt-get wouldn’t downgrade it—the only choice was to sledge-hammer it and reinstall:

sudo apt-get remove nodejs
sudo apt-get install nodejs

Then you have to upgrade the command-line client!

Now npm is broken:

Reference: https://askubuntu.com/questions/1189230/what-is-the-location-of-node-modules

Command to delete all node modules

sudo rm -rf /usr/local/lib/node_modules/
sudo npm install -g npm

It couldn’t find the ghost command in my $PATH, so I had to kill the SSH connection, and then log back in:

Ghost won’t let you upgrade directly from v3 to v4; you need to upgrade the v3 version first, and then jump to v4 afterwards:

Staggered upgrade commands

ghost update v3
ghost start

… but it just hung here for a long time

• persisted after a reboot

Attempting the upgrade anyway

Tying up loose ends

Ghost command still hangs

Fix here: https://forum.ghost.org/t/problems-starting-ghost-2-25-1-on-debian-9-9/7912/8

Modify the systemD unit to point to the new Node install:

Commands to edit the systemd service

sudo vim /etc/systemd/system/ghost_blog.service
sudo systemctl daemon-reload

SONIC BOOM YOU WIN!

Suppose you are building a UI widget that accepts a list of image URLS.  When asked, the widget is supposed to download every image, scale it, and set it as an icon for a button.

A naive approach might be to use Python’s https://pypi.org/project/requests/ package to make a get request inside of a for loop, like this:

class MyContainerClass:
   def build_ui(self, data):
       for url in urls:
          res_img = requests.get(url)
     
          # waiting, waiting, waiting...

          pixmap = QPixmap()
          pixmap.loadFromData(res_img.content)
          pixmap = pixmap.scaled(240, 140)
            
          myButton = QtWidgets.QPushButton()
          myButton.setIcon(QtGui.QIcon(pixmap))
          myButton.clicked.connect(self.select_img)

This approach exposes you to a litany of problems:

• The UI will be totally blocked which the images are downloaded
• The images could be large, the Internet connection spotty, and each one could take a long time
• The images are downloaded one-after-the-other, meaning that your total download time will be the sum of the time it takes for every single request!

We can do better!

What is the fundamental nature of the problem?

The `requests` package downloads content synchronously (one-at-a-time), and we want to download everything in parallel.

How do I fix it?

There are many solutions to this problem.  One approach is to use https://pypi.org/project/grequests/ to spawn an event loop, where each request can be fired off at the same time, downloaded in parallel, and then processed in it returns.

Since we’re using PyQt5, which is backed by the excellent Qt UI library, we already have pre-existing classes that do exactly what we want.

Let’s skill up before we proceed

Read up on these Qt classes, if you’re not already familiar with them:

• QNetworkAccessManager (QNAM): https://doc.qt.io/qt-5/qnetworkaccessmanager.html
• QNetworkReply: https://doc.qt.io/qt-5/qnetworkreply.html
• Qt signals and slots: https://doc.qt.io/qt-5/signalsandslots.html

We will be using QNAM’s  `get` method to fire off the request.

** Here’s where this tuturial is different than all the other examples I found **

Instead of listening to QNAM’s “finished” signal, which will fire repeatedly for every single image, we’re going to connect a function to QNetworkReply’s `finished` signal, which you can see in the Qt documentation:

Architecture of the solution

We’re going to create a downloader class that inherits from QObject—that way, if we set this downloader as a child of the UI widget it’s supposed to update, the downloader will be able to access that widget to set its image.  As a bonus, Qt will take care of memory management, so that the downloader object will not be garbage-collected which the async downloads are running in the background.

The downloader will keep track of its own QNetworkReply, so that it can pull the image data out of the response when the `finished` signal fires.

Finally, we will instantiate our own QNAM instance and attach it to the container class.  This QNAM will be passed into each downloader instance, so they can schedule their own URL download.

Let’s take a look at the solution

Image downloader class (net_io.py):

from PyQt5.QtCore import pyqtSignal, QObject
from PyQt5.QtGui import QPixmap, QIcon


class ImgDownloader(QObject):
    def __init__(self, parent, req):
        self.req = req
        super(ImgDownloader, self).__init__(parent)

    def set_button_image(self, img_binary):
        pixmap = QPixmap()
        pixmap.loadFromData(img_binary)
        pixmap = pixmap.scaled(240, 140)
        self.parent().setIcon(QIcon(pixmap))

    def start_fetch(self, net_mgr):
        self.fetch_task = net_mgr.get(self.req)
        self.fetch_task.finished.connect(self.resolve_fetch)

    def resolve_fetch(self):
        the_time = time()
        the_reply = self.fetch_task.readAll()
        self.set_button_image(the_reply)

And here’s the new main loop:

from .net_io import ImgDownloader


class myContainerClass():
    def __init__(self):
        self.download_queue = QtNetwork.QNetworkAccessManager()

    def build_ui(self, urls):
        for url in urls:
            myButton = QtWidgets.QPushButton()
            myButton.clicked.connect(self.select_img)

            req = QtNetwork.QNetworkRequest(QUrl(url))
            downloader = ImgDownloader(self.form.btn, req)
            downloader.start_fetch(self.download_queue)

What did we just accomplish?

The QNAM download utilizes Qt signals and slots, so it does the actual download work “off the main thread”—which means that our UI is still responsive.
The downloads are accomplished in parallel, so we are no longer blocked by having to download them one-at-a-time, especially if some of the images download more slowly.

All of the download requests are fired off immediately.  QNAM rate limits these to 6 connections at a time, and this can be easily configured if desired.
The QNAM is managed by the container class, and each ImgDownloader instance is managed by the UI element (myButton) that it pertains to.

Suggestions or Comments?

Tweet at https://twitter.com/raphaeltraviss so that I can get this blog post updated—it takes a village to raise a developer.

References

• https://stackoverflow.com/questions/38923978/object-going-out-of-scope-and-being-garbage-collected-in-pyside-pyqt
• https://stackoverflow.com/questions/55641561/have-pyqt5-labels-dynamically-update-while-sending-http-requests
• https://stackoverflow.com/questions/44224454/how-to-call-a-method-asynchronously-in-pyqt5-using-python3
• http://zetcode.com/pyqt/qnetworkaccessmanager/
• https://gist.github.com/bootchk/9025575

Yesterday, I was an employee at a well-respected software consultancy.  Today, I’m completely alone in my office, and there’s nothing but silence.

Yet, I feel completely at peace with this moment, and gracious that I have the opportunity to embark on my life’s work.

Today I want to share with you four strategies that I used to overcome my fears of income, loneliness, and failure.

Starting with a single customer

One of the major problems with freelancing is that it takes about a year to build up a consistent client stream: and “no clients” means “no income”.  I solved the financial portion of this problem my living frugally, saving money, and doing expensive automotive and home repairs myself.

However, an ice-cold fear still gripped me:

What if I can’t find any clients???

In my case, this fear was particularly difficult to overcome, because I didn’t have time to freelance “on the side” while maintaining a family, a full-time job, and a household—that would have meant starting out as a freelancer with zero income, which would have been terrible for my mental health, forcing me into a scarcity mindset from the get-go.

To mitigate this fear, I kept my eyes and ears open for a full year, prior to leaving my job, searching for a tiny 5hr/week project, that I knew I could maintain while working full time.

When I embarked on my full-time freelance journey this morning, I started with an existing client relationship, and an existing income stream.

Even though the financial impact of this project was small, the benefit to my emotional health was huge.

A strong daily routine

In some ways, the age of COVID was very good to me.  I was able to work from home, which improved my productivity quite a bit.  More importantly, without the confines of the office or the daily two-way commute, I was able to set up a rigid daily routine of spiritual work, exercise, and writing software.

As time went on, I became very aware of my daily rhythm: when I was alert, when I was hungry, and when I was tired and unproductive.

I engineered my work day to revolve around my personal rhythms, and created a strict schedule that I could adhere to every day, and find success.

This morning, I woke up at the same exact time that I always have, turning on my computer, and began working, exactly within the same strict schedule where I’ve found comfort and success over the last year.

An awareness of the fleeting-ness of life

When COVID hit, it nearly stalled my freelance journey, because I feared different bad scenarios and consequences that could happen in the future:

• what if my family developed COVID; would my new ACA healthcare plan cover those costs?
• what if, after leaving my current company, I found myself alone and outcasted from the local dev community?
• what if my home needed a major repair, and I couldn’t cover the costs?

These fears were partially solved by creating contingency plans and strategies for handling the situations, should they arise.

However, the final bit of strength to overcome these fears came from a strange place:

Even before the COVID lockdown began, I made it a point to go on a daily walk.  During these walks, I encountered homeless people sleeping in the park, car accidents, and dried leaves being crunched under my feet.

During these moments, I would wonder about those people involved in the car accident, and how there was nothing they could have done to prevent that from happening.  Or I’d pass homeless people, imagining the situations they were born into, that they had no control over, and wondering if they'd ever find a way out of their suffering.

Simply being aware of the big-ness of the world, and the different cards that Fate could deal a person at any time, outside of human control, was enough to place my fears into their correct perspective—trifles, in the grand scheme of things.

A list of the good situations you’re leaving behind, and what created them

I’ve saved the best source of strength for last—spoiler alert: it’s another awareness, an act of consciousness that you can practice and improve over time.

Whenever I’ve contemplated leaving different jobs in the past, I always found myself gripped by an icy-cold fear: “Whoa, I have it pretty good here, am I really sure that I want to leave?  What if these good times never return?” This fear is compounded when I witness scores of people that I’m surrounded by, suffering in different ways, in their different jobs and situations that they find themselves in.

During these moments of fear, I’d list out all the ways that I was “winning” at my current job, and how I was so grateful and thankful at the fact that these opportunities existed at all:

You have it too good here; don’t go

But then I examined each of these situations, and asked myself, “What created the situation to begin with?”.  In all cases, I found that the good things in my life were created by two things:

• various external opportunities that I couldn’t control
• an internal decision to excel at whatever I was doing, which I could control

That second item was the key.  After walking through these various situations in my mind, I realized they were created by my internal essence expressing itself, not by the situation themselves, which come and go.

After making this list, I realized that the good times were created when I allowed my enthusiasm to through me and into my work, and therefore anything I could do to cultivate that enthusiasm would lead to a positive outcome.

And now here I am, doing my life’s work, and writing this blog post.

Good luck on your own freelancing journey, if you are considering it, and I hope these places where I've found strength will also help you on your own journey.

As a web developer coming into game development, I started off my projects using Git.  This worked well for code-heavy projects, where I was just using the Unreal Starter content, and other odd assets here-or-there.

However, when other projects needed multi-gigabyte assets from the asset store, I had two problems:

1. The size of the repo was vastly beyond the max size of the Git hosting providers, such as GitHub and GitLab
2. Each time these large binary files are edited, Git needs to store a revision, on every client that pulls the repo

To solve these problems, I looked to setting up Perforce, on my own Digital Ocean droplet.  In this article, I'll explain how I did it, especially the troubleshooting steps that I used to fixe my mistakes.

The Droplet

When I looked at the Perforce minimum system requirements at https://community.perforce.com/s/article/2957, I was surprised at how low they were.  I went with the lowest-tier droplet, and installed Ubuntu 18.04 on it:

<img src="https://blog.skyleafdesign.com/content/images/2019/11/image-4.png">

The Perforce installation

I followed the installation guide at https://www.perforce.com/manuals/p4sag/Content/P4SAG/install.linux.packages.install.html, and there were no surprises.  Add a GPG key and Apt repo, and install the helix-p4d package.  Done.

Next, head over to the post-install steps, and run the config script, which will ask you questions, and write your config file at /etc/perforce/p4dctl.conf/your_service.conf.  I accepted all the default options, which caused some problems...

Whoops!  We need a case-insensitive install

If we look at Epic's Peforce recommendations, we see that they recommend a case-insensitive install, to avoid compatibility problems in the future.  This is a great opportunity to flip into troubleshooting mode, and get familiar with the Perforce admin commands.

The first thing we need to do is look at the p4dctl commands, to find a command to stop all running services:

sudo p4dctl stop -a

Next, we need to delete our config:

sudo rm /etc/perforce/p4dctl.conf/your_service.conf

And finally, clean up the deleted service's saved data:

sudo rm -rf /opt/perforce/servers/your_sevice

Re-run the configuration script, and this time, choose case-insensitive!  Your new config file will be written to /etc, and you can start the server again with sudo p4dctl start -a.

Deleting a depot

In my case, I had already created a case-sensitive depot, which I needed to fully delete.

Looking at the p4 depot command, we can delete the depot metadata:

p4 depot -d my_depot

Done!

Setting a custom port

My server spends all day responding to SSH brute force attacks from Chinese servers, which I've partially mitigated by using non-standard ports for most of my services.  To set up a custom port for you Perforce service, just add it to your config file in /etc:

Setting up type maps

In the Unreal/Perforce integration guide, they recommend setting up typemaps to handle the Unreal binary content.  I haven't done this yet, but it's probably best to run the P4 commands to set those up:

p4 -P YourPassword typemap

I haven't set this up yet, because I'm not sure if it affects the entire server, or just a particular depot.  I plan on using this server for other C++ projects, and I wouldn't want my Unreal typemaps to interfere with that.

Creating your first depot

On your client machine, head over to the Helix Core GUI client download page, and download the bundle--it contains P4Admin, P4V, and some other tools.

First, we need to create a depot.  Open up P4Admin, and connect to your droplet's IP address and custom port.  Go to the Depots tab, and right-click to create your new depot:

That's it for P4Admin!

Creating a workspace

Finally, we need to set up a workspace (with cross-platform settings), and populate the depot with the initial files.

Open P4V and log in to your server.  Go to the Workspaces tab and right-click to create a new workspace.  Right-click to clear all depots, except for the depot you just created.  It'll prompt you to add files. Click on the advanced tab, and check the box for compression and rmdir.

For cross-platform work, set the line-ending handling to shared.

In an ideal world, Visual Studio and Xcode would be able to handle any type of line ending, but for now, we need this option checked--this will make Perforce work like Git's default autocrlf.  I think it's insane for a version control system to manage line endings, but it is a necessity from my limited experience

Ending

There you go! You should be able to set up workspaces and build the project on any machine, checkout files, and submit Changelists.

Where to go from here?

As it stands, we need to manually checkout files in P4V, before we work on them in the Unreal editor.  To streamline and automate this process, head over to the Perforce/Unreal integration instructions, and set it up!