Book Review: "On The Clock" by Emily Guendelsberger

Summary: "On The Clock" by Emily Guendelsberger tells you about the lives of hourly wage workers. It is a stunning book. Please read it.

Most of the books I read come recommended from friends or book reviews. Most of them get glowing reviews, so it is no surprise when the books are great.

Every so often I pick something up at random, and am pleasantly surprised at its quality. Very rarely, I pick something at random and it is profound and enlightening. This book was a random pick, I expected little.

"On The Clock" by Emily Guendelsberger is profound, enlightening and much more. Emily (I'm Californian, we're on first name basis with everyone) writes about the lives of front-line wage workers in three different companies. She works for a month at an Amazon fulfillment center, then a month at a customer call center, then a month at McDonalds, all in the US. She is honest about her past as a journalist, and observes her life, and the lives of her co-workers.

A lot of ink is spilled on the unfairness of capitalism, most of it from a safe distance, and from a high perch. Most people talking about low-paying jobs have never worked in those jobs themselves, or worked in those jobs decades ago, when conditions were better. CEOs who talk about how well their companies are run have never worked as wage-workers in any company, let alone their own companies. Most consumers have never done these jobs. Emily is the first person who braves these jobs to tell you how terrible they can be. Here's a sample of her message, via an article she wrote on the topic of fast-food worker burnout. The burnout is both metaphorical and literal. A 2015 survey found that 79% of fast-food employees had been physically burnt on the job in the prior year, some more than once.

Burnout is a trendy, rich-person luxury. In these jobs you see far worse. You see income insecurity up close. You see employees choosing to work when sick, you see employees support each other during difficult times, and the unending misery of the work. Workers pushed to their physical limit, on their feet, and walking all day. But hey, single-packages of painkillers are provided by the employer. For free! Or workers stretched to capacity, handling support call after support call, with short breaks in the middle, handling verbally abusive callers.  Overworked, pushed, paid too little.  No health insurance, no money and no time, with their life revolving around their job. No savings, no retirement, it is a bleak life, and one that most of us don't have an experience with.

Ok, so obvious question, "If these jobs are so bad, why don't these workers do something else...?" The short answer is that all jobs are like this at the bottom. No employer has to provide wholesome, nourishing jobs, so they don't. You can choose between a physically demanding job, or a verbally abusive job, but they are all terrible in some way. Attrition is high, workers are expected to 

I'd bet you hate your cell-phone or Internet provider? Why haven't you switched providers? Because the other company is just as bad, or you don't have a choice of providers. Same concept.

This book made me ponder and lament, well past I had read it. I am reminded of Emily's idea of doing a tour-of-duty as a wage worker. Every manager, every CEO, every elected representative should have do a contemporary hourly job. Having flipped burgers in 1980 is not enough, the US employment market has come a long way since then. This book is similar to "Nickel and Dimed", by Barbara Ehrenreich, which is mentioned in the introduction. Things are different this time around, as more companies outsource their dirty work. So while Emily works for McDonalds, Amazon and AT&T, she is never directly employed by them. This way the companies can, correctly, claim that they treat their employees well. The Amazon delivery person is rarely employed by Amazon too. It's the invisible tier of jobs, too shitty to even mention in a company's press reports. Everyone wants a photo-op with "Real America", and then go back to their mansion, their enormous paycheck and their private jet.

This book has the potential to shape the national debate on capitalism, income disparity, and difference in compensation between front-line workers and CEOs.

Please read this book, it is a revelation.

The Oversubscribed American Child

Kids in the Bay Area are oversubscribed, both in summer holidays and on evenings and weekends of school-days. During the school year, children spend time in after-school activities, coming home shortly after 5pm, when the parents are done with their work. Sorry, I misspoke. Children don't even come home themselves, they are driven back by parents.

In the evenings, kids are busy with soccer, football, basketball, piano, chess, martial arts. Sometimes all of the above. The parents are shuttling the kid back and forth to these events, and are themselves unavailable for any social commitments of their own. As a result, the parents social life is restricted to meeting the parents of other kids at all these events.

All these activities are enriching. When you were a child, you did not have access to a tennis court, and your child loves tennis. There is no harm in this one activity. Then again, your child also has an interest in singing, dance, basketball, and swimming. Which activity can we deny them, possibly closing the door to a life-long joy?

Near the swimming pool, you see parents coaxing, encouraging, threatening, cheering as the kid swims laps. The 10 year old is getting ready for the swim team. This happens thrice a week, after school, or before school. There are swim meets, there are practices, and the parent tugs along for the ride.

Sure, the kids have friends, but how deep can a friendship be when you are only available on specific time slots, a few times a week. The swim kids are the kids' friends. Your child needs to make the swim team, to retain their social life.

No longer a tiger mom, there is now a tiger child. In fact, tigers all around: the parents are themselves ambitious, chasing a promotion at work, and a soccer victory for the child, sometimes from the same soccer field. Where do you draw a line between the ambition and desires of the child and that of the parent?

A family just moved in and their kid would play on a few occasions with other children in the neighborhood. A few weeks later, the school year starts. In the evening, this kid was invited to join in a game. Alas, the kid is busy, and is only available from 4:30-5:30pm. On Fridays. That was the only time in the entire week that this kid is free to be a child. My full-time job keeps me busy, and yet, I have more free time.

Ah, but you are independent, and think for yourself. Your kids are free-range. Tell me, how free-range can your children be, when all other children are at camp? During summer vacations, your children are bored stiff at home; as everyone around them is busy all day, unavailable, away at soccer camp, swimming camp, then music camp, week after week for the whole summer? You child begs you to enroll them in the same camp as their friend, that is the only time they can both be together. Parents mail out schedules to see if kids' friends can enroll in similar camps over the summer. Stay free-range and your child has nobody to play with. Enroll your child in some camp, and at least they'll have some social life. And all that enrichment.

Soon, there are no children to be seen. Can you can tell if school is in session from walking around your neighborhood? Does anything change from May to July?

Everyone knows the suburbs are criminally empty. Distances are vast, and every friend is twenty minutes away on foot. The library, the ice-cream shop, ... everything is far. Your child cannot cross that six-lane arterial road by themselves! No bike lanes, and anyway they'll get hit by an SUV that cannot see them. Soon enough, they have to be driven. There are some lucky cities and some fortunate locations, but the vast majority is not navigable by 12-year olds.

"When I grew up", ... starts every critique. This is the usual refrain because parents have access to just two childhoods: their own, and their kids. All conclusions are drawn from these two data-points, until a grandparent comes by to cast a tie-breaking third. Was your generation different from this? What about the grandparents?

I have no doubt that this will lead to many world-class artists, musicians, and athletes. The next Michael Phelps is swimming near you, as their parent yells from shore. But this small amount of Great comes at the cost of a lot of Good. That one Olympic winner will cost us a generation of lost childhoods. Thousands of children will grow up to be detached from others: lonely, anxious, uncomfortable in their skin. Cellphones, social media, TV, gaming will get blamed. Why are kids so lonely these days? Because they don't have friends, and have never learned to cultivate real life friendships. They were ferried around in the backs of SUVs, in comfortable car seats, from the chess club to the soccer meet, and then to ballet. Everything was regimented, sanitized, clinical.

The real world is messy. Being an adult requires comfort with this mess. True friendships grow over time, born in experimentation, nurtured by shared experiences and bad collective judgement. You make a friend when you help the kid with a sprained ankle. When you disagree and fight over how you were treated. You realize you have a true friend when he helps you jump the fence and lies about it. You don't need 'play dates', you need play. You grow to be an adult through the risk-free life of a child, discovering the contours of the world.

Encrypted, private github repositories

 Github allows unlimited private repositories.  Sometimes, though, you need additional privacy.  I store many of my passwords and other sensitive information inside plain text files, and would like to version them with git and then upload them to github.  Even with private repositories, these text files are available to github employees, stored in cleartext in backups, available to law enforcement, etc. I'd like these files to be stored with encryption at Github, to ensure that only I have access to these files.  While I could do this with my own git instance, I like the availability, backups and worldwide reach of github, and I want to avoid maintaining one more thing.

This tutorial shows you how to have encrypted repositories at Github. In an encrypted repository, the contents of the repository at Github are encrypted with GnuPG. On all machines outside your own, these files are encrypted and secure. Nobody, not even employees at Github can examine the contents of the files. Without your gpg secret key, the entire repository at github is opaque to people.

The repositories can be private or public. For an encrypted repository, there is little difference between a public and a private repository, as the contents are not visible without your secret gpg key.  The benefits to having a private repository is that nobody can tell the repository exists, and you have an additional hurdle of an ssh key to download the repository. The benefit of a public repository is that you can have many contributors, and you can git clone the repository from anywhere, as far as you have the gpg secret key to decrypt the contents.

For this mini-tutorial, you should already have git working with ssh keys.  If you don't, follow these instructions. This tutorial is for Linux.

1. Packages: On Linux, install the relevant packages:

apt install git-remote-gcrypt gnupg

2. Github repository: Create a private repository in github. I created an example public repository here though you might want to create a private repository.

export PROJECT=username/reponame.git

3. Create gpg keys: if you don't already have one using

gpg --full-generate-key

Remember the passphrase used here.  If you lose/forget this passphrase, you will not be able to decrypt this repository, ever.  The keyring is usually stored in .gnupg/pubring.kbx 

List the keys, and write down the key-id:

gpg --list-keys

export KEY_ID=<what you got above>

For example, my KEY_ID is BAB59DAD24A79C69D4D3E5451482830DA6C99CAA.

4. Configure git: Go over to your existing git repository (or initialize one, and then add files to it)

Specify this ID as the one that you will encrypt with:

git config --global user.signingkey $KEY_ID

And as the participant allowed to contribute to this project

git config remote.origin.gcrypt-participants $KEY_ID

5. Remote: Add the upstream remote like this:

git remote add origin gcrypt::git@github.com:$PROJECT

git push origin master

That should prompt you for your gpg password you entered when creating your key.  Now, you have a git repository that is hosted at github, and the contents are encrypted.  This is an example of what you should see.  At this point, the repository could be private or public, since nobody can read the contents unless they have your GPG key.

6. Clone: To clone this repository on another machine, remember to add gcrypt:: before the git@github.com, and the new machine should have both the ssh keys (to clone a private repository) and the gnupg keys (to decrypt the contents of the repository)

git clone gcrypt::git@github.com:$PROJECT

For example, I do:

git clone gcrypt::git@github.com:youngelf/example-encrypted-repository.git

Done right, it should request the gpg password for the private key, and show you this output from gpg

gpg: Good signature from "Your name <email@example.com>"

7. git pull, git add, git push

At this point, this is a normal git repository, you can use git commands.  I find it more convenient to use the git clone'd directory, as it sets the configuration for branch.master.merge and branch.master.remote correctly.

Python, pip, venv, package managers

 Upgrading an old computer from one Ubuntu LTS version to the next: 20.04 to 22.04,...

I came across an annoying error when running do-release-upgrade:

AttributeError: 'UbuntuDistroInfo' object has no attribute 'get_all' 

This shows up in other forms, folks online complain about errors of the kind:

AttributeError: 'DistUpgradeController' object has no attribute 'tasks'

Now the usual reaction is to blame Ubuntu's release. Free Software, and it's lack of reliability, ...

Except that I've upgraded several computers from 20.04 to 22.04, so I know that the process works well.  Quite unlike my usual nature, I decided to investigate.

The fix: pip uninstall distro_info

I'll spare you the full investigation, because life is short.  The end-result is that this is caused by Python's incredibly brittle package management.  On an Ubuntu system, you have both packages installed by the package manager (dpkg/apt) and by pip.  And pip can install software for the whole system, when run by root.  In this case, the package 'distro-info' was installed by root using pip, and that overrides whatever distro_info object exists in Python.  Since the Ubuntu installer expects certain behavior from the class UbuntuDistroInfo, the package install fails early.

The fix, again: pip uninstall distro_info

Having solved it, let's reflect on the incredibly broken nature of Python packaging. Why was a user unable to upgrade their Ubuntu system, and why was pip at the bottom of the mess?

First, the package namespace should be unique.  The package distro_info should either not conflict, or Ubuntu's package manager should create a unique name that doesn't conflict. It should be called 'ubuntu_distro_info'.  If these are maintained by the same team, then new versions should be very careful removing methods from previous versions.  Software versioning is complicated, but there are many lessons learned over the years.

Second, the parallel universes that exist between pip and dpkg/apt are a mess. root should not be allowed to pip install packages. 

Third, venv is a crutch. Package maintenance is difficult, and dependencies are tied specific version of packages, and so you need virtual environments to create each parallel universe.  This gets the job done, but pushes a lot of maintenance headache to the end-user of the packages.  Now, in addition to the dpkg/pip mess, you individual pip messes in subdirectories scattered all over your filesystem.

Fourth, python versus python3. In Debian-based systems, python and python3 have different namespaces for packages, and this further confounds the issue. So if you install python-libraryname, you might also have to install python3-libraryname.

Finally, python errors are broken. This is a pet peeve of mine.  Python's errors are the bottom of the stack, and the full stack trace.  But these are completely unhelpful in explaining what might be involved. If package versions are in flux all the time, python software should start off verifying versions and sanity.  Imagine if this failure had happened half-way in the install process, leading to a broken machine. Fragile systems need defensive programming.

All this leads to an incredibly brittle software setup: where packages are frozen in time (in venv directories), but also no way of querying what versions of software exist on the system.  Once an environment is set-up, there is little confidence that it will continue working.

Christone "Kingfish" Ingram

I recently heard an artist called Christone Ingram, who goes by the stage name "Kingfish". He plays the blues on guitar. It blew me away: the guitaring, the singing.  After many years, I have come across a contemporary artist who is awe-inspiring.

It all began when I chanced upon an album called "662" by someone holding a Stratocaster.  I wasn't expecting much.  What could have been just another blues musician turned out to be a genius of our time, for each song was delightful.

If you like blues, or classic rock, give it a listen.
  This Youtube video, for example, gives you some idea of his level of skill. You can also preview his music on his website: the album I heard is called "662", and that's his second album.  His first album is called "Kingfish".

I find 'influencers' hollow.  Many current influencers are popular solely because they're popular. Some of them might be good looking, but they have no skill beyond that accident of birth. For every influencer that takes our attention, there are real artists, folks with skill, folks working hard. We ought to devote  our time on folks who have skill, who advance art.

Book Review: Systems Performance 2nd ed, by Brendan Gregg

Summary: "Systems Performance", by Brenden Gregg covers end-to-end performance for Linux-based systems. If you run Linux software, you will learn a lot from this book.

From its rough and loose beginnings, Linux has become a force in the commercial world. Linux is the most pervasive, most readily available system that you can experiment with.  Starting from the $10 Raspberry Pi to the multi-million dollar Top 500 supercomputers, Linux runs on everything: laptops, desktops, phones, cloud instances.

Despite widespread adoption, there is little documentation to get a thorough understanding of system performance. I routinely see veteran engineers struggle with performance bottlenecks. Folks revert to running 'top', and trying to infer everything from its limited output.  The easy answer is to over-provision hardware or cloud instances to cover up sloppy performance. A better answer is to get a solid understanding of end-to-end performance; to find and eliminate bottlenecks.

"Systems Performance", by Brenden Gregg covers the entire area of end-to-end performance of all components: CPU, RAM, network, block devices.  The second edition of this book is focussed on Linux, and covers many tools and utilities that are critical to understanding every level of the stack. If you have written any software on Linux, or intend to write any software on Linux, you need a copy.

First, the good:

1. There is an overview at the beginning, and then a deep-dive on specific system resources (CPU, RAM, block devices, network). You read the overview to understand the system at the top-level, and based on your system and bottlenecks, you can read the in-depth sections.
2. There's coverage of pre-BPF tools (perf, sar, ftrace) in addition to the newer BPF-era tools like bcc and bpftrace. 'perf' probes are easier to use, and available on more architectures, for instance. BPF-based tools can be a slog to install, or might not have good support on fringe architectures and older kernels. No single tool can cover every need, and good engineers need to understand the full tool landscape. This book provides a wide overview of most tools.
3. The book provides a methodical look of the full system, with tools targeting individual levels of the system components (example diagram). This process helps isolate the problem to the correct component.

The not-so-good:

1. The book is repetitive. Since it expects some readers will start reading a deep-dive, it repeats the USE methodology at the start of most chapters. Folks reading it cover-to-cover will find themselves wondering if they have seen the material already.
2. Print quality is worse than the previous edition. The fonts are thin and dim, the pages bleed through, and the graphs need more contrast. The first edition was a high quality printed book, and the second edition is worse in this department. Since this is a reference book, a physical copy is better than an ebook. You will mark pages, put sticky notes, and highlight tools that are more pertinent to your work. Luckily, the binding holds up to heavy use.
   I really wish the third edition comes with better print quality, and is hard-bound.

Every software engineer should be familiar with end-to-end performance: how to think about it, how to locate trouble spots, and how to improve the system.  This book will give you a firm foundation of performance that should help on most desktop, server, and cloud systems. 

You will probably not get this understanding from a scattershot reading of online documentation and Stack Overflow articles. Online articles are limited in scope and accuracy, and don't provide a comprehensive view of how to think about performance. This topic deserves a book-length treatment.

Image Courtesy: Brenden Gregg

Tensorflow 2.8 and Jax 0.1.76 for NO AVX cpus

 In what has become a tradition, I compiled Tensorflow for my no-avx CPU.  This time, the installation was more complicated because of a dependency on jaxlib. I had installed jax either through pip3 or through Debian's repositories (apt-get tool). The jaxlib was compiled with AVX support and would not work on my computer.

So I spent some time getting Jax sources and compiling those without AVX support.

Here are the two files for older Intel CPUs:

jaxlib-0.1.76-cp38-none-manylinux2010_x86_64.whl

tensorflow-2.8.0rc0-cp38-cp38-linux_x86_64.whl

Unless you have compiled your own jaxlib, you will need to download both. The jaxlib should be useful with the native 'jax' install from pip3 since the jax library only contains Python code.  As I understand, the jax library does not contain native code.

You could also use the jaxlib in isolation for playing with Jax.

To install them, download the whl files to disk, and run 

pip3 install filenameHere.whl

These were compiled on a cpu with the following flags in the output of /proc/cpuinfo:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities

Both these wheels work great on my Core 2 duo and another Pentium CPU that didn't have AVX support.  Compiled with Python 3.8, they should work for most Linux distributions, assuming the dependencies (numpy, absl-py, scipy, flatbuffers, tensorboard, ...) are installed. pip3 should get the dependencies you don't have. None of the dependencies contain any native code that requires AVX instructions.

Tensorflow 2.5 without AVX

 I was playing with Tensorflow, needed a new version, and realized that a Tensorflow release without AVX still doesn't exist.  My prior post on Tensorflow without AVX had been beneficial to people, so here's Tensorflow 2.5 for Linux.

Tensorflow 2.5 for Linux without AVX support (155 Megabytes)

and the prior link

Tensorflow 2.3 for Linux without AVX support (126 Megabytes)

Download the file, and then run

$ pip3 install -U filename.whl

Cloud of DOS machines

tl:dr;  ssh guestdos@dos.eggwall.com

Retro computing is in. New computers and new systems are always fun. But to really appreciate the arc of history, use something old. Something ancient. Something you never really learned.

To help you get your retro computing fix, here's a bank of DOS machines. Rather than MSDOS (copyrighted, etc), I've got FreeDOS, an open-source implementation of DOS. There is the editor 'edit' and 'edlin'. 'foxcalc' is a sweet calculator. 'help' will tell you what to do. For the Assembly geeks, there is always 'debug'.

This is running on my custom cloud of DOS machines. Connect to your instance today. You can use either ssh or telnet:

$ ssh guestdos@dos.eggwall.com

Telnet works too:

$ telnet dos.eggwall.com

Username: 'guestdos', no password.

When you are done, type 'halt' and travel back to the present time.

Look around the file system. There's compilers for C, Pascal, an IDE, emacs, vi, BASIC, and some games. Using a system this old gives you an appreciation for systems today. You miss the many conveniences you take for granted.  You are also closer to the machine, you can modify arbitrary memory addresses. Nothing to get in your way.

Powered by the FreeDOS project, and magic.

Book Review: UNIX and Linux System Administration Handbook, Evi Nemeth et al

 tl:dr; If you are a Linux/BSD user, this book will help you understand your system.

There are Stack Overflow answers, random blogs and tech articles. You can find man pages. What they lack is a comprehensive view of the full system, with historical background, a theme tying the topics together. They lack clarity, depth or accuracy. Nearly always, they lack humor.

UNIX and Linux System Administration Handbook is a very broad overview of all aspects of Unix system maintenance and upkeep: going from installation, package installation, and routine chores. It has in-depth coverage of critical sub-systems like networking and disks. These are covered over many chapters, devoted to hardware, services, best practices. There is deep coverage of user management, security issues for sites, authentication, logging. It has the obligatory chapter on shell programming, with bash & Python.  There's information on config management tools like Ansible. It has much more than a system administrator would need through their first year. It is a large book and you will find yourself revisiting specific sections. It helps to keep an electronic copy for quick searches. The index and the table of contents are ideal for an admin with a paper copy.

The book gives you a superficial overview of virtualization, cloud systems, containers (Docker), and container orchestration tools like Kubernetes, Docker Swarm, Apache Mesos or Marathon. These sections are light: it gives you a background, why these systems exist and how to think about these systems. You can start a journey there if you want. These chapters were added later, and the coverage is satisfactory. Users need to have a rough idea of these, but each of those topics deserve a book-level treatment. The authors have kindly provided references to chase for the interested readers.

I've been a long-time Linux user, and I found the book's treatment of systemd level-headed and valuable. It provided a great preface with motivations for systemd, how it has evolved, and how best to use systemd to keep a system humming along and keep the system manageable.

The book was funny in all the right parts, without overdoing the humor or trying too hard.

It was published in 2017, but most of the sections are quite current. The book covers major Linux distributions (CentOS/RedHat and Debian/Ubuntu) and FreeBSD. The FreeBSD sections were great, especially alongside Linux so I had a Rosetta stone. The book's primary author, Evi Nemeth, is one of my favorite technical authors. Sadly, she passed away before this edition of the book was published, and I commend the current authors for carrying on her tradition, and their work on this edition.

In 2021, the diversity in Unix systems has shrunk incredibly. Solaris is history; FreeBSD is relegated to core adherents and the occasional technical Mac OS user. Cloud systems are gathering behind Linux instances. The ease of creating a Linux instance means that more users will find themselves responsible for a Linux machine. Unix used to mean SunOS: something formal, expensive. Something meriting a real education. Now, Linux powers everything from your phone to your Top500 supercomputer. A cloud instance can be yours for dollars a month. The power is still there, if you can invest time in learning it.

Veteran system administrators will enjoy filling gaps in their knowledge. The casual user will gain a deeper understanding of their system with this light, engaging, fun read.

Unicode URLs

tl:dr; Get URLs that cannot be verbally communicated.

URLs can only be in English characters (really, Roman characters) and a select few special characters like -.  So how is はじめよう.みんな a full domain name, even though it is written in Japanese (Hiragana script) and a dot?

The answer lies in character encodings, and translation done by the browser.

There is an encoding of characters called Unicode. That is what allows English, Japanese, Hindi, and other characters to be written on a single document.  URLs are Internet "addresses", the stuff that shows up in the address bar of your browser. These URLs can only be coded in ascii characters (a-z, a-Z, 0-9 etc). Due to their long history, URLs cannot be specified in Unicode. But creative minds have been hard at work here. There is a coding for Unicode that uses only the characters that are allowed in URLs. This encoding is called punycode.  Punycode URLs start out with 'xn--', but the browser transparently renders the unreadable combination of ascii characters into their corresponding unicode characters.

So in the https://はじめよう.みん example above, はじめよう is xn--p8j9a0d9c9a and is みん is xn--q9jyb4c. So the full URL is https://xn--p8j9a0d9c9a.xn--q9jyb4c/  If you can remember all those characters, you can type them out by hand. Your browser, detecting the punycode beginnings 'xn--' will decode the characters xn--p8j9a0d9c9a into はじめよう and xn--q9jyb4c into みん.

Try this translation yourself using online punycode converters.

This mechanism works for Top Level Domains (the .com in mail.google.com), domain names (the google in maill.google.com), or sub-domains (mail in mail.google.com)

What do you do with it? You can create URLs that look short but can only be recreated if the person can read/write the language. Here's an example: विक्रम.eggwall.com. This is a URL that can be clicked on but can only be verbally communicated by Hindi speakers.

You can mix-and-match different scripts. Here's an example with different scripts that I cannot even read. In the screenshot here, you have a URL. Try typing that address out.

There are some Top Level Domains (TLDs) that don't allow for punycode domain names. .fun, for instance doesn't allow punycode URLs. Try registering विक्रम.fun, for example. But you can always register a subdomain with punycode. So can have विक्रम.circuits.fun.

FreeBSD, the alternative

 I like FreeBSD. Something about a barebones, simple UNIX that is clean at the core. Not particularly a pragmatic choice, I'll gladly admit. For most needs, a Linux installation is both simpler and more capable. More people are familiar with Linux, so this is a niche world.

But FreeBSD is a hobby, and a particularly gratifying hobby.

The simplicity of FreeBSD is a huge draw, you can get very far with an ancient Intel machine and 1 GB of RAM. The source code is a pleasure to read, the system is easy to modify, and there is relatively little of it on a base install. Full system upgrades are clean and easy, the ports system is elegant though often impractical.

For most people, the easiest way to run FreeBSD would be to use a virtual image, and boot it up using qemu (Linux) or Hyper-V (Windows). Both work very well, both support networking, which is really what you're looking for. Face it, you're not booting into FreeBSD for its giant catalog of games or media utilities.

Get the VM images from here. qcow2 works for QEMU, vhd works for Hyper-V, and vmdk works for VMWare. 

For QEMU, I use the following commandline:

qemu-system-x86_64 \
        -m 2048 \
        -enable-kvm \
        -hda bsd-with-jail.img \
        -net user,hostfwd=tcp::10023-:22 \
        -net nic \
        -curses

That specifies:

• (-m 2048) 2048 MB RAM
• (-enable-kvm) use the Linux KVM hypervisor
• (-hda bsd-with-jail.img) the disk bsd-with-jail.img as the first hard disk 
• (-net nic)  a full Network Interface Card, make your FreeBSD setup trivial as it can get a network interface called em0
• (-net user, hostfwd=tcp::10023-:22) network translation is user-level, so no root permissions required and forward port 10023 on the host to port 22 on the guest. This allows you to ssh to the FreeBSD guest by ssh'ing port 10023 on the host.
• (-curses) Use the text-based interface using libncurses rather than a graphical interface. Great for running on cloud instances, or through GNU screen so you can leave it on forever.

Other ports can be forwarded for running web servers, dns servers, ...

The same setup can be replicated on Hyper-V. For network, you can use the default switch. Port forwarding is more complicated, and you could achieve it with ssh tunnels on the Windows host or in the FreeBSD guest.

FreeBSD runs fine with 1GB of RAM, and is a great system to learn the fundamentals of computing. You can play with Dtrace, create constrained environments called Jails, or modify the behavior of a clean UNIX kernel.  A lot of this knowledge translates to Mac OS, as the OS X kernel is a hybrid of FreeBSD and Mach, and the userland is very similar to FreeBSD.

You can easily run a dozen jails on 1G of RAM, depending on what you do with them.  Each jail is separated from the other and can be a low-cost, throwaway computing environment. Similar to Docker, or Linux namespaces, root in a jail is safe, and cannot damage the host FreeBSD system.

Set up your own UNIX sandbox today!

Mac OS Kernel (XNU) source, on github

 Few people know that Apple releases the source code for many components of their MacOS software. Many are surprised when they hear it.

Apple has a full site for their open source releases. Their most recent Mac OS release is for 11.0.1 (Big Sur). Not all source is released. There are the usual UNIX utilities: Perl, Python, Bash, and userland tools that make a functioning system. Functioning, but not always updated, as Apple doesn't always track the latest versions. They still ship Python 2.5, and other tools can be many years old. This is why you need to get a functional, bugfixed UNIX userland with homebrew or other tools.

The most interesting source there would probably be the kernel, called XNU. It is a combination of Mach, a microkernel, and BSD, a monolith. There's some Solaris code there, some Joyent code, some Sun code, NeXT, UC Berkeley, Carnegie Mellon code, and of course, Apple code.

Apple does have an official github profile, which also hosts xnu. But it was last updated two years ago, and stands at version 4903, while the latest version is 7195.

Their open source browser is passable but doesn't show every file properly. I find it much easier to read through code using Emacs so I pulled it into a github repository if anyone wants to clone it or browse it online.

Image credit: Apple Inc

Offline-first, the design of great picture gallery websites

tl:dr; How to use service workers to make your website responsive and functional offline

Websites have come a long way from the boring world of static <html> markup. There's a lot of new functionality in browsers. This new functionality brings benefit for big websites, but these techniques are useful for small personal websites as well.

One of the more important changes has been the support of Service Worker in many browsers. A Service Worker is Javascript code that can intercept network requests, and satisfy them locally on the client machine, rather than making server calls. There's a lot you can do with Service Workers, but I was most interested in making my home picture gallery work offline.

I wanted to allow a person to load up the gallery and be able to view it on their mobile phone or desktop even if the connection is lost. My photographs are shared with static html that I create using sigal, a simple image gallery software written in Python. It uses the Galleria library to create web pages that are self-contained. Since the galleries are static html & Javascript, they make a great case study for simple and fast web pages. In the current gallery, the images are downloaded as they are needed, with the next few images prefetched so the user doesn't have to wait. I wanted to make this entirely offline-first, so the images are downloaded, stored offline. Each image in my gallery is 150kb, and galleries have 10-20 images in them. The entire gallery is roughly 4mb, which is tiny. As a result, it loads fast and can be cached offline.

You can always implement your own Service Worker, the interface is straight-forward. If you just want to use Service Workers for browser-side caching, there is a much simpler alternative. We're going to use the upup library, which is a Javascript library that can be configured to store content offline, and cache it.

First,  Service Workers need HTTPS support. Get yourself a certificate with LetsEncrypt. This is a nonprofit that has issued 225 Million certificates already, and are valiantly helping the entire web move to 100% HTTPS. Get their certificate if you don't have it. Heck, get HTTPS support even if you don't want offline first on your website. I heartily endorse them, I have moved to HTTPS thanks to them. You should too.

Now, let's add UpUp to your website. It is important where you place it. The upup library can answer requests for a specific scope (subdirectory of your website). It sets its scope based on its location on your website. Since you want the library to serve content to your entire website, you want the library to be as close to the root level of your website. Let's see a concrete example in action.

Let's say your website is gallery.eggwall.com. If you put the javascript at gallery.eggwall.com/js/  then the javascript can only cache offline content for /js and not for gallery.eggwall.com/g_Aircrafts/.  To serve content for the entire subdomain gallery.eggwall.com you want the Javascript at gallery.eggwall.com/

We're going to put it at gallery.eggwall.com, with this in the html:

    <script src="https://gallery.eggwall.com/upup.min.js"></script>

So download the upup library and put the contents of the dist/ directory at the root level of your website. Putting random javascript like this is usually a bad idea, so examine the source code, make sure you understand what the Javascript is doing. The service worker source is much more important than the framework code.

Now that you have the library in place, invoke the UpUp.start method, and give it the base web page (index.html) and all the content that you want it to cache. The references here have to be relative to the location of the upup.sw.min.js. If you put the library in the root of your page, all the references here have to be relative to your root page:

    <script>
      UpUp.start({
          'content-url': 'g_Aircraft/index.html',
          'assets': [
              "/static/jquery-3.3.1.min.js",
              "/static/galleria.min.js",
              "/static/themes/classic/galleria.classic.min.js",
              "/static/plugins/history/galleria.history.min.js",

For simple pages like this, I find it helpful to include the <base> tag to remind me that everything is relative to the root:

<base href="https://gallery.eggwall.com">

 On this gallery, all images and content is stored in the subdirectory g_Aircraft/. All thumbnails are stored in g_Aircraft/thumbnails/. So you want to load up all the images in upup.start:

    <script>
      UpUp.start({
          'content-url': 'g_Aircraft/index.html',
          'assets': [
              "/static/jquery-3.3.1.min.js",
              "/static/galleria.min.js",
              "/static/themes/classic/galleria.classic.min.js",
              "/static/plugins/history/galleria.history.min.js",
                 'g_Aircraft/_DSC1984.JPG',
                 'g_Aircraft/thumbnails/_DSC1984.JPG',
                 'g_Aircraft/_DSC1986.JPG',
                 'g_Aircraft/thumbnails/_DSC1986.JPG',
                 'g_Aircraft/_DSC1989.JPG',
                 'g_Aircraft/thumbnails/_DSC1989.JPG',
                 'g_Aircraft/_DSC1991.JPG',
                 'g_Aircraft/thumbnails/_DSC1991.JPG',
                 'g_Aircraft/_DSC1992.JPG',
                 'g_Aircraft/thumbnails/_DSC1992.JPG',
                 ...
                 'g_Aircraft/_DSC2148.JPG',
                 'g_Aircraft/thumbnails/_DSC2148.JPG',

          ]
      });

    </script>

You don't need to change anything on the server for this. I use nginx, but anything serving out static pages will do just fine. Offline-first changes your server-side metrics because many requests are handled directly on the client. So you won't be able to see when the client loaded the page again. Browser-side caching messes with these numbers too, so if you will have to roll your own Javacript if you want perfect interaction tracking.

These changes are fine for browsers that don't support service workers. Older browsers will be served the static content. Since they don't initialize a Service Worker, all requests will go to the server, as before. The Upup.start section just gets ignored. Browser-side caching will continue working as before, too.

With this, the UpUp service worker will cache all the content specified in assets above. The user can go offline, and the page still functions normally. The gallery demo is available if you want to play with it.

Service Workers add complexity. You can debug the site using Chrome Developer tools -> "Application" -> "Service Workers" or "Web Developer" -> Application -> "Service Workers" in Firefox. You want to check if the service worker initialized and is storing content in "Cache storage" -> upup-cache.

Here's a demo video on Android. You can see the user load up the site on their mobile browser, go offline, and still navigate normally.

Audio feature extraction for Machine Learning

tl:dr; Books and papers for audio processing, for building Machine Learning models.

I've been experimenting with Machine Learning for audio files.

Much machine learning literature for music deals with MIDI files, which are a digital format for specifying notes, duration and loudness. This is the format to use for models that work on the level of individual notes. A simple introduction for training such models is using the book "Hands On Machine Learning 2nd edition" (2019), An exercise in Chapter 15 (RNNs) introduces you to Bach chorales, and shows how to generate chords from digial music. Google's Magenta project has datasets and models for such discrete note-level training, generation and inference.

While MIDI is a convenient format for discrete music, most music data is stored as waveforms rather than MIDI. These are either raw WAV / FLAC, or encoded with the MP3 or Ogg Vorbis encoding. Extracting features from these files is considerably harder and requires a good understanding of audio analysis and the kinds of features that these waveforms represent. Depending on the audio stream, some understanding of music structure might come in handy.

Essentia is a freely-available library for handling audio information for music analysis, with bindings for C++ and Python. A Python tutorial on Essentia covers some of the basics.

A survey paper "An Evaluation of Audio Feature Extraction toolboxes", by Moffat D., Ronan D., and Reiss J.D. (2015) covers some toolkits in a variety of languages.

In order to use any toolkit for feature extraction, you need to know what features to look for, and which algorithms to select. This is a fast-moving area of research. The book: "Fundamentals of Music Processing", by Meinard Müller (2016) covers all the background on audio encoding, music representation, and analysis algorithms. The first two chapters cover the core concepts, and chapters 3 onwards dive into individual topics, and can then be read in parallel. This allows software engineers to understand the basics, and then immediately focus on the task at hand. This book is dense and requires a firm understanding of Linear Algebra. Once you know the terminology, you can read the relevant papers on feature types and either use a readily available library, or write the extraction code yourself.

Finally, pre-trained models in Essentia allow you to use existing models for classification tasks. An online demo exists to test out the functionality in a browser.

Tailscale: the best, secure, private VPN you need

 tl:dr; Tailscale easily sets up remote machines as though they were on your local network (VPN)

Tell me if this situation sounds familiar: you have a variety of machines, at home and at remote locations. All combinations: behind a proxy/NAT,  connected directly, a mix of Linux and Mac/Windows systems, a mixture of physical hardware and cloud instances. You want these machines to behave like they're on a local network and to use them without jumping through hoops. You could access these machines with proxies like ngrok or other tunneling software like ssh, but that's complicated.

You could set up a VPN, but that is time consuming and difficult to manage. Setting up a VPN requires skill and is difficult to get right across host platforms or architectures.

Tailscale is an excellent, simple-to-setup and secure VPN, with clients available for all major systems and architectures. You use an existing authentication (like your Gmail address), download the client software for your platform, and authenticate by navigating to a web address. Setting it up is refreshingly simple. It even sets up a dns, so you can refer to your machines by their hostname: p31 can be used instead of the full VPN IP address like 100.107.137.29.

I've used a variety of VPN systems in the past; I've also set up my own tunnels using different providers; I've rolled my own tunnels from first principles. Compared to existing systems, Tailscale is easier to setup, efficient, and has great network performance. Network latencies are lower than traditional hub-and-spoke systems, which relay through a central server. If the central server is located far from both VPN'd machine, network performance is usually poor.

Right now I'm pinging machines that are behind a NAT, accessing web pages on a different physical network, all by referring to simple hostnames. There's arm32, arm64, x64, different operating systems, physical and cloud instances that all appear as a local Class-A network. This is like magic!

Tailscale is also great for working on projects on your cloud or local instance without exposing it to the wild Internet traffic.

Image courtesy: Wikipedia.

Generating art with Tensorflow

tl:dr; Thought-provoking, computer-generated art

That is computer generated: the output of transfer learning using Tensorflow.

I was playing with some transfer learning, training a convolutional model on famous works of art, and applying the style to my own photographs when I realized the process could be done in reverse. So instead of taking the style of a famous artist, and applying it to my photographs, I've taken the style of my photographs and applied it to famous art. Here are the two pictures, the content is a famous Kandinsky image, and the style applied was that of a tiger in San Francisco.

Here's another. A famous Monet that has been modified with the style of a giraffe from the same zoo. The original image is pretty serene, and I love its transformation into a drug-fueled nightmare.

And finally, my favorite, two mundane pictures that together make evocative art.

The content image is from the page linked below, and the lion is my own photograph. Either picture is average by itself.

A computer and a human are more superior at chess than a computer alone, or a human alone. The same can be said about art. A human combined with a well-written software can make something amazing. 

This is based on the Tensorflow style transfer tutorial.

Best Practices with Tensorflow: Test All Your Inputs Early

Tensorflow model training can take a long time. Depending on the machine, even the first epoch can take many hours to run. In this case, the remaining arguments like validation_data are only evaluated much later. Incorrectly formed input will be detected many hours later, deep in the stack, with incomprehensible errors. By this time, you might have forgotten what the shapes of the input or their expected data type is. You want to detect failures early, when you run model.fit() rather than a day later.

One easy way to avoid this is to wrap all model.fit() calls in a single method, which tests itself out with a smaller input. Here's the idea. If you have training data (X_train and y_train), and validation data (X_valid and y_valid), you take the first 10 observations, and call the training method itself.

This run will be fast, because you are training a model with a tiny set of observations, and the validation is done with a tiny set of values too.

It also ensures that any changes to the model.fit() line are correctly reflected both in the test_shapes run and in the real run. Any training done on the model is minor: a real run with the full data and full set of epochs will change the model weights, so this is harmless. In case you are using a pre-trained model, you can clone the model before trying the test run.

If you are using data generators, then the same mechanism can be used. Create a new tfdata object with a few observations both on the training data, and another with the validation data. Pass that to your model training function.

def fit_e9_model(model, X_train, y_train,
                 X_valid, y_valid, epochs,
                 batch_size=32, verbose=0,
                 test_shapes=True):

    # This fails after a day, when the validation data is incorrectly shaped.                              
    # This is a terrible idea. Failures should be early.                                                   

    # The best way to guard against it is to run a small fit run with                                      
    # a tiny data size, and a tiny validation data size to ensure that                                     
    # the data is correctly shaped.                                                                        

    if (test_shapes):
        print ("Testing with the first 10 elements of the input")
        X_small = X_train[:10,:,:,:]
        y_small = y_train[:10]
        X_valid_small = X_valid[:10,:,:,:]
        y_valid_small = y_valid[:10]
        # Call ourselves again with a smaller input. This confirms                                         
        # that the two methods are calling the same .fit() method, and                                     
        # that the input is correctly shaped in the original method too.                                   
        fit_e9_model(model, X_small, y_small,
                     X_valid_small, y_valid_small,
                     epochs=epochs, verbose=verbose,
                     batch_size=5,
                     test_shapes=False)

    # If that worked, then do a full run.                                                                  
    history_conv = model.fit(x=X_train, y=y_train, batch_size=batch_size,
                             validation_data=(X_valid, y_valid),
                             epochs=epochs, verbose=verbose)
    return history_conv



history = fit_e9_model(model, X_train, y_train, X_valid, y_valid, epochs=10)

User-facing tools should have user-understandable errors

Here's some simple Tensorflow code to create a CNN, and train it. 

import tensorflow as tf
import tensorflow.keras as keras

def create_model(optimizer="sgd"):
    deep_model = keras.models.Sequential([
        keras.layers.Conv2D(64, 7, activation="relu", padding="same", 
                            input_shape=[1, 28, 28], name="input"),
        keras.layers.MaxPooling2D(1,name="firstPool"),
        keras.layers.Conv2D(128, 3, activation="relu", padding="same", 
                            name="first_conv_1"),
        keras.layers.Conv2D(128, 3, activation="relu", padding="same", 
                            name="first_conv_2"),

        keras.layers.MaxPooling2D(1, name="secondPool"),
        keras.layers.Conv2D(256, 3, activation="relu", padding="same", 
                            name="second_conv_1"),
        keras.layers.Conv2D(256, 3, activation="relu", padding="same", 
                            name="second_conv_2"),

        keras.layers.MaxPooling2D(1, name="thirdPool"),

        keras.layers.Flatten(name="flatten"),
        keras.layers.Dense(128, activation="relu", name="pre-bottneck"),

        keras.layers.Dropout(0.5, name="bottleneckDropout"),
        keras.layers.Dense(64, activation="relu", name="bottleneck"),

        keras.layers.Dropout(0.5, name="outputDropout"),
        keras.layers.Dense(10, activation="softmax", name="output"),
    ])
    
    deep_model.compile(loss="sparse_categorical_crossentropy",
                      optimizer=optimizer,
                      metrics=["accuracy"])

    return deep_model

def fit_model(model, X_train, y_train, X_valid, y_valid, epochs):
    history_conv = model.fit(X_train, y_train, validation_data=[X_valid, y_valid],
                             epochs=epochs, verbose=0)
    return history_conv

def plot_history(history, name):
    c10.plot_training(history, name, show=True)
    
model = create_model()
history = fit_model(model, X_train, y_train, X_valid, y_valid, epochs=10)
plot_history(history, "naive_deep_mnist")

When you run it, it causes this enormous error: 

  ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-26-95a7830ebd3c> in <module>
     41 
     42 model = create_model()
---> 43 history = fit_model(model, X_train, y_train, X_valid, y_valid, epochs=10)
     44 plot_history(history, "naive_deep_mnist")

<ipython-input-26-95a7830ebd3c> in fit_model(model, X_train, y_train, X_valid, y_valid, epochs)
     34 
     35 def fit_model(model, X_train, y_train, X_valid, y_valid, epochs):
---> 36     history_conv = model.fit(X_train, y_train, validation_data=[X_valid, y_valid], epochs=epochs, verbose=0)
     37     return history_conv
     38 

/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109 
    110     # Running inside `run_distribute_coordinator` already.

/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1096                 batch_size=batch_size):
   1097               callbacks.on_train_batch_begin(step)
-> 1098               tmp_logs = train_function(iterator)
   1099               if data_handler.should_sync:
   1100                 context.async_wait()

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781 
    782       new_tracing_count = self._get_tracing_count()

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    694     self._graph_deleter = FunctionDeleter(self._lifted_initializer_graph)
    695     self._concrete_stateful_fn = (
--> 696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
    697             *args, **kwds))
    698 

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857 

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3063     arg_names = base_arg_names + missing_arg_names
   3064     graph_function = ConcreteFunction(
-> 3065         func_graph_module.func_graph_from_py_func(
   3066             self._name,
   3067             self._python_function,

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602 

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:747 train_step
        y_pred = self(x, training=True)
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:975 __call__
        input_spec.assert_input_compatibility(self.input_spec, inputs,
    /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/input_spec.py:191 assert_input_compatibility
        raise ValueError('Input ' + str(input_index) + ' of layer ' +

    ValueError: Input 0 of layer sequential_17 is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [None, 28, 28]

Compiling Tensorflow without AVX support, a Googler's perspective

tl:dr; Tensorflow compilation teaches you about the complexity of present-day software design.

Compiling Tensorflow is a curious experience. If I put on my external-user hat, the process is baffling.  Many Tensorflow choices are motived by development practices inside Google, rather than common open source development idioms. And so, as a Google engineer, I can explain what is going on and the motivations for the choices.

Why compile Tensorflow?

I want to run Tensorflow on an old Intel CPU that doesn't have AVX-instruction support. These are special "vector" instructions that speed the computation on large data streams, and are available on relatively new Intel and AMD processors. The solution is to compile Tensorflow from source as Tensorflow prepackaged binaries (after version 1.6) are compiled expecting AVX support on the processor. 

No problem: I'm an engineer and have done my share of large system compilations. I can do this.

Dependencies

Tensorflow compilation has only been tested on gcc 7.3.0, which was released in November, 2019. The latest version of gcc shipping with Ubuntu 20.04 is 9.3.0. If a user is compiling software from source, they are probably going to use a recent version of the compiler toolchain. I doubt most users will install an old version of gcc on their machine (or in a Docker image) just to compile Tensorflow. I didn't either, and went with gcc-9.3 with fingers crossed.

Perhaps it is the complexity of software development today. With the pace of development and releases: you cannot possibly support all versions of gcc, all versions of Ubuntu, Debian, Mac OS, Windows, all combinations of compute architecture: x86, x86_64, x86 with AVX, arm64, GPU with cuda. Add to this all the complexity of different target platforms: python, C++, ...

Unlike a few years ago, compilers like gcc and llvm are themselves being updated frequently. This is great, as bugs can be fixed, but it leads to a large burden on supporting different toolchains.

Lessons

Tensorflow downloads its own version of llvm. Instead of relying on the system version of llvm, which might have its own quirks, it just gets everything.

That's not all: Tensorflow downloads all its dependencies: boringssl, eigen3, aws libraries, protobuf libraries, llvm-project from github or their respective repositories. I suspect most of these go into //third-party. 

It is an interesting choice to download most of these rather than expecting them installed locally. On one level, it reduces the complexity in figuring why Tensorflow builds on Ubuntu versus Fedora, or on FreeBSD. But managing these packages also adds complexity. How do you know which version of protobuf or llvm to check-out, and what if those dependencies are no longer available.

The obvious complexity is that you have to compile your dependencies too. A user might already have a pre-compiled llvm and the protobuf library.

If anything, the Tensorflow style looks similar to Android Open Source (AOSP) or FreeBSD's Ports collection.  In both of these, the downloaded repository creates a parallel universe of source and objects. You compile everything from scratch. The notable difference between FreeBSD is that the output of FreeBSD's Ports are installed in /usr/local/ and are then available to be used on the system. After you compile protobufs for Tensorflow, you still don't have the protobuf library available to the wider system.

The reason for this is probably because Google engineers compile the whole world. Google production binaries shouldn't rely on the specific version of eigen3 you happen to have on your development machine. Instead, you get a specific version of eigen3 from the repository (the "mono" repo), and use that. Ditto for llvm. Most of this open-source dependency code does not diverge too far from upstream, as bugfixes are reported back to the authors. This provides some sanity of dependencies. I suspect the version of llvm or eigen3 chosen are the same versions that were in the mono repo at the time Tensorflow 2.4 was released. 

This differs from other large open source projects. You are expected to have all the dependencies locally if you are to compile Emacs. It needs libjpeg, so install that through apt or yum. Then you realize you need x11 libraries. Ok, go get those separately. Cumbersome, and it increases the risk of a failure at runtime as your version of libjpeg might not be what the authors tested against.

Bazel does help when compiling everything. On a subsequent run, it won't need to recompile boring_ssl. Inside Google, the build system reuses objects from prior runs of other engineers which vastly speeds up individual's compiliation. An open source developer does not benefit from this on their first compile of Tensorflow. They're starting out cold. Their subsequent compile runs are sped up, of course, but how often do you compile Tensorflow again? You generate the Python Wheel, rm -Rf the checked out repo, and carry on with your Python data analysis.

Another quirk: at the end, the bazel server is still running on that machine, and it shut down after many hours of disuse. This might be fine for Google engineers who will be compiling other software, soon. For them, the cost of keeping bazel up and running is small compared to the benefit from the pre-warmed caches and memory. I suspect independent open source developers are baffled why bazel is holding on to 400+Mb of RAM, hours after the compilation is done.

The choice of bazel itself is interesting. Most open source software uses the 'make' tool, despite its numerous flaws. Bazel is an open source implementation of an internal Google build tool, so Tensorflow uses that. Even Android AOSP uses make, since bazel wasn't available in open-source back in 2008 when Android AOSP was released.

Other systems

Let's see how other systems manage this sort of complexity.

PyTorch, by comparison offers a rich selection of choices. You select the build, the OS to run on, which package manager you want to use (Conda/Pip/...) whether you have CUDA installed, and if so, which version. It finally tells you exactly how to get PyTorch installed on your system.

 

This raises a question: why can't Tensorflow be available as a pre-compiled binary in a lot more configurations? The Intel Math Kernel Library (MKL), for example, is available in 50 variants packaged natively for Ubuntu: libmkl-avx, libmkl-avx2, libmkl-avx512, libmkl-avx512-mic, libmkl-vml-avx, ... These are all variants for specific Intel CPUs, to extract the maximum possible performance from each system. Tensorflow is similar: it is built to efficiently process compute-intensive workloads. Why isn't Tensorflow available in 50 different variants, targetting avx, no-avx, avx2, avx512, ...?

Here, I am guessing the choices are due to the Google-engineer/Open Source divide. At Google, most engineers run a specific kind of machine, and so the pre-compiled binaries target these workstations and similar CPUs on cloud compute farms. Most internal (and Google cloud) users don't deviate from these computing setups. So Tensorflow on a Core2Duo from 2006, or ARM32, or ARM64 isn't a high priority. This is a real lost opportunity, because compiling multiple targets can be automated. The real cost here is of maintenance. If you do provide Tensorflow on a Core2Duo or arm32, you are implicitly providing support for it.

The open source answer here would be to appoint a maintainer for that architecture. The Macintosh PowerPC port of Linux is still maintained by Benjamin Herrenschmidt, among others. He cares about that architecture, so he helps keep it up and running. The community will probably maintain a no-avx binary if you empower them. 

 The Linux kernel is also an incredibly complex system. You are building an operating system kernel, which by definition is hardware architecture and device specific. Even in 2020, you can build the Linux kernel for machines as varied as PowerPC, ARM, MIPS, Intel x64, and Intel 386. Of course, you can choose between AVX support and not.  The Linux kernel depends on very few external libraries, and is almost entirely self-contained. It compiles with make, and generates targets that work on many more architectures than Tensorflow. It has a huge configuration system, with many many choices. Most of the complexity is the skill and expertise in understanding the options and selecting them. You can always take an existing kernel configuration from a running system, and then run 'make menuconfig' to modify the specific options you want to change.

The comparison might not be entirely fair, though. The Linux kernel has been in active development for many decades. It was always developed in a decentralized way, and therefore has perfected open source development development and release. The open source process has also been shaped by the quirks of the Linux kernel to the point where it is difficult to tell whether Linux influences open source or open source influences Linux.

 

Outcome

It took a few hours of computation on the old machine, all four CPUs were busy for a whole day. But at the end, I have Tensorflow for Ubuntu 20.04 for x86_64 without AVX support. I tried this out on a Celeron and a Core2 machine, and it works great. Tensorflow is perfect for the old machines where you can run model training for a few hours, turn the screen off, and leave it alone.

Since I have the Wheel compiled for Python3, here it is if anyone needs Tensorflow 2.4 without AVX support for Ubuntu 20.04 and Python3. If you need another version, find my email address and mail me.

Just for fun, I'd love to compile PyTorch from source as well. It seems to follow the open source paradigm closely: you install specific dependencies using yum/apt, and it uses those directly.

Conclusion

Tensorflow compilation was an interesting process to see from the outside. The compilation process is far more complicated due to the wealth of dependencies. While most users of Tensorflow are aware of the complexity of Tensorflow, a lot more complexity is visible when compiling the system. The choices here are motivated by some development practices at Google, and also make an interesting case study for large system design.

 

Disclaimer: I'm a Google employee, but these are my own opinions from the public Tensorflow project. I did not examine any Google confidential systems to arrive at these observations.