tag:blogger.com,1999:blog-191639752024-03-12T16:12:48.332-07:00Vikram AggarwalTechnology and parentingNehahttp://www.blogger.com/profile/07192174935373922490noreply@blogger.comBlogger123125tag:blogger.com,1999:blog-19163975.post-55601886152887308522022-08-12T23:07:00.001-07:002022-08-12T23:07:00.150-07:00Python, pip, venv, package managers<p> Upgrading an old computer from one Ubuntu LTS version to the next: 20.04 to 22.04,...</p><p>I came across an annoying error when running do-release-upgrade:</p>
<pre class="bbcode_code" style="background: none repeat-x rgb(239, 239, 239); border: 1px inset; direction: ltr; font-family: "Ubuntu Mono", monospace; font-size: 14px; height: 42px; line-height: 14px; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 6px;">AttributeError: 'UbuntuDistroInfo' object has no attribute 'get_all' </pre><p>This shows up in other forms, folks online complain about errors of the kind:</p><pre class="bbcode_code" style="background: none repeat-x rgb(239, 239, 239); border: 1px inset; direction: ltr; font-family: "Ubuntu Mono", monospace; font-size: 14px; height: 42px; line-height: 14px; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 6px;">AttributeError: 'DistUpgradeController' object has no attribute 'tasks'</pre>
<p><br /></p><p>Now the usual reaction is to blame Ubuntu's release. Free Software, and it's lack of reliability, ...</p><p>Except that I've upgraded several computers from 20.04 to 22.04, so I know that the process works well. Quite unlike my usual nature, I decided to investigate.</p><p>The fix: <span style="font-family: Roboto Mono;">pip uninstall distro_info</span></p><p><br /></p><p>I'll spare you the full investigation, because life is short. The end-result is that this is caused by Python's incredibly brittle package management. On an Ubuntu system, you have both packages installed by the package manager (dpkg/apt) and by pip. And pip can install software for the whole system, when run by root. In this case, the package 'distro-info' was installed by root using pip, and that overrides whatever distro_info object exists in Python. Since the Ubuntu installer expects certain behavior from the class UbuntuDistroInfo, the package install fails early.</p><p>The fix, again: <span style="font-family: Roboto Mono;">pip uninstall distro_info</span></p><p><br /></p><p>Having solved it, let's reflect on the incredibly broken nature of Python packaging. Why was a user unable to upgrade their Ubuntu system, and why was pip at the bottom of the mess?</p><p>First, the package namespace should be unique. The package distro_info should either not conflict, or Ubuntu's package manager should create a unique name that doesn't conflict. It should be called 'ubuntu_distro_info'. If these are maintained by the same team, then new versions should be very careful removing methods from previous versions. Software versioning is complicated, but there are many lessons learned over the years.</p><p>Second, the parallel universes that exist between pip and dpkg/apt are a mess. root should not be allowed to pip install packages. </p><p>Third, venv is a crutch. Package maintenance is difficult, and dependencies are tied specific version of packages, and so you need virtual environments to create each parallel universe. This gets the job done, but pushes a lot of maintenance headache to the end-user of the packages. Now, in addition to the dpkg/pip mess, you individual pip messes in subdirectories scattered all over your filesystem.</p><p>Fourth, python versus python3. In Debian-based systems, python and python3 have different namespaces for packages, and this further confounds the issue. So if you install python-libraryname, you might also have to install python3-libraryname.</p><p>Finally, python errors are broken. This is a pet peeve of mine. Python's errors are the bottom of the stack, and the full stack trace. But these are completely unhelpful in explaining what might be involved. If package versions are in flux all the time, python software should start off verifying versions and sanity. Imagine if this failure had happened half-way in the install process, leading to a broken machine. Fragile systems need defensive programming.</p><p><br /></p><p>All this leads to an incredibly brittle software setup: where packages are frozen in time (in venv directories), but also no way of querying what versions of software exist on the system. Once an environment is set-up, there is little confidence that it will continue working.</p><p><br /></p><div><br /></div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-62363210449919591052022-02-27T10:45:00.000-08:002022-02-27T10:45:00.180-08:00Christone "Kingfish" Ingram<p>I recently heard an artist called <a href="https://www.christonekingfishingram.com//kingfish-home/">Christone Ingram</a>, who goes by the stage name "Kingfish". He plays the blues on guitar. It blew me away: the guitaring, the singing. After many years, I have come across a contemporary artist who is awe-inspiring.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg30qskm7OIiWVrNmECOdpUG-ibX1-VQSf-AuNYspUWqH-HnfRlwdqCCLzM_HklxoQiO6kN7GcSN99aw-gOF8WwNhDGzI4ItaXhUwjQ2-liOsCHtjrInWesFHKxxLWGLWNCWxxlDz1k1hEq5iWxqy82WWYK1a3b9f1iYoMeAQXtxINzyoV-72w=s1500" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1500" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/a/AVvXsEg30qskm7OIiWVrNmECOdpUG-ibX1-VQSf-AuNYspUWqH-HnfRlwdqCCLzM_HklxoQiO6kN7GcSN99aw-gOF8WwNhDGzI4ItaXhUwjQ2-liOsCHtjrInWesFHKxxLWGLWNCWxxlDz1k1hEq5iWxqy82WWYK1a3b9f1iYoMeAQXtxINzyoV-72w=s320" width="256" /></a></div>It all began when I chanced upon an album called "662" by someone holding a Stratocaster. I wasn't expecting much. What could have been just another blues musician turned out to be a genius of our time, for each song was delightful.<p></p><p><br /></p><p>If you like blues, or classic rock, give it a listen.<br /> This <a href="https://www.youtube.com/watch?v=GaQ0IDLw6qs">Youtube</a> video, for example, gives you some idea of his level of skill. You can also preview his music on his website: the album I heard is called "<a href="https://www.christonekingfishingram.com/albums/662/">662</a>", and that's his second album. His first album is called "<a href="https://www.christonekingfishingram.com/albums/kingfish/">Kingfish</a>".</p><p><br /></p><p>I find 'influencers' hollow. Many current influencers are popular solely because they're popular. Some of them might be good looking, but they have no skill beyond that accident of birth. For every influencer that takes our attention, there are real artists, folks with skill, folks working hard. We ought to devote our time on folks who have skill, who advance art.</p><p><br /></p><p><br /></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-11390089208914663952022-02-20T09:40:00.008-08:002022-02-20T09:56:24.813-08:00Book Review: Systems Performance 2nd ed, by Brendan Gregg<p></p><div class="separator" style="clear: both; text-align: left;">Summary: <a href="https://www.brendangregg.com/systems-performance-2nd-edition-book.html">"Systems Performance", by Brenden Gregg</a> covers end-to-end performance for Linux-based systems. If you run Linux software, you will learn a lot from this book.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">From its rough and loose beginnings, Linux has become a force in the commercial world. Linux is <b>the</b> most pervasive, most readily available system that you can experiment with. Starting from the $10 Raspberry Pi to the multi-million dollar <a href="https://www.top500.org/statistics/list/">Top 500 supercomputers</a>, Linux runs on everything: laptops, desktops, phones, cloud instances.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Despite widespread adoption, there is little documentation to get a thorough understanding of system performance. I routinely see veteran engineers struggle with performance bottlenecks. Folks revert to running 'top', and trying to infer everything from its limited output. The easy answer is to over-provision hardware or cloud instances to cover up sloppy performance. A better answer is to get a solid understanding of end-to-end performance; to find and eliminate bottlenecks.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><a href="https://www.brendangregg.com/systems-performance-2nd-edition-book.html">"Systems Performance", by Brenden Gregg</a> covers the entire area of end-to-end performance of all components: CPU, RAM, network, block devices. The second edition of this book is focussed on Linux, and covers many tools and utilities that are critical to understanding every level of the stack. If you have written any software on Linux, or intend to write any software on Linux, you need a copy.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">First, the good:</div><div class="separator" style="clear: both; text-align: left;"><ol style="text-align: left;"><li>There is an overview at the beginning, and then a deep-dive on specific system resources (CPU, RAM, block devices, network). You read the overview to understand the system at the top-level, and based on your system and bottlenecks, you can read the in-depth sections.</li><li>There's coverage of pre-BPF tools (perf, sar, ftrace) in addition to the newer BPF-era tools like bcc and bpftrace. 'perf' probes are easier to use, and available on more architectures, for instance. BPF-based tools can be a slog to install, or might not have good support on fringe architectures and older kernels. No single tool can cover every need, and good engineers need to understand the full tool landscape. This book provides a wide overview of most tools.</li><li>The book provides a methodical look of the full system, with tools targeting individual levels of the system components (<a href="https://github.com/iovisor/bpftrace/blob/master/images/bpftrace_probes_2018.png">example diagram</a>). This process helps isolate the problem to the correct component.</li></ol><div><br /></div><div>The not-so-good:</div><div><ol style="text-align: left;"><li>The book is repetitive. Since it expects some readers will start reading a deep-dive, it repeats the USE methodology at the start of most chapters. Folks reading it cover-to-cover will find themselves wondering if they have seen the material already.</li><li>Print quality is worse than the previous edition. The fonts are thin and dim, the pages bleed through, and the graphs need more contrast. The first edition was a high quality printed book, and the second edition is worse in this department. Since this is a reference book, a physical copy is better than an ebook. You will mark pages, put sticky notes, and highlight tools that are more pertinent to your work. Luckily, the binding holds up to heavy use.<br />I really wish the third edition comes with better print quality, and is hard-bound.</li></ol><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgBmPJizuTQwVW3uVOc40DflWjBmlOljLkil1b9qexU7sTAOhQRSWCxclMxWasFRCrLYSsl2rG1-ynfbW1LTOC7Ugc0mn-7BYO2dIrGLjvSpxbp-Z3Ne4MalqeWHZACjN28KKFS2wO7UWfSiVRK4odSCchd2UbFzjN9a9Yjrbru4RiJBpuE-jE" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img alt="" data-original-height="1099" data-original-width="842" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEgBmPJizuTQwVW3uVOc40DflWjBmlOljLkil1b9qexU7sTAOhQRSWCxclMxWasFRCrLYSsl2rG1-ynfbW1LTOC7Ugc0mn-7BYO2dIrGLjvSpxbp-Z3Ne4MalqeWHZACjN28KKFS2wO7UWfSiVRK4odSCchd2UbFzjN9a9Yjrbru4RiJBpuE-jE=w306-h400" width="306" /></a></div><br /></div></div><div>Every software engineer should be familiar with end-to-end performance: how to think about it, how to locate trouble spots, and how to improve the system. This book will give you a firm foundation of performance that should help on most desktop, server, and cloud systems. </div><div><br /></div><div>You will probably not get this understanding from a scattershot reading of online documentation and Stack Overflow articles. Online articles are limited in scope and accuracy, and don't provide a comprehensive view of how to think about performance. This topic deserves a book-length treatment.</div></div><p></p><p></p><div style="text-align: left;"><br /></div><div style="text-align: left;">Image Courtesy: <a href="https://www.brendangregg.com/systems-performance-2nd-edition-book.html">Brenden Gregg</a></div><p></p><div class="separator" style="clear: both; text-align: left;"><br /></div>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-20774505810401412392022-01-03T09:37:00.008-08:002022-01-03T09:42:37.401-08:00Tensorflow 2.8 and Jax 0.1.76 for NO AVX cpus<p> In what has become a tradition, I compiled Tensorflow for my no-avx CPU. This time, the installation was more complicated because of a dependency on jaxlib. I had installed jax either through pip3 or through Debian's repositories (apt-get tool). The jaxlib was compiled with AVX support and would not work on my computer.</p><p>So I spent some time getting Jax sources and compiling those without AVX support.</p><p>Here are the two files for older Intel CPUs:</p><p><a href="https://www.dropbox.com/s/ykm12r1lngjz48o/jaxlib-0.1.76-cp38-none-manylinux2010_x86_64.whl?dl=1">jaxlib-0.1.76-cp38-none-manylinux2010_x86_64.whl</a></p><p><a href="https://www.dropbox.com/s/hq5dcrmw1geaq08/tensorflow-2.8.0rc0-cp38-cp38-linux_x86_64.whl?dl=1">tensorflow-2.8.0rc0-cp38-cp38-linux_x86_64.whl</a></p><p><br /></p><p>Unless you have compiled your own jaxlib, you will need to download both. The jaxlib should be useful with the native 'jax' install from pip3 since the jax library only contains Python code. As I understand, the jax library does not contain native code.</p><p>You could also use the jaxlib in isolation for playing with <a href="https://github.com/google/jax">Jax</a>.</p><p>To install them, download the whl files to disk, and run </p><p><span style="font-family: Roboto Mono;">pip3 install filenameHere.whl</span></p><p><br /></p><p>These were compiled on a cpu with the following flags in the output of <span style="font-family: Roboto Mono;">/proc/cpuinfo</span>:</p>
<p><span style="font-family: Roboto Mono;"><b>
flags</b> : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities
<br /></span></p><p><br /></p><p>Both these wheels work great on my Core 2 duo and another Pentium CPU that didn't have AVX support. Compiled with Python 3.8, they should work for most Linux distributions, assuming the dependencies (numpy, absl-py, scipy, flatbuffers, tensorboard, ...) are installed. pip3 should get the dependencies you don't have. None of the dependencies contain any native code that requires AVX instructions.</p><p><br /></p><p><br /></p><p></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-42811974663326465562021-06-02T20:00:00.007-07:002022-01-03T09:47:07.610-08:00Tensorflow 2.5 without AVX<p> I was playing with Tensorflow, needed a new version, and realized that a Tensorflow release without AVX still doesn't exist. My prior post on <a href="https://www.eggwall.com/2020/09/compiling-tensorflow-without-avx.html">Tensorflow without AVX</a> had been beneficial to people, so here's Tensorflow 2.5 for Linux.</p><p><a href="https://www.dropbox.com/s/7dp7tmzvlo475rp/tensorflow-2.5.0-cp38-cp38-linux_x86_64.whl?dl=1">Tensorflow 2.5 for Linux without AVX support</a> (155 Megabytes)</p><p>and the prior link</p><p><a href="https://www.dropbox.com/s/rnmn205f1skj139/tensorflow-2.3.0-cp38-cp38-linux_x86_64.whl?dl=1">Tensorflow 2.3 for Linux without AVX support</a> (126 Megabytes)</p><p><br /></p><p>Download the file, and then run</p><p><span style="font-family: Roboto Mono;">$ pip3 install -U filename.whl</span></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-62000769518550061712021-03-12T18:54:00.010-08:002021-03-12T20:36:29.842-08:00Cloud of DOS machines<p>tl:dr; <span style="font-family: Roboto Mono;">ssh guestdos@dos.eggwall.com</span></p><p><br /></p><p>Retro computing is in. New computers and new systems are always fun. But to really appreciate the arc of history, use something old. Something ancient. Something you never really learned.</p><p>To help you get your retro computing fix, here's a bank of DOS machines. Rather than MSDOS (copyrighted, etc), I've got FreeDOS, an open-source implementation of DOS. There is the editor 'edit' and 'edlin'. 'foxcalc' is a sweet calculator. 'help' will tell you what to do. For the Assembly geeks, there is always 'debug'.</p><p>This is running on my custom cloud of DOS machines. Connect to your instance today. You can use either ssh or telnet:</p><p><span style="font-family: Roboto Mono;">$ ssh guestdos@dos.eggwall.com</span></p><p><br /></p><p>Telnet works too:</p><p><span style="font-family: "Roboto Mono";">$ telnet dos.eggwall.com</span></p><p>Username: 'guestdos', no password.</p><p><br /></p><p>When you are done, type 'halt' and travel back to the present time.</p><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX1KzFPE5ghrVaZgMaadkHrYNr4g7GpTZNV9o8HVrSvpZsY-fValznCe3EPK1TcS4TtCBw3GiVCCIPzkRE2fSSoDz_toJ-tL3QByoPJLlWlgoFNON0Y7NERiqB52pXi0iqLlSHJA/s744/Screenshot+from+2021-03-12+18-00-48.png" style="margin-left: 1em; margin-right: 1em;"><img alt="DOS" border="0" data-original-height="511" data-original-width="744" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX1KzFPE5ghrVaZgMaadkHrYNr4g7GpTZNV9o8HVrSvpZsY-fValznCe3EPK1TcS4TtCBw3GiVCCIPzkRE2fSSoDz_toJ-tL3QByoPJLlWlgoFNON0Y7NERiqB52pXi0iqLlSHJA/w640-h440/Screenshot+from+2021-03-12+18-00-48.png" title="DOS" width="640" /></a></div><p><br /></p><p>Look around the file system. There's compilers for C, Pascal, an IDE, emacs, vi, BASIC, and some games. Using a system this old gives you an appreciation for systems today. You miss the many conveniences you take for granted. You are also closer to the machine, you can modify arbitrary memory addresses. Nothing to get in your way.</p><p>Powered by the <a href="https://www.freedos.org/download/">FreeDOS</a> project, and magic.</p><p><br /></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-21909811557210308022021-03-09T20:16:00.006-08:002021-03-09T21:08:09.082-08:00Book Review: UNIX and Linux System Administration Handbook, Evi Nemeth et al<p><b> tl:dr</b>; If you are a Linux/BSD user, this book will help you understand your system.</p><p><br /></p><p>There are Stack Overflow answers, random blogs and tech articles. You can find <a href="https://man7.org/linux/man-pages/man1/find.1.html">man pages</a>. What they lack is a comprehensive view of the full system, with historical background, a theme tying the topics together. They lack clarity, depth or accuracy. Nearly always, they lack humor.</p><p><br /></p><p><a href="https://www.pearson.com/us/higher-education/program/Nemeth-UNIX-and-Linux-System-Administration-Handbook-5th-Edition/PGM143215.html">UNIX and Linux System Administration Handbook</a> is a very broad overview of all aspects of Unix system maintenance and upkeep: going from installation, package installation, and routine chores. It has in-depth coverage of critical sub-systems like networking and disks. These are covered over many chapters, devoted to hardware, services, best practices. There is deep coverage of user management, security issues for sites, authentication, logging. It has the obligatory chapter on shell programming, with bash & Python. There's information on config management tools like Ansible. It has much more than a system administrator would need through their first year. It is a large book and you will find yourself revisiting specific sections. It helps to keep an electronic copy for quick searches. The index and the table of contents are ideal for an admin with a paper copy.</p><p><br /></p><p>The book gives you a superficial overview of virtualization, cloud systems, containers (Docker), and container orchestration tools like Kubernetes, Docker Swarm, Apache Mesos or Marathon. These sections are light: it gives you a background, why these systems exist and how to think about these systems. You can start a journey there if you want. These chapters were added later, and the coverage is satisfactory. Users need to have a rough idea of these, but each of those topics deserve a book-level treatment. The authors have kindly provided references to chase for the interested readers.</p><p><br />I've been a long-time Linux user, and I found the book's treatment of systemd level-headed and valuable. It provided a great preface with motivations for systemd, how it has evolved, and how best to use systemd to keep a system humming along and keep the system manageable.</p><p><br /></p><p>The book was funny in all the right parts, without overdoing the humor or trying too hard.</p><p><br /></p><p>It was published in 2017, but most of the sections are quite current. The book covers major Linux distributions (CentOS/RedHat and Debian/Ubuntu) and FreeBSD. The FreeBSD sections were great, especially alongside Linux so I had a Rosetta stone. The book's primary author, <a href="https://en.wikipedia.org/wiki/Evi_Nemeth">Evi Nemeth</a>, is one of my favorite technical authors. Sadly, she passed away before this edition of the book was published, and I commend the current authors for carrying on her tradition, and their work on this edition.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvnGXr_wRAfwryRMC8-ZiWWWUui6oG_gyFaM9_B3vu9tCQw6Dbja4ld_oRIsn8IL4B9oqpUNNdEymjNS3mBAIp6wb6bier_s79rL7lZ2LMn7SMgt-qmdqEB1NbvizPZpE4syQSlA/" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img alt="" data-original-height="325" data-original-width="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvnGXr_wRAfwryRMC8-ZiWWWUui6oG_gyFaM9_B3vu9tCQw6Dbja4ld_oRIsn8IL4B9oqpUNNdEymjNS3mBAIp6wb6bier_s79rL7lZ2LMn7SMgt-qmdqEB1NbvizPZpE4syQSlA/s16000/250w.jpeg" /></a></div><br /><p></p><p>In 2021, the diversity in Unix systems has shrunk incredibly. Solaris is history; FreeBSD is relegated to core adherents and the occasional technical Mac OS user. Cloud systems are gathering behind Linux instances. The ease of creating a Linux instance means that more users will find themselves responsible for a Linux machine. Unix used to mean SunOS: something formal, expensive. Something meriting a real education. Now, Linux powers everything from your phone to your Top500 supercomputer. A cloud instance can be yours for dollars a month. The power is still there, if you can invest time in learning it.</p><p><br /></p><p>Veteran system administrators will enjoy filling gaps in their knowledge. The casual user will gain a deeper understanding of their system with this light, engaging, fun read.</p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-27536077487782787042021-02-14T11:16:00.007-08:002021-02-17T18:37:53.807-08:00Unicode URLs<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvslnj-RqnWLmXO1hP8kBI-pyCMhwCoJlx15gCx_b7Idp_ZuVtxA4y5-QEIrdlHVc-3p0XnnZp9IDIqPUDn_BecAj-YZKcN5nU1iWSFInQy5pxfcnUpfk5QOzpusVOBFFGzkCI8w/s332/Screenshot+from+2021-02-14+10-44-41.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="71" data-original-width="332" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvslnj-RqnWLmXO1hP8kBI-pyCMhwCoJlx15gCx_b7Idp_ZuVtxA4y5-QEIrdlHVc-3p0XnnZp9IDIqPUDn_BecAj-YZKcN5nU1iWSFInQy5pxfcnUpfk5QOzpusVOBFFGzkCI8w/s16000/Screenshot+from+2021-02-14+10-44-41.png" /></a></div><b>tl:dr</b>; Get URLs that cannot be verbally communicated.<p></p><p><br /></p><p>URLs can only be in English characters (really, Roman characters) and a select few special characters like -. So how is<a href="https://xn--p8j9a0d9c9a.xn--q9jyb4c/"> はじめよう.みんな</a> a full domain name, even though it is written in Japanese (Hiragana script) and a dot?</p><p><br /></p><p>The answer lies in character encodings, and translation done by the browser.</p><p>There is an encoding of characters called <a href="https://en.wikipedia.org/wiki/Unicode">Unicode</a>. That is what allows English, Japanese, Hindi, and other characters to be written on a single document. URLs are Internet "addresses", the stuff that shows up in the address bar of your browser. These URLs can <i>only</i> be coded in ascii characters (a-z, a-Z, 0-9 etc). Due to their long history, URLs cannot be specified in Unicode. But creative minds have been hard at work here. There is a coding for Unicode that uses only the characters that are allowed in URLs. This encoding is called <a href="https://en.wikipedia.org/wiki/Punycode">punycode</a>. Punycode URLs start out with 'xn--', but the browser transparently renders the unreadable combination of ascii characters into their corresponding unicode characters.</p><p><br /></p><p>So in the <a href="https://xn--p8j9a0d9c9a.xn--q9jyb4c/">https://はじめよう.みん</a> example above, はじめよう is xn--p8j9a0d9c9a and is みん is xn--q9jyb4c. So the full URL is https://xn--p8j9a0d9c9a.xn--q9jyb4c/ If you can remember all those characters, you can type them out by hand. Your browser, detecting the punycode beginnings 'xn--' will decode the characters xn--p8j9a0d9c9a into はじめよう and xn--q9jyb4c into みん.</p><p>Try this translation yourself using online <a href="https://www.punycoder.com/">punycode converters</a>.</p><p><br /></p><p>This mechanism works for Top Level Domains (the .<span style="color: #800180;"><b>com</b></span> in mail.google.<span style="color: #800180;"><b>com</b></span>), domain names (the <span style="color: #38761d;"><b>google</b></span> in maill.<span style="color: #38761d;"><b>google</b></span>.com), or sub-domains (<b><span style="color: #0b5394;">mail</span></b> in <b><span style="color: #0b5394;">mail</span></b>.google.com)</p><span class="VIiyi" jsaction="mouseup:BR6jm" jsname="jqKxS" lang="hi"><span class="JLqJ4b ChMk0b" data-language-for-alternatives="hi" data-language-to-translate-into="auto" data-phrase-index="0" jsaction="agoMJf:PFBcW;usxOmf:aWLT7;jhKsnd:P7O7bd,F8DmGf;Q4AGo:Gm7gYd,qAKMYb;uFUCPb:pvnm0e,pfE8Hb,PFBcW;f56efd:dJXsye;EnoYf:KNzws,ZJsZZ,JgVSJc;zdMJQc:cCQNKb,ZJsZZ,zchEXc;Ytrrj:JJDvdc;tNR8yc:GeFvjb;oFN6Ye:hij5Wb" jscontroller="Zl5N8" jsdata="uqLsIf;_;$11" jsmodel="SsMkhd" jsname="txFAF"><span jsaction="click:qtZ4nf,GFf3ac,tMZCfe; contextmenu:Nqw7Te,QP7LD; mouseout:Nqw7Te; mouseover:qtZ4nf,c2aHje" jsname="W297wb"><div><span class="VIiyi" jsaction="mouseup:BR6jm" jsname="jqKxS" lang="hi"><span class="JLqJ4b ChMk0b" data-language-for-alternatives="hi" data-language-to-translate-into="auto" data-phrase-index="0" jsaction="agoMJf:PFBcW;usxOmf:aWLT7;jhKsnd:P7O7bd,F8DmGf;Q4AGo:Gm7gYd,qAKMYb;uFUCPb:pvnm0e,pfE8Hb,PFBcW;f56efd:dJXsye;EnoYf:KNzws,ZJsZZ,JgVSJc;zdMJQc:cCQNKb,ZJsZZ,zchEXc;Ytrrj:JJDvdc;tNR8yc:GeFvjb;oFN6Ye:hij5Wb" jscontroller="Zl5N8" jsdata="uqLsIf;_;$11" jsmodel="SsMkhd" jsname="txFAF"><span jsaction="click:qtZ4nf,GFf3ac,tMZCfe; contextmenu:Nqw7Te,QP7LD; mouseout:Nqw7Te; mouseover:qtZ4nf,c2aHje" jsname="W297wb"><br /></span></span></span></div>What do you do with it? You can create URLs that look short but can only be recreated if the person can read/write the language. Here's an example: <a href="http://xn--11b4cfs4cze.eggwall.com/">विक्रम.eggwall.com</a>. This is a URL that can be clicked on but can only be verbally communicated by Hindi speakers.</span></span></span><div><br /></div><div><br /></div><div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgI4ihIz8e6aDnggnZ93izK7unmTirgyw9naPSG7sV9vn35NI8kHOYcSHtCqc3WwQ5lVylO82dzDbEiY57Ynu8qUDc9kIwXY-PhaM57o_fiaNprRtv2CusBCodukZTdnejjKw_qlw/s523/Screenshot+from+2021-02-14+11-04-15.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="43" data-original-width="523" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgI4ihIz8e6aDnggnZ93izK7unmTirgyw9naPSG7sV9vn35NI8kHOYcSHtCqc3WwQ5lVylO82dzDbEiY57Ynu8qUDc9kIwXY-PhaM57o_fiaNprRtv2CusBCodukZTdnejjKw_qlw/s16000/Screenshot+from+2021-02-14+11-04-15.png" /></a></div><span class="VIiyi" jsaction="mouseup:BR6jm" jsname="jqKxS" lang="hi"><span class="JLqJ4b ChMk0b" data-language-for-alternatives="hi" data-language-to-translate-into="auto" data-phrase-index="0" jsaction="agoMJf:PFBcW;usxOmf:aWLT7;jhKsnd:P7O7bd,F8DmGf;Q4AGo:Gm7gYd,qAKMYb;uFUCPb:pvnm0e,pfE8Hb,PFBcW;f56efd:dJXsye;EnoYf:KNzws,ZJsZZ,JgVSJc;zdMJQc:cCQNKb,ZJsZZ,zchEXc;Ytrrj:JJDvdc;tNR8yc:GeFvjb;oFN6Ye:hij5Wb" jscontroller="Zl5N8" jsdata="uqLsIf;_;$11" jsmodel="SsMkhd" jsname="txFAF"><span jsaction="click:qtZ4nf,GFf3ac,tMZCfe; contextmenu:Nqw7Te,QP7LD; mouseout:Nqw7Te; mouseover:qtZ4nf,c2aHje" jsname="W297wb">You can mix-and-match different scripts. Here's an example with different scripts that <i>I cannot even read.</i> In the screenshot here, you have a URL. Try typing that address out.</span></span></span></div><div><span class="VIiyi" jsaction="mouseup:BR6jm" jsname="jqKxS" lang="hi"><span class="JLqJ4b ChMk0b" data-language-for-alternatives="hi" data-language-to-translate-into="auto" data-phrase-index="0" jsaction="agoMJf:PFBcW;usxOmf:aWLT7;jhKsnd:P7O7bd,F8DmGf;Q4AGo:Gm7gYd,qAKMYb;uFUCPb:pvnm0e,pfE8Hb,PFBcW;f56efd:dJXsye;EnoYf:KNzws,ZJsZZ,JgVSJc;zdMJQc:cCQNKb,ZJsZZ,zchEXc;Ytrrj:JJDvdc;tNR8yc:GeFvjb;oFN6Ye:hij5Wb" jscontroller="Zl5N8" jsdata="uqLsIf;_;$11" jsmodel="SsMkhd" jsname="txFAF"><span jsaction="click:qtZ4nf,GFf3ac,tMZCfe; contextmenu:Nqw7Te,QP7LD; mouseout:Nqw7Te; mouseover:qtZ4nf,c2aHje" jsname="W297wb"><br /></span></span></span></div><div><span class="VIiyi" jsaction="mouseup:BR6jm" jsname="jqKxS" lang="hi"><span class="JLqJ4b ChMk0b" data-language-for-alternatives="hi" data-language-to-translate-into="auto" data-phrase-index="0" jsaction="agoMJf:PFBcW;usxOmf:aWLT7;jhKsnd:P7O7bd,F8DmGf;Q4AGo:Gm7gYd,qAKMYb;uFUCPb:pvnm0e,pfE8Hb,PFBcW;f56efd:dJXsye;EnoYf:KNzws,ZJsZZ,JgVSJc;zdMJQc:cCQNKb,ZJsZZ,zchEXc;Ytrrj:JJDvdc;tNR8yc:GeFvjb;oFN6Ye:hij5Wb" jscontroller="Zl5N8" jsdata="uqLsIf;_;$11" jsmodel="SsMkhd" jsname="txFAF"><span jsaction="click:qtZ4nf,GFf3ac,tMZCfe; contextmenu:Nqw7Te,QP7LD; mouseout:Nqw7Te; mouseover:qtZ4nf,c2aHje" jsname="W297wb"><br /></span></span></span></div><div>There are some Top Level Domains (TLDs) that don't allow for punycode domain names. .fun, for instance doesn't allow punycode URLs. Try <a href="https://domains.google.com/registrar?s=%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A5%8D%E0%A4%B0%E0%A4%AE.fun&hl=en">registering विक्रम.fun</a>, for example. But you can always register a subdomain with punycode. So can have विक्रम.circuits.fun.</div></div>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-29809663990004084952021-01-19T23:30:00.103-08:002021-01-21T19:33:39.302-08:00FreeBSD, the alternative<div class="separator"><div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="196" data-original-width="178" src="https://www.freebsd.org/layout/images/beastie.png" /></div></div><p></p> I like FreeBSD. Something about a barebones, simple UNIX that is clean at the core. Not particularly a pragmatic choice, I'll gladly admit. For most needs, a Linux installation is both simpler and more capable. More people are familiar with Linux, so this is a niche world.<p></p><p>But FreeBSD is a hobby, and a particularly gratifying hobby.</p><p><br /></p><p>The simplicity of FreeBSD is a huge draw, you can get very far with an ancient Intel machine and 1 GB of RAM. The source code is a pleasure to read, the system is easy to modify, and there is relatively little of it on a base install. Full system upgrades are clean and easy, the ports system is elegant though often impractical.</p><p>For most people, the easiest way to run FreeBSD would be to use a virtual image, and boot it up using qemu (Linux) or Hyper-V (Windows). Both work very well, both support networking, which is really what you're looking for. Face it, you're not booting into FreeBSD for its giant catalog of games or media utilities.</p><p>Get the <a href="https://ftp.tw.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/12.2-RELEASE/amd64/Latest/">VM images from here</a>. qcow2 works for QEMU, vhd works for Hyper-V, and vmdk works for VMWare. </p><p>For QEMU, I use the following commandline:</p>
<div class="highlight"><pre><span></span>qemu-system-x86_64 <span class="se">\</span>
-m <span class="m">2048</span> <span class="se">\</span>
-enable-kvm <span class="se">\</span>
-hda bsd-with-jail.img <span class="se">\</span>
-net user,hostfwd<span class="o">=</span>tcp::10023-:22 <span class="se">\</span>
-net nic <span class="se">\</span>
-curses
</pre></div>
<p>That specifies:</p><p></p><ul style="text-align: left;"><li>(<b>-m 2048</b>) 2048 MB RAM</li><li>(<b>-enable-kvm</b>) use the Linux KVM hypervisor</li><li>(<b>-hda bsd-with-jail.img</b>) the disk bsd-with-jail.img as the first hard disk </li><li>(<b>-net nic</b>) a full Network Interface Card, make your FreeBSD setup trivial as it can get a network interface called em0</li><li>(<b>-net user, hostfwd=tcp::10023-:22</b>) network translation is user-level, so no root permissions required and forward port 10023 on the host to port 22 on the guest. This allows you to ssh to the FreeBSD guest by ssh'ing port 10023 on the host.</li><li>(<b>-curses</b>) Use the text-based interface using libncurses rather than a graphical interface. Great for running on cloud instances, or through GNU screen so you can leave it on forever.</li></ul><div><br /></div><div>Other ports can be forwarded for running web servers, dns servers, ...</div><div><br /></div><div><br /></div><div>The same setup can be replicated on Hyper-V. For network, you can use the default switch. Port forwarding is more complicated, and you could achieve it with ssh tunnels on the Windows host or in the FreeBSD guest.</div><div><br /></div><div><br /></div><div>FreeBSD runs fine with 1GB of RAM, and is a great system to learn the fundamentals of computing. You can play with <a href="http://dtrace.org/guide/chp-intro.html">Dtrace</a>, create constrained environments called <a href="https://www.freebsd.org/doc/handbook/jails.html">Jails</a>, or modify the behavior of a clean UNIX kernel. A lot of this knowledge translates to Mac OS, as the OS X kernel is a hybrid of FreeBSD and Mach, and the userland is very similar to FreeBSD.</div><div><br /></div><div>You can easily run a dozen jails on 1G of RAM, depending on what you do with them. Each jail is separated from the other and can be a low-cost, throwaway computing environment. Similar to Docker, or Linux namespaces, root in a jail is safe, and cannot damage the host FreeBSD system.</div><div><br /></div><div>Set up your own UNIX sandbox today!</div><div><br /></div><br /><div><br /></div><div><br /></div><p></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-37979388424191642982021-01-07T21:21:00.001-08:002021-01-23T11:23:54.966-08:00Mac OS Kernel (XNU) source, on github<p> Few people know that Apple releases the source code for many components of their MacOS software. Many are surprised when they hear it.</p><p>Apple has a <a href="https://opensource.apple.com/">full site for their open source releases</a>. Their most recent <a href="https://opensource.apple.com/release/macos-1101.html">Mac OS release is for 11.0.1</a> (Big Sur). Not all source is released. There are the usual UNIX utilities: Perl, Python, Bash, and userland tools that make a functioning system. Functioning, but not always updated, as Apple doesn't always track the latest versions. They still ship Python 2.5, and other tools can be many years old. This is why you need to get a functional, bugfixed UNIX userland with homebrew or other tools.</p><p><br /></p><p>The most interesting source there would probably be the kernel, called <a href="https://opensource.apple.com/source/xnu/xnu-7195.50.7.100.1/">XNU</a>. It is a combination of Mach, a microkernel, and BSD, a monolith. There's some Solaris code there, some Joyent code, some Sun code, NeXT, UC Berkeley, Carnegie Mellon code, and of course, Apple code.</p><p>Apple does have an official github profile, which also <a href="https://github.com/apple/darwin-xnu">hosts xnu</a>. But it was last updated two years ago, and stands at <a href="https://github.com/apple/darwin-xnu/commit/a449c6a3b8014d9406c2ddbdc81795da24aa7443">version 4903</a>, while the latest version is 7195.</p><p><br /></p><p>Their open source browser is passable but doesn't show every file properly. I find it much easier to read through code using Emacs so I pulled it into a github repository if anyone wants to <a href="https://github.com/youngelf/xnu/tree/main/xnu-7195.50.7.100.1">clone it or browse it online</a>.</p><p><br /></p><img border="0" data-original-height="286" data-original-width="497" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRD5udpyyr6fEDBSvdYkH4C9SwzyKpzyc6mUixLQbjB9_sWguXs0FRpoXdP_aHHuL9eHfQuBmofoIj3M7PoFPeebmJdKJ6APy9RIEh_uRhnLO-KT-5-9SItXpfyG7lFV1MxQFVeQ/s16000/Screenshot+from+2021-01-07+21-19-18.png" /><br /><p>Image credit: <a href="https://github.com/youngelf/xnu/blob/main/xnu-7195.50.7.100.1/osfmk/kern/sched_amp.c">Apple Inc</a></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-40364853866432941292020-12-29T00:00:00.011-08:002022-01-22T19:45:55.735-08:00Offline-first, the design of great picture gallery websites<b>tl:dr</b>; How to use service workers to make your website responsive and functional offline
<div><br /></div><div><br /></div><div>Websites have come a long way from the boring world of static <span style="font-family: courier;"><html></span> markup. There's a lot of new functionality in browsers. This new functionality brings benefit for big websites, but these techniques are useful for small personal websites as well.</div><div><br /></div><div>One of the more important changes has been the <a href="https://jakearchibald.github.io/isserviceworkerready/">support of Service Worker</a> in many browsers. A Service Worker is Javascript code that can intercept network requests, and satisfy them locally on the client machine, rather than making server calls. There's a lot you can do with Service Workers, but I was most interested in making my home picture gallery work offline.</div><div><br /></div><div>I wanted to allow a person to load up the gallery and be able to view it on their mobile phone or desktop even if the connection is lost. My photographs are shared with static html that I create using <a href="https://github.com/saimn/sigal">sigal</a>, a simple image gallery software written in Python. It uses the <a href="https://galleriajs.github.io/">Galleria library</a> to create web pages that are self-contained. Since the galleries are static html & Javascript, they make a great case study for simple and fast web pages. In the current gallery, the images are downloaded as they are needed, with the next few images prefetched so the user doesn't have to wait. I wanted to make this entirely offline-first, so the images are downloaded, stored offline. Each image in my gallery is 150kb, and galleries have 10-20 images in them. The entire gallery is roughly 4mb, which is tiny. As a result, it loads fast and can be cached offline.</div><div><br /></div><div>You can always implement your <a href="https://developers.google.com/web/fundamentals/primers/service-workers">own Service Worker</a>, the interface is straight-forward. If you just want to use Service Workers for browser-side caching, there is a much simpler alternative. We're going to use <a href="https://www.talater.com/upup/">the upup library</a>, which is a Javascript library that can be configured to store content offline, and cache it.</div><div><br /></div><div>First, Service Workers need <b>HTTPS support</b>. Get yourself a certificate with <a href="https://letsencrypt.org/">LetsEncrypt</a>. This is a nonprofit that has issued 225 Million certificates already, and are valiantly helping the entire web move to 100% HTTPS. Get their certificate if you don't have it. Heck, get HTTPS support even if you don't want offline first on your website. I heartily endorse them, I have moved to HTTPS thanks to them. You should too.</div><div><br /></div><div>Now, let's add UpUp to your website. It is important where you place it. The upup library can answer requests for a specific scope (subdirectory of your website). It sets its scope based on its location on your website. Since you want the library to serve content to your <u>entire</u> website, you want the library to be as close to the root level of your website. Let's see a concrete example in action.</div><div><br /></div><div>Let's say your website is <span style="font-family: courier;">gallery.eggwall.com</span>. If you put the javascript at <span style="font-family: courier;">gallery.eggwall.com/<span style="background-color: white; color: #800180;">js/</span></span> then the javascript can only cache offline content for <span style="color: #800180; font-family: courier;">/js</span> and not for <span style="font-family: courier;">gallery.eggwall.com/<span style="background-color: white;"><span style="color: #660000;">g_Aircrafts/</span></span></span>. To serve content for the entire subdomain <span style="font-family: courier;">gallery.eggwall.com</span> you want the Javascript at <span style="font-family: courier;">gallery.eggwall.com/</span></div><div><br /></div><div><br /></div><div>We're going to put it at <span style="font-family: courier;">gallery.eggwall.com</span>, with this in the html:</div><div>
<div class="highlight"><pre><span></span> <span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"https://gallery.eggwall.com/upup.min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
</pre></div>
</div><div><br /></div><div><div><br /></div></div><div>So <a href="https://github.com/TalAter/UpUp/tree/master/dist">download the upup library</a> and put the contents of the dist/ directory at the root level of your website. Putting random javascript like this is usually a bad idea, so <a href="https://github.com/TalAter/UpUp/blob/master/src/upup.sw.js">examine the source</a> code, make sure you understand what the Javascript is doing. The <a href="https://github.com/TalAter/UpUp/blob/master/src/upup.sw.js">service worker source</a> is much more important than the <a href="https://github.com/TalAter/UpUp/blob/master/src/upup.js">framework code</a>.</div><div><br /></div><div>Now that you have the library in place, invoke the UpUp.start method, and give it the base web page (<span style="font-family: courier;">index.html</span>) and all the content that you want it to cache. The references here have to be relative to the location of the <span style="font-family: courier;">upup.sw.min.js</span>. If you put the library in the root of your page, all the references here have to be relative to your root page:</div>
<div class="highlight"><pre><span></span> <span class="p"><</span><span class="nt">script</span><span class="p">></span>
UpUp.start({
'content-url': 'g_Aircraft/index.html',
'assets': [
"/static/jquery-3.3.1.min.js",
"/static/galleria.min.js",
"/static/themes/classic/galleria.classic.min.js",
"/static/plugins/history/galleria.history.min.js",
</pre></div>
<div><br /></div><div><br /></div><div><br /></div><div>For simple pages like this, I find it helpful to include the <span style="font-family: courier;"><base></span> tag to remind me that everything is relative to the root:</div>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">base</span> <span class="na">href</span><span class="o">=</span><span class="s">"https://gallery.eggwall.com"</span><span class="p">></span></pre></div><div><br /></div><div><div> On this gallery, all images and content is stored in the subdirectory <span style="color: #660000; font-family: courier;">g_Aircraft/</span>. All thumbnails are stored in <span style="color: #660000; font-family: courier;">g_Aircraft/</span><span style="color: #660000; font-family: courier;">thumbnails/</span>. So you want to load up all the images in upup.start:</div><div><br /></div><div><div><br /></div>
<div class="highlight"><pre><span></span> <span class="p"><</span><span class="nt">script</span><span class="p">></span>
<span class="nx">UpUp</span><span class="p">.</span><span class="nx">start</span><span class="p">({</span>
<span class="s1">'content-url'</span><span class="o">:</span> <span class="s1">'g_Aircraft/index.html'</span><span class="p">,</span>
<span class="s1">'assets'</span><span class="o">:</span> <span class="p">[</span>
<span class="s2">"/static/jquery-3.3.1.min.js"</span><span class="p">,</span>
<span class="s2">"/static/galleria.min.js"</span><span class="p">,</span>
<span class="s2">"/static/themes/classic/galleria.classic.min.js"</span><span class="p">,</span>
<span class="s2">"/static/plugins/history/galleria.history.min.js"</span><span class="p">,</span>
<span class="s1">'g_Aircraft/_DSC1984.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/thumbnails/_DSC1984.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/_DSC1986.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/thumbnails/_DSC1986.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/_DSC1989.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/thumbnails/_DSC1989.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/_DSC1991.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/thumbnails/_DSC1991.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/_DSC1992.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/thumbnails/_DSC1992.JPG'</span><span class="p">,</span>
<span class="p">...</span>
<span class="s1">'g_Aircraft/_DSC2148.JPG'</span><span class="p">,</span>
<span class="s1">'g_Aircraft/thumbnails/_DSC2148.JPG'</span><span class="p">,</span>
<span class="p">]</span>
<span class="p">});</span>
<span class="p"></</span><span class="nt">script</span><span class="p">></span>
</pre></div>
<div><br /></div><div>You don't need to change anything on the server for this. I use nginx, but anything serving out static pages will do just fine. Offline-first changes your server-side metrics because many requests are handled directly on the client. So you won't be able to see when the client loaded the page again. Browser-side caching messes with these numbers too, so if you will have to roll your own Javacript if you want perfect interaction tracking.</div><div><br /></div><div>These changes are fine for browsers that don't support service workers. Older browsers will be served the static content. Since they don't initialize a Service Worker, all requests will go to the server, as before. The Upup.start section just gets ignored. Browser-side caching will continue working as before, too.</div><div><br /></div><div>With this, the UpUp service worker will cache all the content specified in assets above. The user can go offline, and the page still functions normally. The <a href="https://gallery.eggwall.com/g_Aircraft/">gallery demo is available</a> if you want to play with it.</div><div><div><br class="Apple-interchange-newline" />Service Workers add complexity. You can debug the site using Chrome Developer tools -> "Application" -> "Service Workers" or "Web Developer" -> Application -> "Service Workers" in Firefox. You want to check if the service worker initialized and is storing content in "Cache storage" -> upup-cache.</div><div><br /></div></div><div>Here's a demo video on Android. You can see the user load up the site on their mobile browser, go offline, and still navigate normally.</div></div></div><div><br /></div><div><br /></div><div style="position:relative;padding-bottom:56.25%;height:0;overflow:hidden;"> <iframe style="width:100%;height:100%;position:absolute;left:0px;top:0px;overflow:hidden" frameborder="0" type="text/html" src="https://www.dailymotion.com/embed/video/k5z1Y5VYE5aBdjxyWu2" width="100%" height="100%" allowfullscreen > </iframe> </div>
<br /><div><br /></div>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-33845655301707855282020-12-21T20:27:00.006-08:002020-12-22T19:26:25.930-08:00Audio feature extraction for Machine Learning<p><b>tl:dr</b>; Books and papers for audio processing, for building Machine Learning models.</p><p><br /></p><p>I've been experimenting with Machine Learning for audio files.</p><p>Much machine learning literature for music deals with MIDI files, which are a digital format for specifying notes, duration and loudness. This is the format to use for models that work on the level of individual notes. A simple introduction for training such models is using the book "<a href="https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1492032646/ref=sr_1_2?crid=1BA7K313TYTK1&dchild=1&keywords=hands+on+machine+learning+with+scikit-learn+and+tensorflow+2&qid=1608610250&sprefix=hands+on+machine%2Caps%2C220&sr=8-2">Hands On Machine Learning 2nd</a> edition" (2019), An exercise in Chapter 15 (RNNs) introduces you to <a href="https://github.com/ageron/handson-ml2/blob/master/15_processing_sequences_using_rnns_and_cnns.ipynb">Bach chorales</a>, and shows how to generate chords from digial music. Google's <a href="https://magenta.tensorflow.org/">Magenta project</a> has datasets and models for such discrete note-level training, generation and inference.</p><p><br /></p><p>While MIDI is a convenient format for discrete music, most music data is stored as waveforms rather than MIDI. These are either raw WAV / FLAC, or encoded with the MP3 or Ogg Vorbis encoding. Extracting features from these files is considerably harder and requires a good understanding of audio analysis and the kinds of features that these waveforms represent. Depending on the audio stream, some understanding of music structure might come in handy.</p><p><a href="https://github.com/MTG/essentia">Essentia</a> is a freely-available library for handling audio information for music analysis, with bindings for C++ and Python. A <a href="https://essentia.upf.edu/essentia_python_tutorial.html">Python tutorial</a> on Essentia covers some of the basics.</p><p>A survey paper "<a href="https://www.ntnu.edu/documents/1001201110/1266017954/DAFx-15_submission_43_v2.pdf">An Evaluation of Audio Feature Extraction toolboxes</a>", by Moffat D., Ronan D., and Reiss J.D. (2015) covers some toolkits in a variety of languages.</p><p><br /></p><p>In order to use any toolkit for feature extraction, you need to know what features to look for, and which algorithms to select. This is a fast-moving area of research. The book: "<a href="https://www.amazon.com/Fundamentals-Music-Processing-Algorithms-Applications/dp/3319357654/ref=sr_1_1?dchild=1&keywords=fundamentals+of+music+processing&qid=1608524245&sr=8-1">Fundamentals of Music Processing</a>", by Meinard Müller (2016) covers all the background on audio encoding, music representation, and analysis algorithms. The first two chapters cover the core concepts, and chapters 3 onwards dive into individual topics, and can then be read in parallel. This allows software engineers to understand the basics, and then immediately focus on the task at hand. This book is dense and requires a firm understanding of Linear Algebra. Once you know the terminology, you can read the relevant papers on feature types and either use a readily available library, or write the extraction code yourself.</p><p><br /></p><p>Finally, <a href="https://mtg.github.io/essentia-labs/">pre-trained models in Essentia</a> allow you to use existing models for classification tasks. An <a href="https://essentia.upf.edu/demos/deep-learning/autotagging">online demo</a> exists to test out the functionality in a browser.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgREUihyphenhyphenIRtK_GP-JhgDJ66L_SIYp41NDpmpJkFtsQMK0h66s13SImKmy_M_PPhQ5Iofbsu_wzA-jIf5elZ6IPX94iBwiyRIdMOxibtRsCticI0_BVui_yF-2cTdtwFBAz6rn9eFw/s499/fomp.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="499" data-original-width="333" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgREUihyphenhyphenIRtK_GP-JhgDJ66L_SIYp41NDpmpJkFtsQMK0h66s13SImKmy_M_PPhQ5Iofbsu_wzA-jIf5elZ6IPX94iBwiyRIdMOxibtRsCticI0_BVui_yF-2cTdtwFBAz6rn9eFw/s320/fomp.jpg" /></a></div><br /><p><br /></p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-59588952552843926692020-12-11T23:11:00.055-08:002020-12-12T08:33:58.406-08:00Tailscale: the best, secure, private VPN you need<p><b> tl:dr</b>; Tailscale easily sets up remote machines as though they were on your local network (VPN)</p><p><br /></p><p>Tell me if this situation sounds familiar: you have a variety of machines, at home and at remote locations. All combinations: behind a proxy/NAT, connected directly, a mix of Linux and Mac/Windows systems, a mixture of physical hardware and cloud instances. You want these machines to behave like they're on a local network and to use them without jumping through hoops. You could access these machines with proxies like ngrok or other tunneling software like ssh, but that's complicated.</p><p>You could set up a VPN, but that is time consuming and difficult to manage. Setting up a VPN requires skill and is difficult to get right across host platforms or architectures.</p><p><br /></p><p><a href="https://tailscale.com/">Tailscale</a> is an excellent, simple-to-setup and secure VPN, with clients available for all major systems and architectures. You use an existing authentication (like your Gmail address), download the client software for your platform, and authenticate by navigating to a web address. Setting it up is refreshingly simple. It even sets up a dns, so you can refer to your machines by their hostname: p31 can be used instead of the full VPN IP address like 100.107.137.29.</p><p>I've used a variety of VPN systems in the past; I've also set up my own tunnels using different providers; I've rolled my own tunnels from first principles. Compared to existing systems, Tailscale is easier to setup, efficient, and has great network performance. Network <a href="https://tailscale.com/blog/how-tailscale-works/">latencies are lower than traditional hub-and-spoke systems</a>, which relay through a central server. If the central server is located far from both VPN'd machine, network performance is usually poor.</p><p>Right now I'm pinging machines that are behind a NAT, accessing web pages on a different physical network, all by referring to simple hostnames. There's arm32, arm64, x64, different operating systems, physical and cloud instances that all appear as a local Class-A network. This is like magic!</p><p>Tailscale is also great for working on projects on your cloud or local instance without exposing it to the wild Internet traffic.</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhl4fQvtGo6YNHv9IcDclf2ep4Pyi6gNr4_ETOb9H6FDVtC7WcjjIEk5IIqs4BRBpscUJ3fQ4mmWqLxo91K7pUxuB6udHT_qYHW1VaWxbybjjfYFCDExtTyuTPx6vZV3XZv_Lh8Qg/s627/Bus_Topology.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="338" data-original-width="627" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhl4fQvtGo6YNHv9IcDclf2ep4Pyi6gNr4_ETOb9H6FDVtC7WcjjIEk5IIqs4BRBpscUJ3fQ4mmWqLxo91K7pUxuB6udHT_qYHW1VaWxbybjjfYFCDExtTyuTPx6vZV3XZv_Lh8Qg/s16000/Bus_Topology.png" /></a></div><br /></div><br /><p><br /></p><p>Image courtesy: <a href="https://commons.wikimedia.org/wiki/File:Bus_Topology.png">Wikipedia</a>.</p><br />Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-56786037996912667972020-11-28T19:59:00.003-08:002020-11-28T20:16:21.051-08:00Generating art with Tensorflow<p><b>tl:dr;</b> Thought-provoking, computer-generated art</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihu__ayJXhoCkNWsq7EF_EE9EXRyw5MsW4RUq-YxS6Kuv2A6g0HOXGszn3q1-tpSvC0MtfeqHYLJCNf_vgke58UdGC7iyp3GT81Z4cZm65_341uuhLfdCjVhJbA49v4gOEbo1DXQ/s710/kan-tiger.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="474" data-original-width="710" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihu__ayJXhoCkNWsq7EF_EE9EXRyw5MsW4RUq-YxS6Kuv2A6g0HOXGszn3q1-tpSvC0MtfeqHYLJCNf_vgke58UdGC7iyp3GT81Z4cZm65_341uuhLfdCjVhJbA49v4gOEbo1DXQ/s16000/kan-tiger.png" /></a></div><p>That is computer generated: the output of transfer learning using Tensorflow.</p><p>I was playing with some transfer learning, training a convolutional model on famous works of art, and applying the style to my own photographs when I realized the process could be done in reverse. So instead of taking the style of a famous artist, and applying it to my photographs, I've taken the style of my photographs and applied it to famous art. Here are the two pictures, the content is a famous Kandinsky image, and the style applied was that of a tiger in San Francisco.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKwUTI8WBknXkCrIcnapkNwkZteHyFCcM_1-icYeSUy8jJ_Be9j6yZ-0FB-eBOKAC4SdQgJ5tilBEQLQy46ZfW-cLfJDLaH3q4rh9N8ACtvyoGOgtaWEncRlLXTsqJPjlYDYSKqg/s713/tiger.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="250" data-original-width="713" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKwUTI8WBknXkCrIcnapkNwkZteHyFCcM_1-icYeSUy8jJ_Be9j6yZ-0FB-eBOKAC4SdQgJ5tilBEQLQy46ZfW-cLfJDLaH3q4rh9N8ACtvyoGOgtaWEncRlLXTsqJPjlYDYSKqg/s16000/tiger.png" /></a></div><div><br /></div><p>Here's another. A famous Monet that has been modified with the style of a giraffe from the same zoo. The original image is pretty serene, and I love its transformation into a drug-fueled nightmare.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgb7XmiMnVVbXP-rzQpnFHRjNwc3IP3SAl7IJ1I_dAfV8CSl1w39MiN-MtZ-BQ9QR9ZALo6MSMzcIvd731M8eNzDy5sh2tLldMYnBUO-Vm0Bw9Htun-rAnJ1O72Vkqm7_rLVT103A/s710/monet-giraffe.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="557" data-original-width="710" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgb7XmiMnVVbXP-rzQpnFHRjNwc3IP3SAl7IJ1I_dAfV8CSl1w39MiN-MtZ-BQ9QR9ZALo6MSMzcIvd731M8eNzDy5sh2tLldMYnBUO-Vm0Bw9Htun-rAnJ1O72Vkqm7_rLVT103A/s16000/monet-giraffe.png" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNlnNyaLoBHQSQhOO9f-s6O7Ou3H6_jhbxgeQJRxFYvpcoG6fIT9ORc0xxBFm_8mukOdmMzxHTrG11eCYgeJRRwTubv1vidqOtTDieqtDF7_TeYrfxJIkKEr2sMgPaK4trgQp9VQ/s710/monet.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="504" data-original-width="710" height="454" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNlnNyaLoBHQSQhOO9f-s6O7Ou3H6_jhbxgeQJRxFYvpcoG6fIT9ORc0xxBFm_8mukOdmMzxHTrG11eCYgeJRRwTubv1vidqOtTDieqtDF7_TeYrfxJIkKEr2sMgPaK4trgQp9VQ/w640-h454/monet.png" width="640" /></a></div><p>And finally, my favorite, two mundane pictures that together make evocative art.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKR4weBHju6KzW19e1j1T0zQQhA9mXhMsSCA-_DeuTHpmc8dup_CxsqGq20WIwSsVsdnZN07-EuL31EV5T2AxiVqoVRWx8MCm2LindLNom2Wy8aN_ZcelCWdiYuI-5zxtfKonJ4w/s710/dog-styled.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="589" data-original-width="710" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKR4weBHju6KzW19e1j1T0zQQhA9mXhMsSCA-_DeuTHpmc8dup_CxsqGq20WIwSsVsdnZN07-EuL31EV5T2AxiVqoVRWx8MCm2LindLNom2Wy8aN_ZcelCWdiYuI-5zxtfKonJ4w/s16000/dog-styled.png" /></a></div><div>The content image is from the page linked below, and the lion is my own photograph. Either picture is average by itself.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga8vZCkdpT82AUwosBc3g8M4hV64TiNnnSAwCh54hiGTD4ZisFCXqrAELIkYoBABLj0IXHCtmrrVryPUju1_1dWymr3Cl4oTMYbonm5pnU5pBCNmTDRQ5q4neIt0tBPyKk4hhwOw/s713/dog.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="298" data-original-width="713" height="269" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga8vZCkdpT82AUwosBc3g8M4hV64TiNnnSAwCh54hiGTD4ZisFCXqrAELIkYoBABLj0IXHCtmrrVryPUju1_1dWymr3Cl4oTMYbonm5pnU5pBCNmTDRQ5q4neIt0tBPyKk4hhwOw/w640-h269/dog.png" width="640" /></a></div><br /><p>A computer and a human are more superior at chess than a computer alone, or a human alone. The same can be said about art. A human combined with a well-written software can make something amazing. </p><p>This is based on the <a href="https://www.tensorflow.org/tutorials/generative/style_transfer">Tensorflow style transfer tutorial</a>.</p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-14593774789599406732020-11-26T18:10:00.004-08:002020-11-26T18:30:35.819-08:00Best Practices with Tensorflow: Test All Your Inputs Early<div>Tensorflow model training can take a long time. Depending on the machine, even the first epoch can take many hours to run. In this case, the remaining arguments like validation_data are only evaluated much later. Incorrectly formed input will be detected many hours later, <a href="/2020/11/tensorflow-errors-are-completely.html">deep in the stack, with incomprehensible errors</a>. By this time, you might have forgotten what the shapes of the input or their expected data type is. You want to detect failures early, when you run model.fit() rather than a day later.</div><div><br /></div><div>One easy way to avoid this is to wrap all model.fit() calls in a single method, which <b>tests itself out</b> with a smaller input. Here's the idea. If you have training data (X_train and y_train), and validation data (X_valid and y_valid), you take the first 10 observations, and call the training method itself.</div><div><br /></div><div>This run will be fast, because you are training a model with a tiny set of observations, and the validation is done with a tiny set of values too.</div><div><br /></div><div>It also ensures that any changes to the model.fit() line are correctly reflected both in the test_shapes run and in the real run. Any training done on the model is minor: a real run with the full data and full set of epochs will change the model weights, so this is harmless. In case you are using a pre-trained model, you can clone the model before trying the test run.</div><div><br /></div><div>If you are using data generators, then the same mechanism can be used. Create a new tfdata object with a few observations both on the training data, and another with the validation data. Pass that to your model training function.</div><div><br /></div><div class="highlight" style="background: rgb(248, 248, 248);"><pre style="line-height: 125%;"><span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">fit_e9_model</span>(model, X_train, y_train,
X_valid, y_valid, epochs,
batch_size<span style="color: #666666;">=32</span>, verbose<span style="color: #666666;">=0</span>,
test_shapes<span style="color: #666666;">=</span><span style="color: green; font-weight: bold;">True</span>):
<span style="color: #408080; font-style: italic;"># This fails after a day, when the validation data is incorrectly shaped. </span>
<span style="color: #408080; font-style: italic;"># This is a terrible idea. Failures should be early. </span>
<span style="color: #408080; font-style: italic;"># The best way to guard against it is to run a small fit run with </span>
<span style="color: #408080; font-style: italic;"># a tiny data size, and a tiny validation data size to ensure that </span>
<span style="color: #408080; font-style: italic;"># the data is correctly shaped. </span>
<span style="color: green; font-weight: bold;">if</span> (test_shapes):
<span style="color: green;">print</span> (<span style="color: #ba2121;">"Testing with the first 10 elements of the input"</span>)
X_small <span style="color: #666666;">=</span> X_train[:<span style="color: #666666;">10</span>,:,:,:]
y_small <span style="color: #666666;">=</span> y_train[:<span style="color: #666666;">10</span>]
X_valid_small <span style="color: #666666;">=</span> X_valid[:<span style="color: #666666;">10</span>,:,:,:]
y_valid_small <span style="color: #666666;">=</span> y_valid[:<span style="color: #666666;">10</span>]
<span style="color: #408080; font-style: italic;"># Call ourselves again with a smaller input. This confirms </span>
<span style="color: #408080; font-style: italic;"># that the two methods are calling the same .fit() method, and </span>
<span style="color: #408080; font-style: italic;"># that the input is correctly shaped in the original method too. </span>
fit_e9_model(model, X_small, y_small,
X_valid_small, y_valid_small,
epochs<span style="color: #666666;">=</span>epochs, verbose<span style="color: #666666;">=</span>verbose,
batch_size<span style="color: #666666;">=5</span>,
test_shapes<span style="color: #666666;">=</span><span style="color: green; font-weight: bold;">False</span>)
<span style="color: #408080; font-style: italic;"># If that worked, then do a full run. </span>
history_conv <span style="color: #666666;">=</span> model<span style="color: #666666;">.</span>fit(x<span style="color: #666666;">=</span>X_train, y<span style="color: #666666;">=</span>y_train, batch_size<span style="color: #666666;">=</span>batch_size,
validation_data<span style="color: #666666;">=</span>(X_valid, y_valid),
epochs<span style="color: #666666;">=</span>epochs, verbose<span style="color: #666666;">=</span>verbose)
<span style="color: green; font-weight: bold;">return</span> history_conv
history <span style="color: #666666;">=</span> fit_e9_model(model, X_train, y_train, X_valid, y_valid, epochs<span style="color: #666666;">=10</span>)
</pre></div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-68778617848434513942020-11-22T17:31:00.015-08:002022-01-03T09:49:46.176-08:00User-facing tools should have user-understandable errorsHere's some simple Tensorflow code to create a CNN, and train it. <div>
<div class="highlight" style="background: rgb(248, 248, 248);"><pre style="line-height: 125%;"><span></span><span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">tensorflow</span> <span style="color: green; font-weight: bold;">as</span> <span style="color: blue; font-weight: bold;">tf</span>
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">tensorflow.keras</span> <span style="color: green; font-weight: bold;">as</span> <span style="color: blue; font-weight: bold;">keras</span>
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">create_model</span>(optimizer<span style="color: #666666;">=</span><span style="color: #ba2121;">"sgd"</span>):
deep_model <span style="color: #666666;">=</span> keras<span style="color: #666666;">.</span>models<span style="color: #666666;">.</span>Sequential([
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Conv2D(<span style="color: #666666;">64</span>, <span style="color: #666666;">7</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, padding<span style="color: #666666;">=</span><span style="color: #ba2121;">"same"</span>,
input_shape<span style="color: #666666;">=</span>[<span style="color: #666666;">1</span>, <span style="color: #666666;">28</span>, <span style="color: #666666;">28</span>], name<span style="color: #666666;">=</span><span style="color: #ba2121;">"input"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>MaxPooling2D(<span style="color: #666666;">1</span>,name<span style="color: #666666;">=</span><span style="color: #ba2121;">"firstPool"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Conv2D(<span style="color: #666666;">128</span>, <span style="color: #666666;">3</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, padding<span style="color: #666666;">=</span><span style="color: #ba2121;">"same"</span>,
name<span style="color: #666666;">=</span><span style="color: #ba2121;">"first_conv_1"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Conv2D(<span style="color: #666666;">128</span>, <span style="color: #666666;">3</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, padding<span style="color: #666666;">=</span><span style="color: #ba2121;">"same"</span>,
name<span style="color: #666666;">=</span><span style="color: #ba2121;">"first_conv_2"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>MaxPooling2D(<span style="color: #666666;">1</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"secondPool"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Conv2D(<span style="color: #666666;">256</span>, <span style="color: #666666;">3</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, padding<span style="color: #666666;">=</span><span style="color: #ba2121;">"same"</span>,
name<span style="color: #666666;">=</span><span style="color: #ba2121;">"second_conv_1"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Conv2D(<span style="color: #666666;">256</span>, <span style="color: #666666;">3</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, padding<span style="color: #666666;">=</span><span style="color: #ba2121;">"same"</span>,
name<span style="color: #666666;">=</span><span style="color: #ba2121;">"second_conv_2"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>MaxPooling2D(<span style="color: #666666;">1</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"thirdPool"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Flatten(name<span style="color: #666666;">=</span><span style="color: #ba2121;">"flatten"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Dense(<span style="color: #666666;">128</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"pre-bottneck"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Dropout(<span style="color: #666666;">0.5</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"bottleneckDropout"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Dense(<span style="color: #666666;">64</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"bottleneck"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Dropout(<span style="color: #666666;">0.5</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"outputDropout"</span>),
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Dense(<span style="color: #666666;">10</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"softmax"</span>, name<span style="color: #666666;">=</span><span style="color: #ba2121;">"output"</span>),
])
deep_model<span style="color: #666666;">.</span>compile(loss<span style="color: #666666;">=</span><span style="color: #ba2121;">"sparse_categorical_crossentropy"</span>,
optimizer<span style="color: #666666;">=</span>optimizer,
metrics<span style="color: #666666;">=</span>[<span style="color: #ba2121;">"accuracy"</span>])
<span style="color: green; font-weight: bold;">return</span> deep_model
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">fit_model</span>(model, X_train, y_train, X_valid, y_valid, epochs):
history_conv <span style="color: #666666;">=</span> model<span style="color: #666666;">.</span>fit(X_train, y_train, validation_data<span style="color: #666666;">=</span>[X_valid, y_valid],
epochs<span style="color: #666666;">=</span>epochs, verbose<span style="color: #666666;">=0</span>)
<span style="color: green; font-weight: bold;">return</span> history_conv
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">plot_history</span>(history, name):
c10<span style="color: #666666;">.</span>plot_training(history, name, show<span style="color: #666666;">=</span><span style="color: green; font-weight: bold;">True</span>)
model <span style="color: #666666;">=</span> create_model()
history <span style="color: #666666;">=</span> fit_model(model, X_train, y_train, X_valid, y_valid, epochs<span style="color: #666666;">=10</span>)
plot_history(history, <span style="color: #ba2121;">"naive_deep_mnist"</span>)
</pre></div>
<div><br /></div>When you run it, it causes this <b>enormous error</b>: </div><span><pre> ---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-26-95a7830ebd3c> in <module>
41
42 model = create_model()
---> 43 history = fit_model(model, X_train, y_train, X_valid, y_valid, epochs=10)
44 plot_history(history, "naive_deep_mnist")
<ipython-input-26-95a7830ebd3c> in fit_model(model, X_train, y_train, X_valid, y_valid, epochs)
34
35 def fit_model(model, X_train, y_train, X_valid, y_valid, epochs):
---> 36 history_conv = model.fit(X_train, y_train, validation_data=[X_valid, y_valid], epochs=epochs, verbose=0)
37 return history_conv
38
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
106 def _method_wrapper(self, *args, **kwargs):
107 if not self._in_multi_worker_mode(): # pylint: disable=protected-access
--> 108 return method(self, *args, **kwargs)
109
110 # Running inside `run_distribute_coordinator` already.
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1096 batch_size=batch_size):
1097 callbacks.on_train_batch_begin(step)
-> 1098 tmp_logs = train_function(iterator)
1099 if data_handler.should_sync:
1100 context.async_wait()
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
778 else:
779 compiler = "nonXla"
--> 780 result = self._call(*args, **kwds)
781
782 new_tracing_count = self._get_tracing_count()
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
821 # This is the first call of __call__, so we have to initialize.
822 initializers = []
--> 823 self._initialize(args, kwds, add_initializers_to=initializers)
824 finally:
825 # At this point we know that the initialization is complete (or less
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
694 self._graph_deleter = FunctionDeleter(self._lifted_initializer_graph)
695 self._concrete_stateful_fn = (
--> 696 self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access
697 *args, **kwds))
698
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
2853 args, kwargs = None, None
2854 with self._lock:
-> 2855 graph_function, _, _ = self._maybe_define_function(args, kwargs)
2856 return graph_function
2857
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
3211
3212 self._function_cache.missed.add(call_context_key)
-> 3213 graph_function = self._create_graph_function(args, kwargs)
3214 self._function_cache.primary[cache_key] = graph_function
3215 return graph_function, args, kwargs
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
3063 arg_names = base_arg_names + missing_arg_names
3064 graph_function = ConcreteFunction(
-> 3065 func_graph_module.func_graph_from_py_func(
3066 self._name,
3067 self._python_function,
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
984 _, original_func = tf_decorator.unwrap(python_func)
985
--> 986 func_outputs = python_func(*func_args, **func_kwargs)
987
988 # invariant: `func_outputs` contains only Tensors, CompositeTensors,
/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
598 # __wrapped__ allows AutoGraph to swap in a converted function. We give
599 # the function a weak reference to itself to avoid a reference cycle.
--> 600 return weak_wrapped_fn().__wrapped__(*args, **kwds)
601 weak_wrapped_fn = weakref.ref(wrapped_fn)
602
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
971 except Exception as e: # pylint:disable=broad-except
972 if hasattr(e, "ag_error_metadata"):
--> 973 raise e.ag_error_metadata.to_exception(e)
974 else:
975 raise
ValueError: in user code:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function *
return step_function(self, iterator)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
return fn(*args, **kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step **
outputs = model.train_step(data)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:747 train_step
y_pred = self(x, training=True)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:975 __call__
input_spec.assert_input_compatibility(self.input_spec, inputs,
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/input_spec.py:191 assert_input_compatibility
raise ValueError('Input ' + str(input_index) + ' of layer ' +
ValueError: Input 0 of layer sequential_17 is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [None, 28, 28]
</pre></span><span><a name='more'></a></span><div><br /></div><div><br /></div><div> Don't worry if you don't understand the error. In fact, that's the whole point. The error is the entire stack trace of the Python execution, with no human-readable information on the failing component.<div><br /></div><div><br /></div><div><br /></div><div><b>Lesson</b>: Tools are meant to be helpful to humans. When the failure is longer and less understandable than the entire source, the tool is unhelpful.</div><div><br /></div><div>The problems are two-fold:</div><div><ul style="text-align: left;"><li>Instead of referring to the layer name, the stack trace points to 'sequential_17'. All the layers have names, and none of them is called sequential_17.</li><li>Clearly the input layer is to blame: it's the only 28x28 input layer. But why are the dimensions expected to be 4? I suspect the input data needs to be reshaped. But it isn't clear what it should be reshaped to.</li></ul></div></div><div>The right thing here is for model.compile(...) and model.fit(...) to run a pass on all the layers, validating their input parameters for common mistakes. If the inputs fail, then it should list out the layer that caused the failure. It should also validate the sizes of previous outputs and present inputs, ensuring that they conform.</div><div><br /></div><div>This error is after many minutes of getting the correct model specified. If you want a real headache of an error, run this code without 'name=' on one of the Dropout layers.</div><div><br /></div><div><br /></div><div>User-facing tools should have user-facing errors.</div><div><br /></div><span></span><div><br /></div><div><br /></div><span></span><div><hr /></div><div><span><!--more--></span>In case you came here because you are facing the same problem with a Conv2D layer, the answer is relatively simple. For grayscale images, you still need the input_shape to be (28, 28, 1). This is because the convolution works on layers, and monochrome images have a single layer. When you do that, you also need to reshape your input to be (size, 28, 28, 1) from (size, 28, 28). </div>
<div class="highlight" style="background: rgb(248, 248, 248);"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic;"># X_train.shape = (55000,28,28)</span>
X_train <span style="color: #666666;">=</span> X_train<span style="color: #666666;">.</span>reshape(<span style="color: #666666;">55000</span>,<span style="color: #666666;">28</span>,<span style="color: #666666;">28</span>,<span style="color: #666666;">1</span>)
<span style="color: #408080; font-style: italic;"># X_valid.shape = (5000,28,28)</span>
X_valid <span style="color: #666666;">=</span> X_valid<span style="color: #666666;">.</span>reshape(<span style="color: #666666;">5000</span>,<span style="color: #666666;">28</span>,<span style="color: #666666;">28</span>,<span style="color: #666666;">1</span>)
<span style="color: #408080; font-style: italic;"># Creating the model</span>
deep_model <span style="color: #666666;">=</span> keras<span style="color: #666666;">.</span>models<span style="color: #666666;">.</span>Sequential([
keras<span style="color: #666666;">.</span>layers<span style="color: #666666;">.</span>Conv2D(<span style="color: #666666;">32</span>, <span style="color: #666666;">4</span>, activation<span style="color: #666666;">=</span><span style="color: #ba2121;">"relu"</span>, padding<span style="color: #666666;">=</span><span style="color: #ba2121;">"same"</span>,
input_shape<span style="color: #666666;">=</span>(<span style="color: #666666;">28</span>, <span style="color: #666666;">28</span>, <span style="color: #666666;">1</span>), name<span style="color: #666666;">=</span><span style="color: #ba2121;">"input"</span>),
<span style="color: #408080; font-style: italic;"># ... and other layers.</span>
<span style="color: #408080; font-style: italic;"># Training</span>
deep_model<span style="color: #666666;">.</span>fit(x<span style="color: #666666;">=</span>X_train, <span style="color: #666666;">...</span>
</pre></div>
<div><br /></div><div><br /></div><div><br /></div><div><br /></div>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-49920601475959215162020-09-19T23:45:00.007-07:002020-09-20T19:45:09.073-07:00Compiling Tensorflow without AVX support, a Googler's perspective<p><b>tl:dr</b>; Tensorflow compilation teaches you about the complexity of present-day software design.</p><p><br /></p><p>Compiling Tensorflow is a curious experience. If I put on my external-user hat, the process is baffling. Many Tensorflow choices are motived by development practices inside Google, rather than common open source development idioms. And so, as a Google engineer, I can explain what is going on and the motivations for the choices.</p><h2 style="text-align: left;">Why compile Tensorflow?</h2><p>I want to run Tensorflow on an old Intel CPU that doesn't have <a href="https://en.wikipedia.org/wiki/Advanced_Vector_Extensions">AVX-instruction support</a>. These are special "vector" instructions that speed the computation on large data streams, and are available on relatively new Intel and AMD processors. The <a href="https://github.com/tensorflow/tensorflow/issues/24548">solution is to compile Tensorflow from source</a> as Tensorflow prepackaged binaries (after version 1.6) are compiled expecting AVX support on the processor. </p><p>No problem: I'm an engineer and have done my share of large system compilations. I can do this.</p><h2 style="text-align: left;">Dependencies</h2><p>Tensorflow compilation has only been tested on gcc 7.3.0, which was released in November, 2019. The latest version of gcc shipping with Ubuntu 20.04 is 9.3.0. If a user is compiling software from source, they are probably going to use a recent version of the compiler toolchain. I doubt most users will install an old version of gcc on their machine (or in a Docker image) just to compile Tensorflow. I didn't either, and went with gcc-9.3 with fingers crossed.</p><p>Perhaps it is the complexity of software development today. With the pace of development and releases: you cannot possibly support all versions of gcc, all versions of Ubuntu, Debian, Mac OS, Windows, all combinations of compute architecture: x86, x86_64, x86 with AVX, arm64, GPU with cuda. Add to this all the complexity of different target platforms: python, C++, ...</p><p>Unlike a few years ago, compilers like gcc and llvm are themselves being updated frequently. This is great, as bugs can be fixed, but it leads to a large burden on supporting different toolchains.</p><h2>Lessons</h2><div>Tensorflow downloads its own version of llvm. Instead of relying on the system version of llvm, which might have its own quirks, it just gets everything.</div><div><br /></div><div>That's not all: Tensorflow downloads all its dependencies: boringssl, eigen3, aws libraries, protobuf libraries, llvm-project from github or their respective repositories. I suspect most of these go into //third-party. </div><div><br /></div><div>It is an interesting choice to download most of these rather than expecting them installed locally. On one level, it reduces the complexity in figuring why Tensorflow builds on Ubuntu versus Fedora, or on FreeBSD. But managing these packages also adds complexity. How do you know which version of protobuf or llvm to check-out, and what if those dependencies are no longer available.</div><div>The obvious complexity is that you have to compile your dependencies too. A user might already have a pre-compiled llvm and the protobuf library.</div><div><br /></div><div>If anything, the Tensorflow style looks similar to Android Open Source (AOSP) or FreeBSD's Ports collection. In both of these, the downloaded repository creates a parallel universe of source and objects. You compile everything from scratch. The notable difference between FreeBSD is that the output of FreeBSD's Ports are installed in /usr/local/ and are then available to be used on the system. After you compile protobufs for Tensorflow, you still don't have the protobuf library available to the wider system.</div><div><br class="Apple-interchange-newline" />The reason for this is probably because Google engineers compile the whole world. Google production binaries shouldn't rely on the specific version of eigen3 you happen to have on your development machine. Instead, you get a specific version of eigen3 from the repository (the "mono" repo), and use that. Ditto for llvm. Most of this open-source dependency code does not diverge too far from upstream, as bugfixes are reported back to the authors. This provides some sanity of dependencies. I suspect the version of llvm or eigen3 chosen are the same versions that were in the mono repo at the time Tensorflow 2.4 was released. </div><div><br /></div><div>This differs from other large open source projects. You are expected to have all the dependencies locally if you are to compile Emacs. It needs libjpeg, so install that through apt or yum. Then you realize you need x11 libraries. Ok, go get those separately. Cumbersome, and it increases the risk of a failure at runtime as your version of libjpeg might not be what the authors tested against.</div><div><br /></div><div><div>Bazel does help when compiling everything. On a subsequent run, it won't need to recompile boring_ssl. Inside Google, the build system reuses objects from prior runs of <i>other engineers</i> which vastly speeds up individual's compiliation. An open source developer does not benefit from this on their first compile of Tensorflow. They're starting out cold. Their subsequent compile runs are sped up, of course, but how often do you compile Tensorflow again? You generate the Python Wheel, rm -Rf the checked out repo, and carry on with your Python data analysis.</div><div><br /></div></div><div><br /></div><div>Another quirk: at the end, the bazel server is still running on that machine, and it shut down after many hours of disuse. This might be fine for Google engineers who will be compiling other software, soon. For them, the cost of keeping bazel up and running is small compared to the benefit from the pre-warmed caches and memory. I suspect independent open source developers are baffled why bazel is holding on to 400+Mb of RAM, hours after the compilation is done.</div><div><br /></div><div>The choice of bazel itself is interesting. Most open source software uses the 'make' tool, despite its numerous flaws. Bazel is an open source implementation of an internal Google build tool, so Tensorflow uses that. Even Android AOSP uses make, since bazel wasn't available in open-source back in 2008 when Android AOSP was released.</div><div><br /></div><h2 style="text-align: left;">Other systems</h2><div>Let's see how other systems manage this sort of complexity.</div><div><br /></div><div>PyTorch, by comparison offers a rich selection of choices. You select the build, the OS to run on, which package manager you want to use (Conda/Pip/...) whether you have CUDA installed, and if so, which version. It finally tells you exactly <a href="https://pytorch.org/get-started/locally/#start-locally">how to get PyTorch installed</a> on your system.</div><div> <a href="https://pytorch.org/get-started/locally/#start-locally"></a><div class="separator" style="clear: both; text-align: center;"><a href="https://pytorch.org/get-started/locally/#start-locally"></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIQU2dJHc58rpxzEGSBppHVYgEwDqc9keNdHlEqVh8EF-lnC6ae9k5pxK0CD5zalzJ1IORHP-hGzQwZr6tAPxJyhH5qrCfKa-wGTSEXOfYtJHqy28UDLEFPI5Y753UqDfNcjHOvw/" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="597" data-original-width="804" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIQU2dJHc58rpxzEGSBppHVYgEwDqc9keNdHlEqVh8EF-lnC6ae9k5pxK0CD5zalzJ1IORHP-hGzQwZr6tAPxJyhH5qrCfKa-wGTSEXOfYtJHqy28UDLEFPI5Y753UqDfNcjHOvw/s16000/Screenshot_2020-09-20_09-15-38.png" /></a></div><div>This raises a question: why can't Tensorflow be available as a pre-compiled binary in a lot more configurations? The Intel Math Kernel Library (MKL), for example, is <a href="https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libmkl-&searchon=names">available in 50 variants</a> packaged natively for Ubuntu: libmkl-<b>avx</b>, libmkl-<b>avx2</b>, libmkl-<b>avx512</b>, libmkl-<b>avx512-mic</b>, libmkl-<b>vml-avx</b>, ... These are all variants for specific Intel CPUs, to extract the maximum possible performance from each system. Tensorflow is similar: it is built to efficiently process compute-intensive workloads. Why isn't Tensorflow available in 50 different variants, targetting avx, no-avx, avx2, avx512, ...?</div><div>Here, I am guessing the choices are due to the Google-engineer/Open Source divide. At Google, most engineers run a specific kind of machine, and so the pre-compiled binaries target these workstations and similar CPUs on cloud compute farms. Most internal (and Google cloud) users don't deviate from these computing setups. So Tensorflow on a Core2Duo from 2006, or ARM32, or ARM64 isn't a high priority. This is a real lost opportunity, because compiling multiple targets can be automated. The real cost here is of maintenance. If you do provide Tensorflow on a Core2Duo or arm32, you are implicitly providing support for it.</div><div>The open source answer here would be to appoint a maintainer for that architecture. The <a href="https://www.kernel.org/doc/linux/MAINTAINERS">Macintosh PowerPC port of Linux</a> is still maintained by <span style="white-space: pre-wrap;">Benjamin Herrenschmidt, among others. He cares about that architecture, so he helps keep it up and running. The community will probably maintain a no-avx binary if you empower them.</span> </div><div><br /></div><div><br /></div><div> The Linux kernel is also an incredibly complex system. You are building an operating system kernel, which by definition is hardware architecture and device specific. Even in 2020, you can build the Linux kernel for machines as varied as PowerPC, ARM, MIPS, Intel x64, and Intel 386. Of course, you can choose between AVX support and not. The Linux kernel depends on very few external libraries, and is almost entirely self-contained. It compiles with make, and generates targets that work on many more architectures than Tensorflow. It has a huge configuration system, with many many choices. Most of the complexity is the skill and expertise in understanding the options and selecting them. You can always take an existing kernel configuration from a running system, and then run 'make menuconfig' to modify the specific options you want to change.</div><div><br /></div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNy0awqrAhbx47Rn0W4pSDzo0JgcPze70SHEI12xrDpLCn6TUoWWeuwtDC87WmKarGthflZSKlLa9OJuoV3P6DnK7WUCGwgZHNCfSPFpb3bAmMUI85Ca-cjtQ7MYANtiWZ1gHnuQ/" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="287" data-original-width="475" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNy0awqrAhbx47Rn0W4pSDzo0JgcPze70SHEI12xrDpLCn6TUoWWeuwtDC87WmKarGthflZSKlLa9OJuoV3P6DnK7WUCGwgZHNCfSPFpb3bAmMUI85Ca-cjtQ7MYANtiWZ1gHnuQ/s16000/Make_menuconfig.png" /></a></div><div><br /></div>The comparison might not be entirely fair, though. The Linux kernel has been in active development for many <u>decades</u>. It was always developed in a decentralized way, and therefore has perfected open source development development and release. The open source process has also been shaped by the quirks of the Linux kernel to the point where it is difficult to tell whether Linux influences open source or open source influences Linux.</div><div><br /></div><div> </div></div><h2 style="text-align: left;">Outcome</h2><div>It took a few hours of computation on the old machine, all four CPUs were busy for a whole day. But at the end, I have Tensorflow for Ubuntu 20.04 for x86_64 without <a href="https://en.wikipedia.org/wiki/Advanced_Vector_Extensions">AVX</a> support. I tried this out on a Celeron and a Core2 machine, and it works great. Tensorflow is perfect for the old machines where you can run model training for a few hours, turn the screen off, and leave it alone.</div><div><br /></div><div>Since I have the Wheel compiled for Python3, here it is if anyone needs <a href="https://www.dropbox.com/s/rnmn205f1skj139/tensorflow-2.3.0-cp38-cp38-linux_x86_64.whl?dl=0">Tensorflow 2.4 without AVX support for Ubuntu 20.04 and Python3</a>. If you need another version, find my email address and mail me.</div><div><p>Just for fun, I'd love to compile PyTorch from source as well. It seems to follow the open source paradigm closely: you install specific dependencies using yum/apt, and it uses those directly.</p><p><br /></p></div><h2 style="text-align: left;">Conclusion</h2><div>Tensorflow compilation was an interesting process to see from the outside. The compilation process is far more complicated due to the wealth of dependencies. While most <i>users</i> of Tensorflow are aware of the complexity of Tensorflow, a lot more complexity is visible when compiling the system. The choices here are motivated by some development practices at Google, and also make an interesting case study for large system design.</div><div> </div><p><b>Disclaimer</b>: I'm a Google employee, but these are my own opinions from the public Tensorflow project. I did not examine any Google confidential systems to arrive at these observations.</p>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-78943562274024423082020-08-09T20:41:00.005-07:002020-08-09T20:41:48.843-07:00Unable to view filesystems other than / using Docker on Linux?<p>I was running a Docker container on my home machine today (Linux, x86_64). Usually Docker is trouble-free and reliable, but today I ended up navigating a deep labyrinth. Since I couldn't find any good documentation on this, and to avoid this problem in the future, I wanted to document it.</p><p>The problem was that Docker refused to mount an external filesystem on the host as a directory on the image. I'll condense the problem to make it easy to understand. Here is the symptom:</p><p>Let's say /mnt has a filesystem that is mounted (an external disk, or a thumb drive). / is the root partition, and the only other physical partition. There are a few files on the thumb drive in /mnt/</p><p><span style="font-family: courier;">host$ ls /mnt</span></p><p><span style="font-family: courier;">fileA fileB directoryA/</span></p><p>The following command works file, and mounts /home/user/work-dir as /external</p><p><span style="font-family: courier;">host$ docker run -it -v ~/work-dir:/external busybox</span></p><p>/external in the docker image correctly shows the contents of /home/user/work-dir, as expected.</p><p><br /></p><p>But, if I point it to the external mount, the directory /external is empty in the docker container</p><p><span style="font-family: courier;">host$ docker run -it -v /mnt:/external busybox</span></p><div>You'd expect /external in the busybox image to contain fileA, fileB and directoryA. But they don't. It gets weirder.</div><div> If you export all of / as /external, then busybox can see everything in / <b>except</b> /mnt/</div><div><p><span style="font-family: courier;">host$ docker run -it -v /:/external busybox</span></p></div><div><div><p><span style="font-family: courier;"># ls /external/mnt</span></p><p>The output is empty, so /external/mnt/ exists, and is empty. /external/bin/ exists, and maps to /bin on the host machine, correctly, as expected.</p></div></div><div>I struggled with this for a long time. strace on docker clearly shows that it thinks /mnt doesn't exist, or is read only, even though it is definitely visible and writable in the host.</div><div><span style="font-family: courier;"><br /></span></div><div><span style="font-family: courier;">host$ date >> /mnt/out; cat /mnt/out</span></div><div><span style="font-family: courier;"><br /></span></div><div><span style="font-family: courier;">Sun 09 Aug 2020 04:26:16 PM PDT</span></div><div><p>So the directory structure is visible, writable, but never in a docker image. Doesn't help if you try all the commands as root. What gives?</p><p><br /></p><p>The problem was that my docker installation was from snap. Snaps have limited visibility of the remaining system. This might be fine for a snap containing a calculator app, but docker needs a lot to function.</p><p><span style="font-family: courier;">$ sudo snap remove docker; apt install docker-ce{,-cli}</span></p><p>When you do that, remember to log out because otherwise bash might have hashed the location of 'docker' to /snap/bin/docker, and it takes a logout to clear bash's state and realize that docker is now at /usr/bin/docker.</p><p><br /></p><p><a href="https://snapcraft.io/">Snaps</a> are a great idea, and I look forward to a time when they are functionally equivalent to packages installed through apt. For now, my experience with docker on snap makes me reluctant to use them. They're an additional layer of complexity to understand and debug, in a system that is already plenty complicated.</p></div>Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-18781155249226699832020-07-22T22:51:00.005-07:002020-07-22T22:51:49.081-07:00Playing flac files on Linux<div dir="ltr" style="text-align: left;" trbidi="on">
It used to be that you just ran flac files through flac123, like so:<br />
<span style="font-family: Courier New, Courier, monospace;">$ flac123 file.flac</span><br />
<br />
That doesn't work any more, and the flac commandline tool only contains a single binary called 'flac' that decodes, encodes, etc.<br />
<br />
Playing with 'mplayer' still works but it produces a lot of errors, and the audio level is low. The trick seems to be to call flac with the decode (-d) option, output to stdout (-c) and then pipe it to alsa player over stdin<br /><br />
<span style="font-family: Courier New, Courier, monospace;">$ flac -c -d /path/to/files/*flac | aplay</span><br />
<br />
<br /></div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-10257155189768011292020-07-18T09:54:00.000-07:002020-07-18T09:54:07.222-07:00Social trackers on banks: Etrade<div dir="ltr" style="text-align: left;" trbidi="on">
Etrade has a single advertiser: Wall Street on Demand.<br />
<br />
This is much better than other financial websites, though I'm not sure what Wall Street on Demand does.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijKb-eQGhYIlLOUL12vlh-irLAV9VCeED_D0lXvNNbXJwMnosth6Y15Ymnwe7oh8AtiWWhWbYJye-ZDhFctUFMaAdg7jKFRm7R32hnk8s7EUCEAgspyzxzNbjiksY8sxEUgg6WNw/s1600/etrade.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="519" data-original-width="581" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijKb-eQGhYIlLOUL12vlh-irLAV9VCeED_D0lXvNNbXJwMnosth6Y15Ymnwe7oh8AtiWWhWbYJye-ZDhFctUFMaAdg7jKFRm7R32hnk8s7EUCEAgspyzxzNbjiksY8sxEUgg6WNw/s1600/etrade.png" /></a></div>
<br /></div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-26503067915839734252020-07-18T09:41:00.000-07:002020-07-18T09:41:03.711-07:00Social tracker on banks: Schwab<div dir="ltr" style="text-align: left;" trbidi="on">
Charles Schwab only has trackers from Tealium, Confirmit and SOASTA mPulse.<br />
<br />
The site works well without it, and I feel a lot safer knowing that these companies are not snooping on my online financial activity.<br />
<br />
It is never a good sign when you have to search online for these company names. They know a lot more about you than you know about them.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimT5QZl1fBnmNJZT9BfC6sjCK8ONKPj-3QPxiBRkKWWS3-zcSBb-O4qtlw1y6QDr8g35-s-RQIqZoXGFiruB6cu6Y7LjAMRd7FXarYcgZscwTGb9Oq-BHZC8A4gnH6TDtHY9X8Ww/s1600/schwab.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="517" data-original-width="579" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimT5QZl1fBnmNJZT9BfC6sjCK8ONKPj-3QPxiBRkKWWS3-zcSBb-O4qtlw1y6QDr8g35-s-RQIqZoXGFiruB6cu6Y7LjAMRd7FXarYcgZscwTGb9Oq-BHZC8A4gnH6TDtHY9X8Ww/s1600/schwab.png" /></a></div>
</div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-65225366003429017302020-07-18T09:17:00.001-07:002020-07-18T09:17:20.353-07:00Social trackers on banks: Vanguard<div dir="ltr" style="text-align: left;" trbidi="on">
Vanguard is much better on their web interface. You only have trackers from Adobe Audience Manager, and Doubleclick.<br />
<br />
There are site analytics from two companies. It is best not to send this data to third-parties like Adobe or AppDynamics, since I don't know what these companies are tracking. They almost certainly have my IP address, time of day, browser and OS version, do they have my bank account number and other details as well?<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvkydFtdsNGEWHU7azIQY6J8HgTzlfeuxCYV28Z1VweSHldDhQtzc7_Ke2_5tQ_6b-f2XRMvtoS-chYIMF9MhClZ8cqCh0Qw8QDKcsT3D_OXNarVsXShEgvc5TvYAfxkzs9f2MWw/s1600/vanguard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="516" data-original-width="581" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvkydFtdsNGEWHU7azIQY6J8HgTzlfeuxCYV28Z1VweSHldDhQtzc7_Ke2_5tQ_6b-f2XRMvtoS-chYIMF9MhClZ8cqCh0Qw8QDKcsT3D_OXNarVsXShEgvc5TvYAfxkzs9f2MWw/s1600/vanguard.png" /></a></div>
</div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-26658506402158587742020-07-18T09:06:00.002-07:002020-07-18T09:14:30.652-07:00Social trackers on banks: Ally bank<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: left;">
There are 12 trackers on ally.com that have nothing to do with my relation with ally.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
These are advertisers, trackers by Snapchat, Adobe, Bing, Pinterest, Qualtrics and Facebook.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The only legitimate tracker that deserves to be there is LivePerson, for customer service. Even that should only start when I have an actual customer support interaction rather than being aware of my page interactions all the time.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
What data are these trackers collecting, and why are they required on a bank website?</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQoIF8MqJvZB0c32bD5L3gL1qheWRzvb5Pk5lToQJY_-KeVONvCpL__s92cMNFyE9qj7iWeHXjE4_PpZX-vKNfwSrSU0JffzzdZFn5AlU3ahcaE7fPcVrTnLYfhfZpM3q5cla1mg/s1600/ally-trackers-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="516" data-original-width="582" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQoIF8MqJvZB0c32bD5L3gL1qheWRzvb5Pk5lToQJY_-KeVONvCpL__s92cMNFyE9qj7iWeHXjE4_PpZX-vKNfwSrSU0JffzzdZFn5AlU3ahcaE7fPcVrTnLYfhfZpM3q5cla1mg/s1600/ally-trackers-1.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8lwcRs23dLosc7BHWqUGyWsJsNaI_fCyjzUXgOmE-AE32lKPnx4UQ_f3miYEGAw5N3MbA98-f9n0K4a5n-4swBH7mWBGacaNHXN-LMJC1C3DpCJKC3Nv6D4QwAFwaluNMhuOwBA/s1600/ally-trackers-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="404" data-original-width="345" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8lwcRs23dLosc7BHWqUGyWsJsNaI_fCyjzUXgOmE-AE32lKPnx4UQ_f3miYEGAw5N3MbA98-f9n0K4a5n-4swBH7mWBGacaNHXN-LMJC1C3DpCJKC3Nv6D4QwAFwaluNMhuOwBA/s1600/ally-trackers-2.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbcBD6qUafSFeNiVt68jjUYUS8Y60Zi7adnypO9Kiu6O-D0sJoXqfdq8wAM9I34poPKZykwKhOec2EtP0sg7wQ7G9IN9uCO_noQfxVBtFtIAG5T8RuIrW13K3zR6sNLcxrBxiW3w/s1600/ally-trackers-3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="383" data-original-width="345" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbcBD6qUafSFeNiVt68jjUYUS8Y60Zi7adnypO9Kiu6O-D0sJoXqfdq8wAM9I34poPKZykwKhOec2EtP0sg7wQ7G9IN9uCO_noQfxVBtFtIAG5T8RuIrW13K3zR6sNLcxrBxiW3w/s1600/ally-trackers-3.png" /></a></div>
<br />
<div class="separator" style="clear: both;">
(Data obtained from <a href="https://chrome.google.com/webstore/detail/ghostery-%E2%80%93-privacy-ad-blo/mlomiejdfkolichcflejclcbmpeaniij/related?hl=en">Ghostery, the privacy extension</a> for Google Chrome)</div>
<div class="separator" style="clear: both;">
<br /></div>
</div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-26012775695550901142020-06-17T16:59:00.002-07:002022-01-22T19:47:57.550-08:00Game review: Human Resource Machine<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://tomorrowcorporation.com/humanresourcemachine">Human Resource Machine</a> is a game about writing programs that make a little person do stuff. Every level has a new task, and the initial tasks start out very easy, and then the difficulty builds up. The final level has you writing a sorting routine in a programming language.<br />
<br />
The programming language is a bit like assembly, for a computer with a single register. The primitives are similar to assembly too, 'jump if zero', for example, along with indirect addressing in later levels. If you know assembly, you might have to adjust to the style: you cannot increment a register, only memory locations.<br />
<br />
Leaving the technicalities aside, it has beautiful music, it has a compelling storyline, and the puzzles are engaging and fun. The levels in the end get difficult, and you might lose interest in solving them.<br />
The levels are a beauty, it works great on Linux.<br />
<br />
The most fascinating thing for me was to watch its adoption on Steam. The game was made five years ago, and about 12% of the players have finished the main plot. This is incredible. That means one in ten people who bought the game have solved the toughest challenge in the main story. That is not the toughest challenge, which is prime factorization. I still haven't solved that one.<br />
<br />
It is made by the same folks who brought us <a href="https://www.eggwall.com/2012/05/impressive-gaming-world-of-goo.html">World of Goo</a> and <a href="https://www.eggwall.com/2013/12/game-review-little-inferno.html">Little Inferno</a>. As is the case, their plot has a sub-text that continues on this game as well.<br />
<br />
Buy the game, you won't regret it. Here's my solution of the last level: and you can see how the main game works. You get letters or numbers at the IN line on the left, your man processes them, and then puts them in the OUT line on the right. The floor is numbered, and that's the main memory. The program you wrote is along the right side of the screen. In the video, the zero-terminated lists are being sorted using a naive insertion sort. And once they are fully sorted, they are output (smallest to largest) in the OUT line. Then the little man sets up the memory and inputs more lists.<br />
<br />
<div style="position:relative;padding-bottom:56.25%;height:0;overflow:hidden;"> <iframe style="width:100%;height:100%;position:absolute;left:0px;top:0px;overflow:hidden" frameborder="0" type="text/html" src="https://www.dailymotion.com/embed/video/k4gWL6dy2kjyDnxyWtX" width="100%" height="100%" allowfullscreen > </iframe> </div>
<br /></div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.comtag:blogger.com,1999:blog-19163975.post-14778397740264165962020-05-22T13:55:00.005-07:002020-05-22T14:03:24.017-07:00Thread-local stack on Linux<div dir="ltr" style="text-align: left;" trbidi="on">
Threads on POSIX systems like Linux share an address space, but have their own registers and execution state like their own stacks. We also know that the stack is mapped in the same virtual memory space.<br />
<br />
The illusion of thread-local stacks is achieved by having the stacks of the different threads start at different locations in virtual memory.<br />
<br />
<a href="https://github.com/youngelf/linux-innards/blob/master/threadStorage.c">threadStorage.c</a> prints the location of the stack.<br />
<br />
Here are numbers from a few architectures:<br />
<b>x86-64</b>: (kernel v4.15)<br />
$ ./threadStorage<br />
one: Address of first variable 0x7fc7bbc90edc<br />
two: Address of first variable 0x7fc7bb48fedc<br />
main: Address of first variable 0x7ffe79c59e1c<br />
three: Address of first variable 0x7fc7bac8eedc<br />
<div>
<br /></div>
<div>
The difference between one and two, and two and three is 0x8392704</div>
<br />
The stack of the main thread starts at 0x7ffe79c59e1c, which is much higher than the stack of the first thread. Since most processes are single-threaded, they get much larger stack frames.<br />
<br />
Numbers from other architectures I had lying around:<br />
<br />
<b>arm32</b>: (kernel v4.19.66)<br />
$ ./threadStorage<br />
one: Address of first variable 0x76e3ee68<br />
two: Address of first variable 0x7663de68<br />
main: Address of first variable 0x7e9c6c30<br />
three: Address of first variable 0x75e3ce68<br />
<div>
<br /></div>
<div>
<b>arm64</b>: (kernel v4.15)</div>
<div>
<div>
$ ./threadStorage </div>
<div>
one: Address of first variable 0xffffab10a9dc</div>
<div>
two: Address of first variable 0xffffaa9099dc</div>
<div>
main: Address of first variable 0xffffe26742ec</div>
<div>
three: Address of first variable 0xffffaa1089dc</div>
</div>
<div>
<br /></div>
<div>
<b>MIPS</b>: (kernel v3.18.140)</div>
<div>
<div>
$ ./threadStorage</div>
<div>
main: Address of first variable 0x7fe384a4</div>
<div>
three: Address of first variable 0x7603ef3c</div>
<div>
two: Address of first variable 0x7683ff3c</div>
<div>
one: Address of first variable 0x77040f3c</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
In all these systems, the addresses are randomized, so successive runs of the same program produce slightly different results. The offsets between the locations in successive runs is constant.</div>
<div>
<br /></div>
The repository is here: <a href="https://github.com/youngelf/linux-innards">https://github.com/youngelf/linux-innards</a><br />
<br /></div>
Vikram Aggarwalhttp://www.blogger.com/profile/14128872612935211717noreply@blogger.com