Wednesday, June 02, 2021

Tensorflow 2.5 without AVX

 I was playing with Tensorflow, needed a new version, and realized that a Tensorflow release without AVX still doesn't exist. Since my prior post on Tensorflow without AVX had been beneficial to a few people, I decided to compile Tensorflow 2.5 for Linux, and put up the wheel here for others' benefit.

Tensorflow 2.5 for Linux without AVX support (155 Megabytes)

and the prior link

Tensorflow 2.3 for Linux without AVX support (126 Megabytes)


Download the file, and then run

$ pip3 install -U <filename.whl>

Friday, March 12, 2021

Cloud of DOS machines

tl:dr;  ssh guestdos@dos.eggwall.com


Retro computing is in. New computers and new systems are always fun. But to really appreciate the arc of history, use something old. Something ancient. Something you never really learned.

To help you get your retro computing fix, here's a bank of DOS machines. Rather than MSDOS (copyrighted, etc), I've got FreeDOS, an open-source implementation of DOS. There is the editor 'edit' and 'edlin'. 'foxcalc' is a sweet calculator. 'help' will tell you what to do. For the Assembly geeks, there is always 'debug'.

This is running on my custom cloud of DOS machines. Connect to your instance today. You can use either ssh or telnet:

$ ssh guestdos@dos.eggwall.com


Telnet works too:

$ telnet dos.eggwall.com

Username: 'guestdos', no password.


When you are done, type 'halt' and travel back to the present time.


DOS


Look around the file system. There's compilers for C, Pascal, an IDE, emacs, vi, BASIC, and some games. Using a system this old gives you an appreciation for systems today. You miss the many conveniences you take for granted.  You are also closer to the machine, you can modify arbitrary memory addresses. Nothing to get in your way.

Powered by the FreeDOS project, and magic.


Tuesday, March 09, 2021

Book Review: UNIX and Linux System Administration Handbook, Evi Nemeth et al

 tl:dr; If you are a Linux/BSD user, this book will help you understand your system.


There are Stack Overflow answers, random blogs and tech articles. You can find man pages. What they lack is a comprehensive view of the full system, with historical background, a theme tying the topics together. They lack clarity, depth or accuracy. Nearly always, they lack humor.


UNIX and Linux System Administration Handbook is a very broad overview of all aspects of Unix system maintenance and upkeep: going from installation, package installation, and routine chores. It has in-depth coverage of critical sub-systems like networking and disks. These are covered over many chapters, devoted to hardware, services, best practices. There is deep coverage of user management, security issues for sites, authentication, logging. It has the obligatory chapter on shell programming, with bash & Python.  There's information on config management tools like Ansible. It has much more than a system administrator would need through their first year. It is a large book and you will find yourself revisiting specific sections. It helps to keep an electronic copy for quick searches. The index and the table of contents are ideal for an admin with a paper copy.


The book gives you a superficial overview of virtualization, cloud systems, containers (Docker), and container orchestration tools like Kubernetes, Docker Swarm, Apache Mesos or Marathon. These sections are light: it gives you a background, why these systems exist and how to think about these systems. You can start a journey there if you want. These chapters were added later, and the coverage is satisfactory. Users need to have a rough idea of these, but each of those topics deserve a book-level treatment. The authors have kindly provided references to chase for the interested readers.


I've been a long-time Linux user, and I found the book's treatment of systemd level-headed and valuable. It provided a great preface with motivations for systemd, how it has evolved, and how best to use systemd to keep a system humming along and keep the system manageable.


The book was funny in all the right parts, without overdoing the humor or trying too hard.


It was published in 2017, but most of the sections are quite current. The book covers major Linux distributions (CentOS/RedHat and Debian/Ubuntu) and FreeBSD. The FreeBSD sections were great, especially alongside Linux so I had a Rosetta stone. The book's primary author, Evi Nemeth, is one of my favorite technical authors. Sadly, she passed away before this edition of the book was published, and I commend the current authors for carrying on her tradition, and their work on this edition.


In 2021, the diversity in Unix systems has shrunk incredibly. Solaris is history; FreeBSD is relegated to core adherents and the occasional technical Mac OS user. Cloud systems are gathering behind Linux instances. The ease of creating a Linux instance means that more users will find themselves responsible for a Linux machine. Unix used to mean SunOS: something formal, expensive. Something meriting a real education. Now, Linux powers everything from your phone to your Top500 supercomputer. A cloud instance can be yours for dollars a month. The power is still there, if you can invest time in learning it.


Veteran system administrators will enjoy filling gaps in their knowledge. The casual user will gain a deeper understanding of their system with this light, engaging, fun read.

Sunday, February 14, 2021

Unicode URLs

tl:dr; Get URLs that cannot be verbally communicated.


URLs can only be in English characters (really, Roman characters) and a select few special characters like -.  So how is はじめよう.みんな a full domain name, even though it is written in Japanese (Hiragana script) and a dot?


The answer lies in character encodings, and translation done by the browser.

There is an encoding of characters called Unicode. That is what allows English, Japanese, Hindi, and other characters to be written on a single document.  URLs are Internet "addresses", the stuff that shows up in the address bar of your browser. These URLs can only be coded in ascii characters (a-z, a-Z, 0-9 etc). Due to their long history, URLs cannot be specified in Unicode. But creative minds have been hard at work here. There is a coding for Unicode that uses only the characters that are allowed in URLs. This encoding is called punycode.  Punycode URLs start out with 'xn--', but the browser transparently renders the unreadable combination of ascii characters into their corresponding unicode characters.


So in the https://はじめよう.みん example above, はじめよう is xn--p8j9a0d9c9a and is みん is xn--q9jyb4c. So the full URL is https://xn--p8j9a0d9c9a.xn--q9jyb4c/  If you can remember all those characters, you can type them out by hand. Your browser, detecting the punycode beginnings 'xn--' will decode the characters xn--p8j9a0d9c9a into はじめよう and xn--q9jyb4c into みん.

Try this translation yourself using online punycode converters.


This mechanism works for Top Level Domains (the .com in mail.google.com), domain names (the google in maill.google.com), or sub-domains (mail in mail.google.com)


What do you do with it? You can create URLs that look short but can only be recreated if the person can read/write the language. Here's an example: विक्रम.eggwall.com. This is a URL that can be clicked on but can only be verbally communicated by Hindi speakers.


You can mix-and-match different scripts. Here's an example with different scripts that I cannot even read. In the screenshot here, you have a URL. Try typing that address out.


There are some Top Level Domains (TLDs) that don't allow for punycode domain names. .fun, for instance doesn't allow punycode URLs. Try registering विक्रम.fun, for example. But you can always register a subdomain with punycode. So can have विक्रम.circuits.fun.

Tuesday, January 19, 2021

FreeBSD, the alternative

 I like FreeBSD. Something about a barebones, simple UNIX that is clean at the core. Not particularly a pragmatic choice, I'll gladly admit. For most needs, a Linux installation is both simpler and more capable. More people are familiar with Linux, so this is a niche world.

But FreeBSD is a hobby, and a particularly gratifying hobby.


The simplicity of FreeBSD is a huge draw, you can get very far with an ancient Intel machine and 1 GB of RAM. The source code is a pleasure to read, the system is easy to modify, and there is relatively little of it on a base install. Full system upgrades are clean and easy, the ports system is elegant though often impractical.

For most people, the easiest way to run FreeBSD would be to use a virtual image, and boot it up using qemu (Linux) or Hyper-V (Windows). Both work very well, both support networking, which is really what you're looking for. Face it, you're not booting into FreeBSD for its giant catalog of games or media utilities.

Get the VM images from here. qcow2 works for QEMU, vhd works for Hyper-V, and vmdk works for VMWare. 

For QEMU, I use the following commandline:

qemu-system-x86_64 \
        -m 2048 \
        -enable-kvm \
        -hda bsd-with-jail.img \
        -net user,hostfwd=tcp::10023-:22 \
        -net nic \
        -curses

That specifies:

  • (-m 2048) 2048 MB RAM
  • (-enable-kvm) use the Linux KVM hypervisor
  • (-hda bsd-with-jail.img) the disk bsd-with-jail.img as the first hard disk 
  • (-net nic)  a full Network Interface Card, make your FreeBSD setup trivial as it can get a network interface called em0
  • (-net user, hostfwd=tcp::10023-:22) network translation is user-level, so no root permissions required and forward port 10023 on the host to port 22 on the guest. This allows you to ssh to the FreeBSD guest by ssh'ing port 10023 on the host.
  • (-curses) Use the text-based interface using libncurses rather than a graphical interface. Great for running on cloud instances, or through GNU screen so you can leave it on forever.

Other ports can be forwarded for running web servers, dns servers, ...


The same setup can be replicated on Hyper-V. For network, you can use the default switch. Port forwarding is more complicated, and you could achieve it with ssh tunnels on the Windows host or in the FreeBSD guest.


FreeBSD runs fine with 1GB of RAM, and is a great system to learn the fundamentals of computing. You can play with Dtrace, create constrained environments called Jails, or modify the behavior of a clean UNIX kernel.  A lot of this knowledge translates to Mac OS, as the OS X kernel is a hybrid of FreeBSD and Mach, and the userland is very similar to FreeBSD.

You can easily run a dozen jails on 1G of RAM, depending on what you do with them.  Each jail is separated from the other and can be a low-cost, throwaway computing environment. Similar to Docker, or Linux namespaces, root in a jail is safe, and cannot damage the host FreeBSD system.

Set up your own UNIX sandbox today!




Thursday, January 07, 2021

Mac OS Kernel (XNU) source, on github

 Few people know that Apple releases the source code for many components of their MacOS software. Many are surprised when they hear it.

Apple has a full site for their open source releases. Their most recent Mac OS release is for 11.0.1 (Big Sur). Not all source is released. There are the usual UNIX utilities: Perl, Python, Bash, and userland tools that make a functioning system. Functioning, but not always updated, as Apple doesn't always track the latest versions. They still ship Python 2.5, and other tools can be many years old. This is why you need to get a functional, bugfixed UNIX userland with homebrew or other tools.


The most interesting source there would probably be the kernel, called XNU. It is a combination of Mach, a microkernel, and BSD, a monolith. There's some Solaris code there, some Joyent code, some Sun code, NeXT, UC Berkeley, Carnegie Mellon code, and of course, Apple code.

Apple does have an official github profile, which also hosts xnu. But it was last updated two years ago, and stands at version 4903, while the latest version is 7195.


Their open source browser is passable but doesn't show every file properly. I find it much easier to read through code using Emacs so I pulled it into a github repository if anyone wants to clone it or browse it online.



Image credit: Apple Inc