This & ThatA bit of everything, mostly about computer stuff and with all the usual caveatsZola2021-02-07T03:18:18+01:00https://zbrox.com/atom.xmlBuild SciPy on Apple Silicon2021-02-07T03:18:18+01:002021-02-07T03:18:18+01:00https://zbrox.com/posts/scipy-on-apple-silicon/<p>Have you managed to get your hands on one of Apple's new machines with their very own Apple SoC? And then you needed to install <code>scipy</code> in some freshly cloned Python project?</p>
<p>Yeah, I went through that and <code>scipy</code> just not budging, failing dramatically during its build process. Here's what I did while waiting for an official native whl package. </p>
<p>Make sure you're using Python 3.9.1 and then just try this out.</p>
<pre style="background-color:#2b303b;">
<code class="language-sh" data-lang="sh"><span style="color:#bf616a;">brew</span><span style="color:#c0c5ce;"> install openblas
</span><span style="color:#bf616a;">pip</span><span style="color:#c0c5ce;"> install</span><span style="color:#bf616a;"> --upgrade</span><span style="color:#c0c5ce;"> pip setuptools wheel
</span><span style="color:#bf616a;">pip</span><span style="color:#c0c5ce;"> install cython
</span><span style="color:#bf616a;">OPENBLAS</span><span style="color:#c0c5ce;">=$</span><span style="color:#a3be8c;">(</span><span style="color:#bf616a;">brew --prefix</span><span style="color:#a3be8c;"> openblas) </span><span style="color:#bf616a;">pip</span><span style="color:#c0c5ce;"> install scipy</span><span style="color:#bf616a;"> --no-binary</span><span style="color:#c0c5ce;"> :all:</span><span style="color:#bf616a;"> --no-use-pep517
</span></code></pre>
<p>Waiting time may vary. Good luck!</p>
Creating thin Docker containers for Rust binaries2020-06-24T18:21:13+02:002020-06-24T18:21:13+02:00https://zbrox.com/posts/thin-docker-containers-for-rust-binaries/<p>Sometimes you need to distribute some tools as Docker containers, for example when you are running some long-running processes in a Kubernetes cluster. Or whatever else, I don't judge. Since nowadays I write all CLI tooling in Rust because it's just pure joy, I decided I need to make myself a small <code>Dockerfile</code> template for building thin images that won't take much disk space and are fast to download and run with only the bare minimum.</p>
<p>To ensure we're gonna end up with a really small Docker image, we need to make it a <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-stage build</a> one. My original plan seemed easy enough. Compile a static binary using the Rust Linux musl toolchain in the first stage. Then move only the binary to a <a href="https://hub.docker.com/_/scratch/">scratch image</a> in the second stage. The scratch image is the most minimal Docker image you can use, with no extra layers, and nothing else in it really. </p>
<p>There are official <a href="https://hub.docker.com/_/rust">Rust Docker images</a>, so that's what I chose as a build stage base. They include <code>rustup</code>, so I just installed the <code>musl</code> toolchain and <code>musl-tools</code>, and I thought I was ready to go. However, when I tried to build the image, the build step in the first phase panicked with the following error:</p>
<pre style="background-color:#2b303b;">
<code><span style="color:#c0c5ce;">thread 'main' panicked at '
Could not find directory of OpenSSL installation, and this `-sys` crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set the `OPENSSL_DIR` environment variable for the
compilation process.
Make sure you also have the development packages of openssl installed.
For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.
</span></code></pre>
<p>Of course, I thought, I'm doing some networking, nowadays encrypted connections are a must. So let's just install <code>libssl-dev</code>, as the error suggests. Close, but no cigar. Musl and openssl-sys-extra didn't play well since the SSL package in Ubuntu is linked against <code>glibc</code>. Makes sense. What's next? I didn't really feel like compiling OpenSSL myself.</p>
<p>With a lit bit of searching, I found the great <a href="https://github.com/clux/muslrust">clux/muslrust</a> Docker image. It's based on <code>Ubuntu Xenial</code> and conveniently has built a <a href="https://github.com/clux/muslrust#c-libraries">lot of things with musl-gcc</a>, like OpenSSL. A big thanks for this effort to <a href="https://github.com/clux">clux, the creator</a>!</p>
<p>After replacing the <code>rust</code> image with <code>clux/muslrust</code> the crate compilation went fine and I thought that's that. But since I kinda rushed it, I totally forgot the small tiny detail of CA certificates. The Docker image was fine, however, the binary was basically not able to do any HTTPS requests since they would error out with certificate errors. Remember, the scratch image is containing basically just the binary.</p>
<p>So after a couple of extra steps to install the CA certificates Ubuntu package and then copy over the certificates from the first to the second stage, I was done. And here's the final result.</p>
<pre style="background-color:#2b303b;">
<code class="language-Dockerfile" data-lang="Dockerfile"><span style="color:#b48ead;">FROM</span><span style="color:#c0c5ce;"> clux/muslrust </span><span style="color:#b48ead;">AS </span><span style="color:#bf616a;">build
</span><span style="color:#b48ead;">WORKDIR </span><span style="color:#c0c5ce;">/usr/src
</span><span style="color:#65737e;"># Update CA Certificates
</span><span style="color:#b48ead;">RUN </span><span style="color:#c0c5ce;">apt update -y && apt install -y ca-certificates
</span><span style="color:#b48ead;">RUN </span><span style="color:#c0c5ce;">update-ca-certificates
</span><span style="color:#65737e;"># Build dependencies and rely on cache if Cargo.toml
# or Cargo.lock haven't changed
</span><span style="color:#b48ead;">RUN </span><span style="color:#c0c5ce;">USER=root cargo new <YOUR_CRATE>
</span><span style="color:#b48ead;">WORKDIR </span><span style="color:#c0c5ce;">/usr/src/<YOUR_CRATE>
</span><span style="color:#b48ead;">COPY</span><span style="color:#c0c5ce;"> Cargo.toml Cargo.lock ./
</span><span style="color:#b48ead;">RUN </span><span style="color:#c0c5ce;">cargo build --target x86_64-unknown-linux-musl --release
</span><span style="color:#65737e;"># Copy the source and build the application.
</span><span style="color:#b48ead;">COPY</span><span style="color:#c0c5ce;"> src ./src
</span><span style="color:#b48ead;">RUN </span><span style="color:#c0c5ce;">cargo install --target x86_64-unknown-linux-musl --path .
</span><span style="color:#65737e;"># Second stage
</span><span style="color:#b48ead;">FROM</span><span style="color:#c0c5ce;"> scratch
</span><span style="color:#65737e;"># Copy the CA certificates
</span><span style="color:#b48ead;">COPY</span><span style="color:#c0c5ce;"> --from=</span><span style="color:#bf616a;">build</span><span style="color:#c0c5ce;"> /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
</span><span style="color:#65737e;"># Copy the statically-linked binary to the second stage
</span><span style="color:#b48ead;">COPY</span><span style="color:#c0c5ce;"> --from=</span><span style="color:#bf616a;">build</span><span style="color:#c0c5ce;"> /root/.cargo/bin/<YOUR_CRATE> .
</span><span style="color:#b48ead;">USER </span><span style="color:#c0c5ce;">1000
CMD ["</span><span style="color:#a3be8c;">./<YOUR_CRATE></span><span style="color:#c0c5ce;">"]
</span></code></pre>
<p>This basic <code>Dockerfile</code> template is available as a <a href="https://gist.github.com/zbrox/716d75c9c8d27016f2f528335617ceb4">GitHub Gist</a>. I use it in a couple of places, one of which is a public crate called <a href="https://crates.io/crates/rmq_monitor">rmq_monitor</a>. The image size turns out to be a tad more than your binary, in the case of <a href="https://hub.docker.com/layers/zbrox/rmq_monitor/0.2.5/images/sha256-32ef6ed0329066d3ad83949c0581782eec5aeed0ebdbcd80707ce20a7208fceb?context=explore">the rmq_monitor image</a> it's only <strong>7Mb</strong> uncompressed.</p>
Nix package manager on macOS 10.15 a.k.a Catalina2020-05-15T20:03:13+02:002020-05-15T20:03:13+02:00https://zbrox.com/posts/nix-on-macos-catalina/<p>Trying to install Nix on macOS Catalina is a tiny bit more difficult than I imagined. I had completely forgotten that Catalina doesn't allow you to touch the root filesystem and the Nix store is at the usual place - <code>/nix</code>. When trying to install Nix, you'll get an error that <code>/nix</code> is not writeable. The way around that is to create a so-called synthetic firmlink (a feature of APFS) which would allow us to mount volumes on the root filesystem. Those volumes are created only on boot so after we define one, we, unfortunately, have to restart the computer.</p>
<p>How do we create those synthetic firmlinks? If you don't already have the <code>/etc/synthetic.conf</code> file, create it with the appropriate permissions.</p>
<pre style="background-color:#2b303b;">
<code class="language-sh" data-lang="sh"><span style="color:#bf616a;">sudo</span><span style="color:#c0c5ce;"> touch /etc/synthetic.conf
</span><span style="color:#bf616a;">sudo</span><span style="color:#c0c5ce;"> chmod 644 /etc/synthetic.conf
</span></code></pre>
<p>In that file add a line with just the text <code>nix</code> in there which tells the system to have a firmlink at <code>/nix</code> on boot. We will later define what's gonna be mounted there. But first, time to restart the computer. Yay!</p>
<p>And here's how to do that. If we assume you have not partitioned your Mac's disk or did something other than just installed macOS, so <code>disk1</code> is your main drive. First, we add an APFS volume with the <code>/nix</code> mount point with a volume name of <code>Nix</code>. The volume name you can change to whatever you want.</p>
<pre style="background-color:#2b303b;">
<code class="language-sh" data-lang="sh"><span style="color:#bf616a;">sudo</span><span style="color:#c0c5ce;"> diskutil apfs addVolume disk1 APFSX Nix</span><span style="color:#bf616a;"> -mountpoint</span><span style="color:#c0c5ce;"> /nix
</span></code></pre>
<p>Then we need to do a couple of more things like enable ownership on the volume (from the man pages of <code>diskutil</code>: <strong>When ownership is enabled, the Owner and Group ID settings that exist on the disk are taken into account for determining access, and exact settings are written to the disk as FSOs are created.</strong>), hide the volume from appearing on the Desktop, and add an <code>fstab</code> entry so it automounts on restart.</p>
<pre style="background-color:#2b303b;">
<code class="language-sh" data-lang="sh"><span style="color:#bf616a;">sudo</span><span style="color:#c0c5ce;"> diskutil enableOwnership /nix </span><span style="color:#65737e;"># enables ownership
</span><span style="color:#c0c5ce;">sudo chflags hidden /nix </span><span style="color:#65737e;"># hides the volume from the Desktop, still shows up on the Finder sidebar
</span><span style="color:#c0c5ce;">echo "</span><span style="color:#a3be8c;">LABEL=Nix /nix apfs rw</span><span style="color:#c0c5ce;">" | </span><span style="color:#bf616a;">sudo</span><span style="color:#c0c5ce;"> tee</span><span style="color:#bf616a;"> -a</span><span style="color:#c0c5ce;"> /etc/fstab </span><span style="color:#65737e;"># add an entry to fstab to automount
</span></code></pre>
<p>After this you can install Nix the usual way and everything should be fine. Ta-da!</p>
Displaying the computed and in use tsconfig2019-04-03T09:29:00+01:002019-04-03T09:29:00+01:00https://zbrox.com/posts/tsconfig-show/<p>Usually, the configuration for the TypeScript compiler is pretty light. In some weird situations when things are not going as expected it is nice to be able to view what is the full <code>tsconfig.json</code> with all the default and computed properties in it. Since <a href="https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-2.html#the-new---showconfig-flag">TypeScript 3.2</a> you can
do that by passing the <code>--showConfig</code> flag.</p>
Getting rust-embedded/cross running on macOS2019-03-30T12:25:00+01:002019-03-30T12:25:00+01:00https://zbrox.com/posts/macos-rust-cross/<p>One of the great promises and benefits of Rust is that you can easily write code that cross-compiles to various different targets. Not only that but the ecosystem is constantly evolving and trying to make this process
as painless as possible. <a href="http://rustup.rs">Rustup</a> makes <a href="https://blog.rust-lang.org/2016/05/13/rustup.html">managing different target toolchains</a> a breeze. However, if you don't want to pollute your environment with all the different dependencies of those target toolchains you can try <a href="https://github.com/rust-embedded/cross">cross</a>. </p>
<span id="continue-reading"></span>
<p>Simply put <code>cross</code> uses Docker containers to build your project. It mimics cargo's CLI arguments as much as possible but instead of using the <code>rustup</code> toolchains for a given target it spins up a Docker container. It then builds your project in it and spins down the container. To put it even simpler - it's great!</p>
<p>Initially, I had missed one important disclaimer. I got super confused when it wasn't working on macOS. As of this writing, the current version of <code>cross</code> (<em>0.1.14</em>) is only working for Linux hosts. On macOS, it will just act as a proxy and let cargo do its usual build process with a different <code>--target</code> argument.</p>
<p>Over the last couple of years, Docker support for macOS has been getting more solid. The kind people of the <a href="https://github.com/rust-embedded/wg#the-tools-team">Rust Tools</a> team have actually made <code>cross</code> work with macOS but just haven't gotten around to making a release yet.</p>
<p>The solution until the next release is simple. Instead of doing</p>
<pre style="background-color:#2b303b;">
<code class="language-sh" data-lang="sh"><span style="color:#bf616a;">cargo</span><span style="color:#c0c5ce;"> install cross
</span></code></pre>
<p>Just do</p>
<pre style="background-color:#2b303b;">
<code class="language-sh" data-lang="sh"><span style="color:#bf616a;">cargo</span><span style="color:#c0c5ce;"> install</span><span style="color:#bf616a;"> --git</span><span style="color:#c0c5ce;"> https://github.com/rust-embedded/cross
</span></code></pre>Opening words2019-03-29T11:37:00+01:002019-03-29T11:37:00+01:00https://zbrox.com/posts/beginning/<p>This place is mostly for me but hopefully, it might help some other people here and there. Here I will store things I stumble on through my programming experience and the occasional mild rants.
Let's see where this goes!</p>