Category: web development

29 Articles
miscellaneous, plotters, web development

I just did my yearly X,Y Coordinates stint with local 5th graders. It’s the 5th time I do it and this year is notable in that I didn’t write anything I wanted to fix for next year. Every time previous I came out thinking I needed to fix a bug, prevent a confusion, or improve something or other. This time it looks like the formula has been refined to its optimum.

Kids will definitely push the limits, and I love that relentless will to push, it’s identical in nature to IT security curiosity. 1 kid wrote some code and then copy/pasted it a bunch of time, no problem there’s an upper limit on instructions. Another tried to hog all the squares by having them draw just a single dot, no problem there’s a cool-off timer that prevents you from blasting through squares. I haven’t had to use it but I also have a censorship mechanism :), I can scribble over any square and the machines will prioritize it.

So this year went really well, I think I can say with confidence now that the magic operates every time, this isn’t just luck with a good batch of kids or other. Every time we launch into “coding”, there’s a moment of sheer teeth grinding where I think it’s going to be a disaster. And every time they are all extremely motivated by the idea of controlling the machines when they hit “submit”, and so they all pull through and help each other out. Once one of them has gotten the machines moving, there’s a real frenzy to figure things out, and then their next drawings get more and more sophisticated. 5th grade might have a few blasé pre-teens who are hard to motivate, and they will inevitably get sucked in. Now they might “whatever” out of the activity after a bit, but even they will want to have done it at least a couple of times :). I particularly like when kids realize they can coordinate action on neighboring squares to do something greater, I purposefully don’t suggest that to them. I’ve gotten good at fending off “learned-helplessness”, not that I was doing it for them before, I’m just quicker to disengage. “You want to control the laser kid? Well you better figure it out”.

The “coding” interface

One of the snag we always hit is kids not able to discern the difference between typing in a URL or doing a search with Google. And a giant middle finger please for all the corpos purposefully blurring lines so kids form the habit early of running anything they might want to do on a computer through Big Corp Inc.

At the end of the day I send the wall plotter on an overnight portrait of a well liked central figure in the school. The next morning when I pick up the machine, the kids get one last wow effect. I’ll make a note of how many “go_to” statements went into the picture, usually several hundred thousands to get them thinking about scale and how curves can really be just a few tiny straight lines. They submit an average of 30 such statements for their cool drawings.

AI, electronics, I.T., web development

I’m teaching a small robotics class, we’re making the small Etch-a-Sketch plotter. Just a couple of kids, good eggs from the 5th grade X,Y coordinates class I teach every year. I’ve already done similar classes in various contexts, it’s a ton of work but very rewarding. One thing I always seek to improve is my Pi image management. Depending on the kids & material, we’ll go through some Linux CLI, or I won’t even want them to touch Linux but we’ll still have to click through some things to get going on Python to control GPIO pins.

What I’ve been wishing for for a while, is a Pi that’s always online everywhere, with the quickest way to get to just Python. Hopefully something web based so you don’t need anything other than your old faithful browser to start throwing code at GPIO pins.

Introducing IPP, the Inherently Programmable Pi! An image you can download, or build.

Step 1 – Python Forking Web Code Editor

And so step 1 is obvious, I want the Ace web based code editor, preset to Python settings, and served by the very Pi it’s meant to write code for. With a few buttons for running code, stopping it, and checking the output.

AI made that a breeze to code, it always blows my mind how well it understands even convoluted assignments. There’s some pretty gnarly stuff going on there. Python runs a web server that serves a coding environment that can fork another Python process, and kill it with varying degrees of prejudice when it won’t go away on its own. I want this packaged a a single Python file with all the bits and pieces bundled in. AI got all this with little help.

Step 2 – Zerotier & Public Bridge

For step 2, I want the Pi to be easily accessible online, no matter what crazy wifi it’s connected to. For this we’re using Zerotier, and an online bridge that is publicly available to forward traffic into it. In my case this is done with Traefik and a Docker container dedicated to forwarding into Zerotier. I unfortunately can’t offer this part of the stack with the image, but you can at least specify a Zerotier network ID as a tunable when building it.

This way, no matter what Wifi network your Pi is connected to, it will always be available via a public URL.

Step 3 – Local Wifi when Nothing Else

I wanted to take the script that turns PlottyBot into a hotspot if it’s not connected to the internet and modernize it for the newest Rapios (Trixie). I also want the web interface to let you connect to existing Wifi networks. This way you are either in a spot with no Wifi, and your Pi spawns a local one which doesn’t route to the internet but you can still program. Or there is pre-configured known internet Wifi within range and your Pi connects to that.

Of course AI yet again turned a multi-hour endeavor into a 30 minutes one. Well, it did mess up some fundamental things but still, I was spared hours of grinding.

Step 4 – Easy Imaging

Finally, I want to create Raspios images with all this and a few tunables baked in. No booting each one to install stuff & tune it for each kid. For this I’m using something I built years ago for mounting *.img files and tweaking them. It’s a Docker container you can pipe an image file into and out stderr comes the customized image. This might be worth its own post, I’ve used this for years to great effect, but that’s beyond the scope of this post. The point is it’s available to use, and here’s how:

Build Your Own Image

First download & decompress the latest Raspios:

wget https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2025-10-02/2025-10-01-raspios-trixie-arm64-lite.img.xz
unxz 2025-10-01-raspios-trixie-arm64-lite.img.xz

Then download & decompress this project’s files to tweak it with:

wget https://ben.akrin.com/downloads/ipp_2025-11-11.zip
unzip ipp_2025-11-11.zip

Finally, create your customized Raspios image with:

sudo docker run --rm -i \
    --name=raspi-image-customizer \
    --cap-add SYS_ADMIN --privileged \
    --device /dev/loop0 \
    --platform linux/amd64 \
    --cpus="1" \
    --memory="500m" \
    -e HOSTNAME=mypi \
    -e HOTSPOT_WIFI_SSID=mypiwifi \
    -e HOTSPOT_WIFI_KEY=raspberry \
    -e ZEROTIER_NETWORK_ID="<ZEROTIER_NETWORK_ID>" \
    -e PIPASSWORD="raspberry" \
    -v ./ipp:/data registry.akrin.com/raspi-image-customizer:latest < 2025-10-01-raspios-trixie-arm64-lite.img 2> ipp.img

–cap-add SYS_ADMIN, –priviledged, and the loop0 device are necessary to mount a file as a disk inside the container.
HOSTNAME is evident
HOTSPOT_WIFI_SSID & HOTSPOT_WIFI_KEY are the settings for the local wifi network that the pi will spawn when it has no known networks to connect to.
ZEROTIER_NETWORK_ID is optional, if specified the pi will join it first chance it gets (when connecting to a managed wifi that routes to the internet).
PIPASSWORD is the password for the pi user, careful SSH is enabled in this image.

You can burn the resulting ipp.img to an SD card. The first boot will be longer than the subsequent ones as stuff gets initialized, but eventually you’ll see a new wifi network pop up. Connect to it, and point your browser to http://hostname.local or if that doesn’t work http://192.168.50.1, you will see the coding web interface. Now that might be enough for your to start coding, but if you want you can click on the wifi network icon at the top right to specify a standard network to connect to. And if you do so, I recommend you have a way to “follow” your Pi, either via Zerotier, or because you manage that network and are able to see what IPs devices get assigned. And if you setup a Zerotier forwarder on top of all this, your pi essentially becomes online to the world the minute it’s on. Now that isn’t just convenient, it also significantly lowers the bar to start coding with kids.

Or… Download a prebuilt one

Here’s the link.

parameters used for building it are:

HOSTNAME=ipp
HOTSPOT_WIFI_SSID=ippwifi
HOTSPOT_WIFI_KEY=raspberry
PIPASSWORD=”raspberry”

No Zerotier, but still everything else.

plotters, web development

Some of the tools I’ve accumulated over the years lend themselves particularly well for teaching. I’ve gotten better at swooping in a classroom with some plotters and talking about math or machine drawing topics. I can make it very interactive.

One tool in particular is https://draw.mandalagaba.com.

It started not so much for teaching, but rather for collaborative art. Very much inspired by r/Place, I created a tool with real estate scarcity, but with my usual focus on drawing. The kicker would be that the art is rendered live on a plotter somewhere in the world, maybe in front of you in a public place, maybe a video stream and you know your pen strokes are executed by a machine thousands of miles away. You claim a square, coordinate with others to make bigger pieces, and a collaborative art piece is born. That was the idea at least. I’ve tried several permutations of this experiment, in public, on a stream, with various rules and online communities, and it simply never generated the enthusiasm I thought it would. I have 95% given up on the idea of a collaborative art piece of the sort. The big plotter still has me wondering hence the remaining 5%, but it is tedious and expensive to deploy for an experiment that has shown no promise at every iteration.

What this website became great for however is teaching as it allows for time sharing a drawing machine across several students. We can explore one concept or other, and then apply it and have the great motivator that we’ll get to control the machine by doing so. 3 years ago I taught a module on X,Y coordinates with the local 5th graders, I didn’t have this site then and it went fine. But the following year I figured I might as well use this dumb collaborative art website I made for something so I integrated it into the coursework. Now that second year The machine’s made an enormous difference in how students engage. Recently I ran this course for the 3rd year with the website and the magic happened again. Both times, kids were very much willing to go through teeth grinding (light X,Y coordinate based coding), and help each other out to see their work realized in the world by a cool machine. The first student that submits a drawing on the site inevitably has everyone jumping off their seat to see the machine move up close. Then they’re even more motivated to do it themselves. I help them with the mistakes they make at first, and once they get it I have a good half an hour of walking around and seeing the cool stuff they draw. During this time I might send one drawing machine on a more intricate plot that can finish quick. Having that plotter wield a pen expertly inevitably draws 2 or 3 kids who just sit and stare at the machine non-stop. I know what they’re going through. The whole experience has them asking so many questions, and I love interacting with the different ways in which they see the world. I also love the opportunity to get to know every kid in town a little better when they go through 5th grade.

I’m quite glad that these tools found a place in education. Sometimes I think of pushing further, there’s definitely something to this formula, and I know teachers often purchase various curricula or interventions that meet a standard or other. But I can’t push all the projects, and so far I’ve refused to let money taint anything plotter related.

Funnily enough, I even had an opportunity to guest lecture in the same way with college art students. Of course we don’t talk about 5th grade X,Y coordinates, and I’m not there to talk about art either. Rather we learn and employ tools and techniques for line art and single instrument machines. It’s only funny because I’m ashamed to admit I’ve been very dismissive of art (except music) my whole life. I never understood it and wasn’t a fan of the carefree personalities gravitating to it. I’ve seen the light now, and I do feel apologetic for the times someone showed me a something they had made and it went completely above my head. But I can’t say I was particularly impressed by the students. I’m definitely curious to try again and see if I can bridge more understanding. In the meantime, I have no issues resonating with 5th graders so I’ve got that going for me :). Except for keeping them focused, that is insanely hard.

I.T., maniacal paranoia, web development

This web thing’s been bugging me for too long. Have you ever tried to background a tab that is playing insufferable & unskippable content, only to find out that the annoyance has paused itself until your eyeballs are known be aimed back at it? Why do browsers honor requests to let websites know if you’re paying attention or not?

This is achieved by relying on the focus and blur events. But there are many UI Elements that rely on them to trigger useful UI responses. Think of a suggestion box that shows up when you click in a search bar for example. The window element though, is one for which I cannot think of a single instance where the focus and blur events at are used to benefit the user. I think a well intended couple of events were generally implemented to every possible elements, but one of them reveals more than was intended and is abused to that effect. Why would ad-blocker not nuke them either? I’ve gone through this rabbit hole several times over the years trying to find an extension or adblocker customization to dismiss these events. Alas, they never seem to have made it into the crosshair as the true annoyance that they are. How do you like to have your browser report how good you are at consuming content as intended?

These events are responsible for more ills than making sure you’re watching, they are a key metric for inferring behavior. As with much of data mining, what’s scary isn’t really the information you’re giving away, it’s what can be inferred from it. In a way these attention events are perfectly suited for the attention age. Particularly though, they matter when they are attached to the window element. As far as I know, that is the only method I’ve seen in the wild that is abused into this purpose.

In any case, since I never could find anything, here’s what I came up with. The best way I found to run user JS on all websites is using Tampermonkey. Then here’s the script I’m running:

// ==UserScript==
// @name         Attention Event Nuker
// @namespace    http://tampermonkey.net/
// @version      2024-05-01
// @description  nukes focus and blur events when attached to the window element
// ==/UserScript==

(function() {
    var old_add_event_listener = EventTarget.prototype.addEventListener ;

    EventTarget.prototype.addEventListener = function(event_name, event_handler) {
        if( this.toString()==window.toString() &&
           (event_name=="blur" || event_name=="focus") ) {
            console.log( "attention event caught: " + event_name + " on: " + window.location.host ) ;
        } else {
            old_add_event_listener.call( this, event_name, event_handler ) ;
        }
    };
})();

Unfortunately I did run into a couple of sites that somehow rely on the events to even work properly. I don’t think I want to reverse engineer them 1 by 1 so I’m adopting a blacklist of sites which is a bit obnoxious. For a while I did have the script report which sites were asking for the events, the results weren’t surprising and showed that pretty much any big site with a baseline of behavioral data mining wants to know what your eyeballs are in front of.

I.T., plotters, web development

I wrote something pretty neat for Plottybot, and for the longest time I thought I should make it available on its own, and detached from the project. Then the most excellent Stuff Made Here guy made a writing machine, and ran into all the issues I ran into which drove me to write my own algorithms for capturing and replaying handwriting. My stuff wasn’t online then and that’s a shame, it was only available by building a Plottybot, or at least using its Pi image. Oh well.

As is tradition, I captured my kids’ handwritings as I do every year some time in the Winter. But this time I made sure to have the new site ready before Thanksgiving so that people could use it as they went and met with loved ones.

So here, this site serves the purpose of capturing one’s handwriting. It supports cursive, character variations, saving, and finally exporting to SVG & GCode. Hopefully this means you can use it with your favorite craft machine for the coolest of personalized projects.

https://ben.akrin.com/plottytools/

I.T., plots, plotters, web development

I built a website for running experiments in collaborative drawing. It’s pretty neat and I’m not going to describe it just yet, but in the process of testing it, I threw at it all kinds of plots I had at the tip of my fingers, and it yielded some pretty cool results.

I may have here my next plotting streak: microplots. Some look predictably bad as they were meant as stress tests, but some came out well enough to make me curious.

More to come on all this soon…

miscellaneous, web development

Mandalagaba is no longer available in Russia or Belarus. That’ll slightly annoy about 30 people per day. It feels wrongly targeted at little people going about their lives. But it also feels like I don’t want to pay server time to provide something nice while they have Putin in power. I say all this knowing full well how close we got to having our very own dickhead motherfucker of the same vein seize power.

I.T., web development

With teachers scrambling to move to online teaching last March, Mandalagaba’s traffic saw a noticeable bump in use which then died off in with the Summer break. With schools now back in session it looks like the bump is back.

I’ve kind of given up on trying to do anything big with it, there’s just so much noise out there it’s really hard to get anything more than an occasional spotlight. Which means the site is completely free of all the perverse incentives which ruin the internet these days. In other words, perfect for educators teaching to kids. I was already happy to know that about a thousand people per day enjoyed doodling with absolutely no strings attached, but it’s a whole other layer of contentment to see teachers use it for teaching. You can tell there’s love, kindness and attention in the way they address their kids. I love finding their videos online.

I often get asked what restrictions there are on the material people produce, or what my privacy policy is. I respond that I don’t have any of either, but the questions bother me for they reflect that everything online comes with strings attached these days.

aesthetics, I.T., web development

It’s been a while since I pushed a feature to Mandalagaba worthy of a post.

I just pushed a tool I call “Fibonacci Assist”. It’s meant to help you draw Fibonacci spirals by overlaying the proper framing based on the beginning of your penstroke.

I was reminded of the existence of this sequence and its ties to nature reading the most excellent children book: Swirl by Swirl. The book is simply magnificent and so I thought I’d do my part to help bring into the world more Fibonacci spirals.

I have yet to play with it using a stylus.

I.T., web development

Someone ripped an early version of Mandalagaba, removed any mention of it, translated it, and is hosting it as their own.

Feels like I somehow made it as a developer to have my work stolen :). And honestly I don’t really give a crap I don’t have too many qualms about using code myself.

Today’s Mandalagaba would be very hard to reverse engineer, it’s way more complex, has more server components, and the client code is more obfuscated. In my opinion that’s how you retain control over your work, at least as a coder.

 

aesthetics, miscellaneous, web development

This Dragon which found its way into the Mandalagaba digest. I haven’t seen anyone really touch recursion yet, and then this shows up.

This is my absolute favorite thing about adding features to Mandalagaba: seeing them used in ways I didn’t even realize were possible. Here recursion is used to create ethereal traits. Of course mirror symmetry is used for the face too but that I had seen before :). The hours spent hammering out recursion are all of a sudden, absolutely worth it.

The artist is here: https://www.instagram.com/laralaubert

I.T., web development

It’s only available on the “pro” facet of Mandalagaba. Oh yeah, Mandalagaba has been broken down into several facets for the purpose of simplifying and focusing activities. The facet where all the brain hurting stuff goes is the pro one. Pro is where one can draw a “radially symmetrical tessellation recursively”.

I tried to keep the menu simple and icon based but it’s still the most complex. Thankfully we have words to describe all that these things do.

Along with recursion, https://plant.mandalagaba.com is out too, using recursion, symmetries and a newly developed center placement to create plants & trees. This one still needs a bit of polish but the main idea is working. I touched a lot of the core engine of Mandagaba to introduce all this, development went pretty well. It was nice to test the flexibility I gave myself during last Winter’s rewrite for a major feature add.

 

I.T., web development

I still have a lot of details to figure out but I almost have recursion nailed. It’s a pretty brutal mental exercise to combine it with the existing radial symmetry & tessellations. As I’ve said before, one of the effects I seek in particular is that of a growing plant. In fact, I plan on dedicating a facet of Mandalagaba to guided drawing for just that. The adjustment of parameters which is very specific for getting a plant will be taken care of, and only the fun parts will remain. A perfect tool for the slacktivists who wants to feel like they’re helping greening up the world while staying glued to their screen.

I sometimes feel bad that so many people find stress, anxiety and even depression relief drawing mandalas on Mandalagaba. I feel like an enabler for giving them such a meaningless escape valve, on a screen at that. As much as I love a plant growing effect, I’m worried people will use it to get their virtual green fix while further removing themselves from nature. Get off your screens, grow an actual plant if you want to grow a virtual one.

aesthetics, I.T., web development

I’ve been working on implementing recursive drawing into Mandalagaba. There are many, many tunables which can be implemented to facilitate recursive drawing. It’s tricky to pick the ones which make for a fun drawing experience. One of the effects that I seek in particular is that of a growing plant.

I.T., web development

The millionth penstroke on Mandalagaba since the code rewrite last February

There’s more data I’d like to pull out of this. For example the average length of a stroke, average time, how many human lives where spent drawing mandalas, et cetera :).

I.T., web development

I took the longest time implementing the fill tool  on Mandalagaba. How hard could it be? Recurse through pixels looking for a color and update them to a new color.

While this method certainly works and is easy to implement, it is also extremely slow. Slow in a way that hangs the browser, yielding infamous messages from the browser.

Here we’ll take a look at various Javascript flood fill implementations along with their drawbacks. Jump to #4 if you are only interested in the best one.

2018-10-04 – edit – added fill algorithm #5 which proposes an alternate approach to filling pixel by pixel. Depending on your project it may or may not serve your needs better.

In all cases, the code is available in the example iframe, look for the function flood_fill.

1. Simple Recursive

Not much to explain here, we simply recursively call the function on adjacent pixels when they match the color we are trying to fill over.

Click to change color / pop-out link

It’s reasonably fast but the problem with this one is that any fill area slightly large yields too much recursion which breaks subsequent JS. This Canvas box is 200x200px and at 300x300px, Firefox complained about:It’s easy to see how this implementation will not satisfy a reasonably featured paint program. Even if your browser let you stack more function on the heap, I would bet it would lead to slowness.

2. Iterative

We simply take the previous idea of looking at adjacent pixels and filling them, and make it iterative instead so the function calls don’t get stacked to a ceiling.

Click to change color / pop-out link

The problem with this is is that is is sloooow. So slow it stalls browser. Most of time is spent having to keep track of pixels_stack. Recursion doesn’t have that need but as we’ve seen, it has other issues.

3. Recursive-Iterative (AKA catch-your-breath iterative)

This is a twist on #2 which every so often, recurses on itself via a setTimeout to let the browser catch it breath a little. It also yields a cool visual effect.

Click to change color / pop-out link

I really like the visual effect, and it makes the slowness tolerable. But the issue is that Mandalagaba has a network engine and allows for re-rendering of one’s work. So synchronization is a big deal, and you know what makes synchronization easy? Not having to worry about it.

So as long as I can help it, my life is a lot easier if the operations the users can perform are atomic. Operations need to be able to be processed one after the other counting on the fact that the ones that came before have completed.

The first 2 solutions are atomic but suck; this 3rd one, however cool it may be, isn’t.

4. The Holy Grail

I’m not sure where this algorithm originated from, but I’ve gotten to know it on this web page which explains it very well (with GIFs!). It is iterative and goes about finding pixels to fill in a much smarter way.

Click to change color / pop-out link

That’s it, no drawback here 🙂 I’ve tested it on large canvasses and this is what is implemented on Mandalagaba. Now of course, in a real application there is a ton more complexity dealing with smoothing edges and blending alpha. I only wanted to expose boiled down versions of these algorithms so they are easier to wrap your mind around.

5. The Holier Grail?

While #4 is fastest algorithm for filling pixel by pixel, I found myself in the need to have a fill operation in for form of a path which is filled with the native HTML5 canvas capabilities. This algorithm is a bit different from all the other ones in that it doesn’t go through every pixel and fills it. What it does is pathfinding to draw the outline of the shape of be filled, and then simply calls the native canvas fill() function. It does come with certain drawbacks and so I wouldn’t recommend it unless you specifically need this sort of approach.

Click to change color / pop-out link

Feel free to ask questions in the comments.

I.T., maniacal paranoia, web development

Full disclaimer:

The subject matter of this post is controversial as it discusses extracting computing resources from the visitors of a website. There are a lot of discussions at the moment centered around web-browser based crypto currency mining. Most paint a deplorable picture of the practice; please keep in mind that there are very desirable paths alongside which these practices can develop. I am not elaborating on these arguments here, I am only describing a method to harness the resources.

Premise

Web browsers are becoming quite powerful for code execution. Between Javascript’s increase in capability, WebAssembly, access to GPU & threading, a web browser today is almost as desirable for computing as the machine it’s running on. Ever since the rise of web-based crypto currency miners, I’ve been thinking of harnessing all that computing power as a single entity: a super computer made of your visitor’s web browsers.

Just like a regular computer cluster, the nodes all participate in a coordinated fashion to solving a single problem. Unlike a regular computer cluster, the nodes are very ephemeral (as website visitors come and go) and can’t talk to each other (no cross site requests).

Here’s a demo of what I came up with:

Right: the super computer control server
Left: one of the web clients contributing to the super computer simply by being connected to a website (& CPU metrics)

[mejsvideo mp4=”http://ben.akrin.com/videos/transient_node_javascript_supercomputer.mov.mp4″ ogg=”http://ben.akrin.com/videos/transient_node_javascript_supercomputer.mov.ogv” webm=”http://ben.akrin.com/videos/transient_node_javascript_supercomputer.mov.webm” poster=”http://ben.akrin.com/videos/transient_node_javascript_supercomputer.mov.jpg” width=”640″ height=”360″]

The problem being solved here is the hashing of 380,204,032 string permutations to find the reverse of a given hash. Problem parameters were chosen to make heavy processing quick for the clients.

Implementation & code samples

At the core of the idea is the websocket technology. It creates a persistent connection between a server and all of the nodes (the visitors of your website). This connection can be used to orchestrate actions between the nodes so that they can act as a concerted entity. From delivering the code to passing messages for coordination, websockets are what make everything possible.

Having a websocket connection to clients dramatically changes what you can do with web clients. They are fully addressable for the duration of their visit. They may show up on a website and be served some pre-established javascript; but with websockets, any javascript can materialize at any time.

Right: the super computer control server
Left: a web client being given an instruction on the fly

 

Slightly tangential but still worth considering, using a web view app, Javascript can pass execution to the app itself. This means code showing up on the websocket can escape the webview bubble and go into app land.

Right: the super computer control server
Left: a web app being given an instruction which percolates to the app layer

 

Now this is nothing new in a lot of ways; apps can be made to get instructions from C&Cs, and websites can get Javascript after the initial page load from dynamic sources. The websocket technique though is as dynamic as it gets (no Ajax pull), it is portable to many browsers and many devices, it is hard to catch looking at a web inspector; lastly, it executes with full access to the context it materialized in.

So we’ve established that websockets can be used to dynamically deliver code to be ran by the nodes. It can also be used for message passing and the overall orchestration of distributing the problem to be solved.

Crackzor.js

6 years ago I wrote a ditributed OpenMPI based password cracker: crackzor. Password cracking is a good distributed problem to solve because it’s a fairly simple problem: run through all the character permutations. The fact that it exhausts a known space also means benchmarking is easy. So to put the idea of a transient node javascript super computer in practice, I rewrote crackzor in JS instead of C, and for websockets instead of OpenMPI.

Every distributed problem is different and crackzor itself isn’t a magic way to distribute any problem to be solved. The magic of crackzor is its ability, given a space of character permutations, to divide it up in chunks which can be processed by the nodes. Given the problem, a start iteration and end iteration, a node can get to work without having to be provided the permutations themselves, thus removing the bandwidth bottleneck.

The first challenge: maximizing usage of the node’s CPU.

Javascript runs single threaded by default, so when the websocket sends code to be ran by a client, by default, the code running as fast as it can will only be able to fill one core of the CPU. A large majority of machines today have many more cores available. So we have to figure out how to use them or our super computer is going to loose a large portion of its processing power right off the bat.

Web workers to the rescue. With HTML5, it’s easy as pie to thread code. The one trick with the code we want to thread is that it can’t be gotten from a file as the web worker documentation suggests. That’s because our code doesn’t come from a static javascript file remember? It shows up out the the blue on the websocket, so it came from the network and is now in memory somewhere => not a file we can refer to.

The solution is to wrap it in a blob as such

[code language=”js”]var worker_code = ‘alert( "this code is threaded on the nodes" );’

window.URL = window.URL || window.webkitURL;

var blob;
try {
blob = new Blob([worker_code], {type: ‘application/javascript’});
} catch (e) {
window.BlobBuilder = window.BlobBuilder || window.WebKitBlobBuilder || window.MozBlobBuilder;
blob = new BlobBuilder();
blob.append(worker_code);
blob = blob.getBlob();
}
workers.push( new Worker(URL.createObjectURL(blob)) ) ;[/code]

Here you’ll notice we have our first layer of encapsulation. The code relevant to the problem we are solving is in the variable worker_code, the rest of the javascript only threads it.

Having distributed amongst a node’s cores, we now look at

the second challenge: distributing between the nodes

This work is obviously up to the websocket server along with subsequent coordination. Without going into too much details, the websocket server keeps track of all the nodes as they come and go, it also keeps track of which ones are working or not, allocates new chunks of the problem to nodes as they become available.

A trick of the websocket server is that it is running at all times to handle node connections. Super computer problems however may change from one day to the next. To address that, I give it a function which reads a file and evals its code; the function is summoned by a process signal. As such:

[code lang=”js”]function eval_code_from_file() {
if( !file_exists("/tmp/code") ) {
console.log( "error: file /tmp/code does not exist" ) ;
} else {
var code = read_file( "/tmp/code" ) ;
code = code.toString() ;
eval( code ) ;
}
}

process.on(‘SIGUSR1’, eval_code_from_file.bind() );[/code]

With this puppy in place, the next time I “kill -USR1 websocket_server_PID”, it will be imbued with new code that did not exist when it started. Does this sound familiar? Yup, javascript is super interesting in the ability it gives you to run arbitrary code at any time with full access to the established context.

Thus arrive the 2nd and 3rd layers of encapsulation, the code which will be distributed to the nodes is in a file which is to be evaled on the websocket server side and sent over the websocket to the clients.

The actual distribution to the nodes is simple, have them connect with a callback to eval code. Something like that:

Client:

[code lang=”js”]var websocket_client=io.connect("http://websocket_server.domain.com") ;
websocket_client.on( "eval_callback",function(data){data=atob(data),eval(data)}.bind() ) ;[/code]

Server:

[code land=”js”]client_socket.emit( "eval_callback", new Buffer("alert(‘this code will run on the client’);").toString("base64") ) ;[/code]

Recapping where we are

So…

  1. all the transient nodes (web browser of website visitors) attach to a websocket server
  2. the websocket server receives SIGUSR1 which signals it to execute new code it gets from a file
  3. this new code gives the websocket server a packaged problem to be solved by the nodes
  4. this new code also instructs how the websocket server will distribute and coordinate the nodes
  5. once the packaged problem to be solved shows up on a node, it is evaled and it contains threading to maximize CPU usage.

And there you have it,

all the pieces you need to make a super computer from your web traffic. I’m choosing not to publish the full code of my implementation for reasons of readability, security and complexity but I can go into more details if asked.

The same way that peer-to-peer protocols made any data available anywhere any time, could this do the same for computing power? Mind=blown, and your CPU along with it.

More tips

  • When choosing a chunk size for clients to work on, it’s important to not pick too big a size. The nodes are very transient and a big chunk size means the chunk’s processing is more likely to be interrupted. Most web browsers also offer to kill poorly coded javascript running berserk and so a small chunk size taking a few seconds and letting the machine catch it’s breath briefly will make it less likely that a browser will notify a user that a script needs to be killed.
  • When encapsulating out the wazoo, keep in mind that Internet Explorer (Edge or whatever it’s called today) doesn’t support backticks.
  • Syntax highlighting will be confused by the strings in strings in strings of encapsulation, it helps to just turn it off.
  • Javascript md5 implementation here: https://gist.github.com/josedaniel/951664
  • I found it necessary to keep track of an average time to solving a chunk so that I may exclude the nodes which are taking too long and polluting the good performance of the supercomputer.
aesthetics, all out geekery, I.T., miscellaneous, web development

Very intense 2 months coding marathon to bring into the world the new version of Mandalagaba.

I completely rewrote the symmetry engine to be universal. When I coded the first version, I only wanted to scratch a specific radial symmetry itch and had to expand on narrowly conceived code to accommodate for features that came up from the tool’s success. With this new version, I instead gave myself a broad framework built for expansion, I can translate any penstroke at any angle in any location. Beyond mandalas, it makes possible tessellations and even the 2 combined.

I used the opportunity to add many features which were lacking: zooming, forking, lines, color picker, et cetera. With many more to come. The interface was rethought to be more accessible. Doing so took much more time than building the core engine.

There is an obscene amount of math that goes behind every pen stroke you draw in the tool. It was kind of fun to go through it again in my life, 20 years later. Even though I had forgotten about it all, it came back nicely. It’s amazing to have the internet as a tool to look up methods, to be able to describe the problem in plain English and have potential solutions thrown at you. It used to be that you needed to know what you needed precisely to find it in a book.

 

I love that Robin copies what I do no matter the understanding level, we’ve had lots of talks about what is going on.

It’s not just the math but also algorithms, languages & infrastructure. Not to toot my own horn but in my 30s, I’ve never felt so intimate with every aspect of an idea’s implementation. It’s extremely enabling to know exactly where to go to achieve X. Honestly though I’m a little burnt out at the moment, something that was supposed to take 10 days took more than 2 months of coding every single night.

My hope is that the new tool becomes a reference online for this type of work. And it’s all 100% free; well… we’ll talk about that in the next post.

Lower tech fun found in a thrift shop

aesthetics, all out geekery, I.T., web development

Between the brutal cold and children, I haven’t had as many untainted brain cycles as I’ve been wishing for; still, I just finished the core engine for a universal way to apply translations to pen strokes. It allowed me to rewrite the mandala engine better, and expand it to allow for tessellations, and really any kind of translations on any center at any orientation. It’s been a ton of ground work so it’s nice to finally get some eye candy :).

I can’t wait to see what the internet does with it. Here’s a preview:

[mejsvideo mp4=”http://ben.akrin.com/videos/tessellation_preview.mov.mp4″ ogg=”http://ben.akrin.com/videos/tessellation_preview.mov.ogv” webm=”http://ben.akrin.com/videos/tessellation_preview.mov.webm” poster=”http://ben.akrin.com/videos/tessellation_preview.mov.jpg” width=”640″ height=”360″]

I.T., web development

Here is an elegant way to use the same caching logic for all function calls which should have a cache. With the proliferation of 3rd party APIs I was quite happy to find a way to address them all with a single mechanism.

[php]function expensive_third_party_call( $param1, $param2 ) {
// universal caching algorithm header
$result = memcached_retrieve( __FUNCTION__ . serialize(func_get_args()) ) ;
if( $result!==null ) {
return $result ;
}

// this is where the third party call actually happens, if we are hit the cache missed
$to_return = /* some super complex and time consuming logic, throw in a couple of web calls*/ ;

// universal caching algorithm footer
memcached_store( __FUNCTION__ . serialize(func_get_args()), $to_return, CACHE_TIMEOUT ) ;
return $to_return ;
}

////////// helper functions bellow //////////
$m = false ;

function memcached_retrieve( $key ) {
global $m ;

$new_key = md5( $key ) ;
if( $m===false ) {
$m = new Memcached() ;
$m->addServer( ‘localhost’, 11211 ) ;
}
$temp = $m->get( $new_key ) ;
$result_code = $m->getResultCode() ;
if( $result_code==Memcached::RES_SUCCESS ) {
return $temp ;
} else if( $result_code==Memcached::RES_NOTFOUND ) {
return null ;
} else {
echo "error: can’t retrieve memcached key {$key} with result_code {$result_code}" ;
}

return null ;
}

function memcached_store( $key, $data, $timeout ) {
global $m ;

$new_key = md5( $key ) ;

if( $m===false ) {
$m = new Memcached() ;
$m->addServer( ‘localhost’, 11211 ) ;
}

// a little heavy handed but we use null to represent that nothing was found in the cache so we can’t have this be the data
if( $data===null ) {
$data = false ;
}

$m->set( $new_key, $data, $timeout ) ;
$result_code = $m->getResultCode() ;
if( $result_code!==Memcached::RES_SUCCESS ) {
echo "error: can’t store memcached key {$key} with result_code {$result_code}" ;
return false ;
} else {
return true ;
}

return false ;
}
[/php]

Requirements

  • apt-get install memcached php-memcached
  • make sure to define CACHE_TIMEOUT

Functioning principle

Using PHP’s awareness of the current function it is in along with the parameters which are passed to it, we derive a unique key which is used so store and retrieve from the cache.

Theme by Bloompixel. Proudly Powered by WordPress