Summarity

Creating Alpine.js Components in Nim

Wed, 27 Dec 2023 00:00:00 +0000

Nowadays, I’m building more and more services using the MASH stack: Mummy, Alpine.js, SQLite, and htmx. This stack provides a performant, pleasant, and straightforward way to develop web applications.

In my first attempt, Findsight, the frontend HTML was generated using server-side-rendered karax. Note that this was not utilizing the karax reactive framework, but its DOM-builder DSL. However, the Alpine component code, state management, various app startup code, and htmx event handlers were all written using inline JS, which proved to be unwieldy as my projects grew.

In this article, we’ll explore how to use Nim as a full stack language for type safety and maintainability. We’ll build a complete example of frontend and backend code, which will demonstrate the important concepts to keep in mind when using Nim with Alpine.

Alpine Primer
Frontend
Backend
Conclusion

Alpine Primer

Consider a single binding:

<span x-show="open">Content</span>

That’s fine, as we’re including the component property open by itself and there’s no active JS code in here. However, even a tiny component store violates this by including a JS object:

<div x-data="{ open: false }">

Or, staying with the same example, an event binding that contains code:

<button x-on:click="open = true">Expand</button>

Side note: x-on can be shortened to @.

These are smaller examples, but in a sufficiently complex app, the complexity of these little inline snippets grows with it.

Fortunately, Alpine provides a way to extract the component (x-data) into a standalone JS object. Here’s an example straight from the documentation:

<div x-data="dropdown">
    <button x-on:click="toggle">...</button>
 
    <div x-show="open">...</div>
</div>
 
<script>
    document.addEventListener('alpine:init', () => {
        Alpine.data('dropdown', () => ({
            open: false,
 
            toggle() {
                this.open = ! this.open
            }
        }))
    })
</script>

And x-bind enables the same for any template directive. Both approaches combined keep the HTML clean.

But now we’d have to write JavaScript, and I’d like to stay in Nim. Nim does have a JS backend, and there are just a few steps needed to create those components in pure Nim.

Frontend

First create a frontend directory. We’ll put all client side code in here, which is going to be compiled to JS from Nim, and then later minified and optimized.

Project Structure

.
├── frontend
│   ├── alpine.nim
│   ├── app.nimble
│   ├── app.nim
│   ├── components.nim
│   └── nim.cfg

The Nim config nim.cfg sets our backend, which is picked up by nimsuggest in VS Code to provide backend-relevant error checks:

backend:js
-d:release

It also forces a release build. Nim-compiled JS, depending on the libraries used, is extremely verbose and the debugging info that is generated is less useful in JS code. By compiling in release mode, we end up with more readable JS.

The resulting JS is still too large to ship directly to users, so we need to optimize it. For this, I’m using Terser, which is the best standalone optimizing minifier (esbuild and the Google Closure Compiler both have similar functionality, but are a less effective). Install it system wide via:

npm install -g terser

Or use npx to invoke it later without prior installation.

Now add it as a build task to app.nimble:

import strformat

bin           = @["app"]
backend       = "js"

requires "nim >= 2.0.0"

after build:
  exec &"terser --compress hoist_funs,keep_fargs=false,unsafe --mangle toplevel {bin[0]}.js -o {bin[0]}.dist.js"

Other than the default terser options, we enable:

additional compression options
- hoist_funs: hoists function declarations
- keep_fargs=false: discards unused function arguments (breaks code which relies on Function.length)
- unsafe: enables all unsafe optimizations (which aren’t actually that unsafe)
global name mangling (toplevel) which enables terser to mangle some of the support functions that Nim emits for e.g. string formatting which would otherwise have quite long gensym names

These options are optimized for this use case and will be different if you’re working on a Nim-JS project that isn’t Alpine. For example, you wouldn’t use global name mangling for JS libraries. Since Alpine component code depends on object properties and methods, they’re unaffected by the mangling algorithm.

Component Contract

In components.nim, let’s define the type interface for both the component state (used with Alpine.data) and the behavior (used with x-bind):

import std / [
  dom
]

type
  DropDown* = ref object
    init*: proc()
    destroy*: proc()

    open*: bool
    toggle*: proc(msg: cstring)
    timer*: Interval
    counter*: cint
    
  DropButton* = ref object
    `type`* = "button"
    `@click`* {.exportc: "['$1']".}: proc()
    `disabled`* {.exportc: "[':$1']".}: proc(): bool

To break down what’s going on here:

std/dom gives us access to a lot of FFI types for interacting with the browser’s object implementations.

The object types are declared as ref, which is the default for JavaScript. Without ref, Nim will use copy semantics, and emit large helper functions for deep-copying, which are not required in this case.

DropDown holds the component state:

init and destroy are object methods (in the JS sense) that are invoked by Alpine when the component is created and destroyed
open will hold the current open/collapsed state
toggle is the method that will change the open state, and also demos parameter passing to object methods
timer demos a component property that refers to a native browser type, in this case a time interval
counter is another property that will be used later to explain object property access from within methods

DropButton holds the binding:

type is an Alpine attribute that determines the DOM type of the bound element
@click (or x-on:click) binds to the event (x-on) click
:disabled is a direct attribute binding that maps the given DOM element attribute value to a component method/property

You may notice the exportc pragmas on the last two properties. This is a necessary workaround for Nim’s name mangling. While some identifiers in Nim can be escaped with `name`, it’s not nearly as flexible as JS’s ['name'], so we have to cheat a bit with exportc’s built-in string formatting:

For @click, the "['$1']" format produces ['@click']
For disabled, the [':$1'] format produces [':disabled']

Alpine Abstraction

We’re almost ready to dive into the component implementation, but we need a few FFI helpers first. In alpine.nim:

proc registerComponent*(name: cstring, component: pointer) 
  {.importjs: "Alpine.data(#, #)".}

proc bindElement*(binding: cstring, actions: pointer)
  {.importjs: "Alpine.bind(#, #)".}

proc env*[T](fn: (proc()), bar: T): (proc())
  {. importjs: "(#).bind(#)" .}

template this*(T: typedesc) =
  var this {.importc, nodecl, inject.}: T

registerComponent and bindElement map to the respective Alpine functions
env is a mapping of JavaScript’s bind function, which allows a caller to set the inner this - this may be confusing at first but is required later as we get into closures over object instances from within methods
this allows us to make Nim aware of the implicit this in a method, and give it a strong type to make nimsuggest and compiler messages more helpful. importc and inject ensure that the code emitted by Nim refers to this literally

Component Implementation: State

In app.nim, we’ll place the actual implementation, starting with the component state and methods:

import std / [
  dom,
  strformat
]

import alpine
import components

proc dropdown(initialOpenState = false): DropDown =
  DropDown(
    init: (proc() =
      this DropDown
      
      this.timer = setInterval(
        (proc() = 
          this DropDown
          inc this.counter
          echo &"Counter is now at {this.counter}"
        ).env(this), # explicit binding
        1_000
      )
      
      echo "Component initialized and timer started"
    ),

    destroy: (proc() =
      this DropDown
      clearInterval(this.timer)
    ),

    open: initialOpenState,
    
    toggle: (proc(msg: cstring) =
      this DropDown
      this.open = not this.open
      echo &"Parameter passed to toggle: {msg}"
    )
  )

There’s a lot more behavior in here than required for a normal dropdown, mostly because I want to show you how to implement other common patterns as well. If you understand what’s happening in this component, you’ll be able to create components of virtually any complexity.

Let’s start with the proc signature:

proc dropdown(initialOpenState = false): DropDown;

Alpine.data (see the example at the beginning of this article) expects a function, which returns a component: a JS object with methods. This is that function. But Alpine also allows parameter passing, usually used for initial state. Here we receive a boolean that indicates the initial state of the dropdown and later copy this into the open property.

All other properties of the Dropdown component are methods. They’re declared inline for brevity, but that is not required. All of them open with this Dropdown, which declares the Nim identifier this with the given type.

In init, on component creation, the timer property is initialized with a new interval timer using the JS function setInterval. Here we have to be careful with scoping. We need to ensure that the inline proc passed to setInterval closes over the correct this. We can’t use this Dropdown on its own in the timer proc, because it is not invoked from the Alpine environment, but from the browser timer callback. Instead, we need to explicitly close over the outer this. There are two ways to do this:

use the env binding we declared before as shown here - this will create a new function that wraps the inner function in a closure environment with the supplied this value
declare a new variable in the method that the callback can close over

For the second variant, we’d have to add a line like var that = this below the this Dropdown in the init proc body, and then refer to that instead of this from the timer callback proc. Personally I consider env to be a cleaner way of achieving this.

Other than that, the init method is simple: create a timer that keeps incrementing the counter and print new values to the console (echo becomes console.log). destroy ensures the timer is deregistered.

toggle is the core functionality of flipping the dropdown’s internal state. It also demos the passing of parameters to component methods.

Component Implementation: Binding

The last thing we have to implement on the frontend is the binding, to keep the HTML as clean as possible:

proc dropbutton(): DropButton =
  DropButton(
    `@click`: (proc() =
      this DropDown
      this.toggle("boo!")
    ),

    `disabled`: (proc(): bool =
      this DropDown
      return this.counter > 10
    )
  )

On a mouse click event, the toggle method is invoked with a parameter. After the counter reaches 11 (so after 10s), the toggle button will be disabled.

Finally, we register the component:

document.addEventListener("alpine:init", proc(e: Event) =
  registerComponent("dropdown", dropdown)
  bindElement("dropbutton", dropbutton)
)

Building the Frontend

A simple nimble build in frontend/ now produces two files:

-rw-r--r--@ 1 turbo  staff    11K Dec 29 23:43 app.dist.js
-rw-r--r--@ 1 turbo  staff    34K Dec 21 18:32 app.js

You may be surprised at how big the compiled artifact is. 34k for a simple dropdown?! What’s actually happened here though is that Nim has emitted standard library functions we’ve used, e.g. for string formatting. That will only happen once. As you add more functionality, those functions are re-used, so the output size doesn’t grow at this rate!

You can also see Terser has done its job and the final artifact is only a third of the size and looks like this:

function n(n){var l=m(n.length),e=0,t=0;n:for(;e<n.length;){var a=n[e];if(a<128)l[t]=String.fromCharCode(a),e+=1;else{var u=m(0);l:e:for(;;){0;var i=a.toString(16);if(1==(null==i?0:i.length)?u.push("%0"):u.push("%"),u.push(i),e+=1,n.length<=e||n[e]<128) // and so on

If you’re size sensitive you should reduce the reliance on std libraries and bundle JS libraries instead, that you then access via FFI. For example, if I remove the strformat import and the echo statements from the app.nim module, we end up with a much smaller file:

-rw-r--r--@ 1 turbo  staff   3.0K Dec 29 23:49 app.dist.js
-rw-r--r--@ 1 turbo  staff   7.6K Dec 29 23:49 app.js

Backend

Our frontend project is now complete and we have to serve it. I’m not going to show you how to build a full MASH app, just the basics of SSR HTML that uses the component we built above.

Here’s the project structure:

.
├── backend
│   └── server.nim
├── frontend
│   ...
└── server.nimble

The nimble file will need to build the frontend first:

srcDir        = "backend"
bin           = @["server"]
backend       = "c"

requires "nim >= 2.0.0"

requires "karax#head"
requires "mummy >= 0.3.5"

before build:
  withDir "frontend":
    exec "nimble build"

We’ll also pull in mummy and karax. The backend for this is c (or any other native backend you want).

Statically including the Frontend Code

I prefer to do this, you may also choose to read it from disk or serve it from a CDN:

import std / [
  strutils,
  strformat
]

import mummy, mummy/routers
import karax / [karaxdsl, vdom, vstyles]

const appJs = "./frontend/app.dist.js".readFile

Rendering the HTML

First, let’s build the <head> section:

proc rHead(): VNode =
  buildHtml head:
    meta(charset = "UTF-8")
    meta(
      name = "viewport", 
      content = @[
        "width=device-width", 
        "initial-scale=1", 
        "maximum-scale=1", 
        "user-scalable=0",
      ].join(", ")
    )
    meta("http-equiv" = "X-UA-Compatible", content = "ie=edge")
    script(
      src = "https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js",
      "defer" = ""
    )

The various meta directives are the minimum required to make sure our page renders fine on all device sizes with sensible defaults.

We also import the Alpine release. The defer attribute causes the browser to fetch the remote script immediately after parsing the <script> tag, but delay execution until after the entire HTML has been parsed. This is important for our init directives to fire deterministically. Usually boolean attributes are activated like attrib = true in karax, but since it conflicts with the built-in Nim keyword we need to use the string version.

Now let’s build the <body>:

proc rBody(): VNode =
  buildHtml body:
    tdiv("x-data" = "dropdown(true)"):
      button("x-bind" = "dropbutton"):
        text "Toggle Me"
      tdiv("x-show" = "open"):
        p: text "I'm inside!"

    verbatim &"""
      <script>
        {appJs}
      </script>
    """

A few notes:

tdiv is karax’ way of avoiding conflicts with the Nim operator div
x-data is the component initializer, also passing the initial open state to our new dropdown
x-bind means the <button> will inherit all behavior from the binding we created in Nim
x-show ensures the inner paragraph is only shown when the open property is true (or truthy)

verbatim includes the bundled frontend code inline. Again, this is a personal preference (avoid another request), you don’t have to do this.

Now emit the entire HTML file as a string:

proc home*(): string =
  let vnode = buildHtml html(lang = "en"):
    rHead()
    rBody()

  result = "<!DOCTYPE html>\n" & $vnode

Serve it

The very last thing to do is to serve this via mummy:

proc indexHandler(request: Request) =
  var headers: HttpHeaders
  headers["Content-Type"] = "text/html; charset=utf-8"
  request.respond(200, headers, home())

var router: Router
router.get("/", indexHandler)

let server = newServer(router)
echo "Serving on http://localhost:8080"
server.serve(Port(8080))

Conclusion

After nimble run, we can view the result at http://localhost:8080:

You can download the complete example repo here.

Never Down: Implementing Auto Fallback in Caddy

Tue, 26 Dec 2023 00:00:00 +0000

I host lots of private (and a few public) projects behind Caddy. I do like its simplicity and obvious happy path. Something I did struggle with at the start of my journey with Caddy, and something I routinely forget again, is how to set up a fallback page. This is a good idea if there’s a single-instanced service that might be taken offline for a few moments due to scheduled or unscheduled maintenance. The solution is to create a second, static upstream.

Let’s take a look at the Caddyfile for Findsight:

:9999 {
	respond "We'll be right back (usually within 3 minutes)!"
}

www.findsight.ai {
	redir https://findsight.ai{uri}
}

findsight.ai {
	handle {
		reverse_proxy localhost:8080 :9999 {
			lb_policy first
			lb_try_duration 1s
			fail_duration 30s
			transport http {
				dial_timeout 500ms
			}
		}
	}
}

Here’s what’s happening:

In normal operation, Caddy reverse proxies requests to localhost:8080 where the main Findsight app listens. The entry point is a domain (findsight.ai), and Caddy will automatically handle certificate provisioning and TLS termination. Additionally, there’s a request rewrite for the www subdomain.

If localhost:8080 would be the only upstream defined and the server goes down (e.g. for database migration) Cloudflare, which is the CDN for Findsight, would display a generic “upstream dead” error and that’s not a good look. It leaves the user guessing: why is it down? When will it be back?

We need to define a fallback. If Caddy considers an entry for reverse_proxy to be dead, it moves on to the next one. That behavior is defined by the lb (load balancing) policy first. Dead upstreams remain dead for 30s. This duration can be tuned to taste, I chose it mainly to avoid requests hitting the server during start-up after a crash, when the port is reserved but requests won’t be processed yet. Now requests hit the internal :9999 upstream, which serves a static response directly from Caddy. Here one could alternatively display a full HTML page.

GitHub code scanning autofix demo

Thu, 30 Nov 2023 00:00:00 +0000

A 5 minute demo of GitHub code scanning and its new AI-powered automatic fix suggestions. Learn more about this feature or request early access.

Building a Hybrid/AI Search Engine for Non-fiction Content

Tue, 18 Apr 2023 00:00:00 +0000

Originally presented at an internal GitHub event, this talk takes you behind the scenes of Findsight AI and it’s three-tier search architecture. Along the way you’ll learn how to implement search across traditional LER/NER tags, how to build a safe interface for SQLite’s FTS5 full text search, and how to optimize retrieval based on embeddings.

Introducing: Findsight

Wed, 05 Apr 2023 00:00:00 +0000

I just launched Findsight AI, available for free on findsight.ai. Findsight helps you discover non-fiction sources like books, articles, papers, and more by comparing their core claims and ideas.

Join more than 10,000 users, and try it right now on findsight.ai, no sign-up required!

What is Findsight?
What’s unique about Findsight?
How does it work?
Notes

What is Findsight?

Findsight is a search engine that lets you explore non-fiction content based on claims — the key points made by sources. You can discover and compare claims from various sources, see how authors tackle issues differently, and navigate to related claims to create your own learning journey. You can also find links to the original book or article. This approach is called syntopical reading.

Findsight combines traditional text and tag search with AI-powered filters that let you find related claims, and even claims that answer questions.

Here’s an example of a result:

Clicking on the title of a work expands this context view. In addition to the highlighted claims (the ones matching the search), you’ll get to see a few more claims surrounding it to get a sense of the narrative of the source. The real fun starts with the explore feature: click the arrow to jump from a claim to related claims. There’s always something new to learn by starting from Random and following a new path.

Findsight’s AI-powered filters enable you to find claims by meaning. Here’s an example of the “answer” filter identifying data that supports the user’s question:

You can find a detailed explanation of Findsight’s search capabilities in the Search Guide on findsight.ai.

What’s unique about Findsight?

Generative AI has inspired exceptional knowledge retrieval tools, but Findsight’s core priorities make it unique:

Top-notch summarization: Findsight employs cutting-edge large language models for high-quality summarization, extracting key claims while minimizing AI-generated inaccuracies — faithfully reflecting the source’s intent.

Robust hybrid search: With a custom search engine tailored for non-fiction, Findsight melds traditional text search and AI-powered retrieval to deliver highly relevant results in most cases.

Discovery, not consumption: Findsight isn’t about consumption; it’s a launchpad to find insights and collect books or articles to explore further. Since a collection of claims can’t capture a source’s entire content, we limit the context returned in searches.

That said, here’s a quick list of related services¹ that you can combine with Findsight to level up your learning:

Elicit: AI research assistant with a rich search interface for common attributes of scientific papers
Shepherd: High-quality human-curated book suggestions with a sprinkle of AI-powered topic exploration
Basmo: Polished app experience for interacting with books using AI chatbots, and share knowledge with other users
Librarian AI: Discover new books based on ones you already have
ChatGPT Retrieval: Early-stage project to enable knowledge retrieval with OpenAI’s ChatGPT

Disclaimer: Findsight/Summarity has no affiliation with any of the projects mentioned above. Follow links at your own risk. Not an endorsement.

How does it work?

This is a highly condensed overview of the tech stack behind Findsight - you don’t need to know this to use it. If you’re interested in the implementation details on any of the parts below, drop me an email and I’ll consider writing a separate post.

Findsight’s AI features are powered by OpenAI’s GPT3, GPT3.5 completion models and Ada embeddings, combined with fine-tuned Azure LER/NER models.

Beyond those services, Findsight is a depencency-free application written in Nim. The larger components are:

a custom OpenAI client with semantic caching and automatic HyDE prompting
an extraction module that automatically cleans and chunks input text, and adapts prompts and chunk size based on self-supervision
a custom KNN / vector-similarity index based on SuperBit encoding and using Weave for data-parallel execution
a hybrid search engine that uses three stages with content-dependent boosting:
- Alpha: binary attribute filtering (e.g. tags)
- Beta: BM25-ranked full-text search using a custom search syntax (SafeTS5) that compiles to provably safe (limited complexity) FTS5 queries (uses NPEG)
- Gamma: a ranked similarity search based on the KNN index, using potentially present Alpha and Beta stages as a pushdown filter
a book metadata engine that identifies and normalizes ISBN catalog data based on book content
a mummy-based server using SQLite for persistent storage

Notes

If you are working on a related project, or spotted a mistake in this list, please get in touch (email in footer) ↩

SQLite-only Rate Limiting

Sun, 02 Apr 2023 00:00:00 +0000

Application-layer rate limiting is a good way to enforce client resource limits, while at the same time allowing you to place the rate limiting configuration and current statistics next to the application data.

This approach should not be employed for defending against application layer or network layer (D)DoS attacks; rather, such defense should be pushed towards the edge, utilizing Web Application Firewalls or analogous policy agents that can drop suspicious traffic with minimal overhead.

In contrast, this approach is more suitable for enforcing fair use limits or limits that are part of a license, which restricts the number of operations an actor can perform.

SQLite is used to:

store the limits configuration, each with a maximum count per configurable time period
store instances of active counters for actors (we’ll use “visitors” for this example)
manage expiration

Interface

The goal is that with a single query executed for every operation, rate limiting calculations are performed automatically:

INSERT INTO visitor_limits(visitor_id, limit_id)
VALUES (1, 1)
ON CONFLICT DO UPDATE SET count = count + 1
RETURNING denied, resets_at;

We inform the database that visitor 1 is attempting to perform an operation limited by limit 1.
If the limit is not yet reached, the current counter is incremented
If the limit is reached, denied will be 1 (SQLite has no booleans), and resets_at is defined as the next UTC unix epoch timestamp after which the limit resets. resets_at is not defined when the request isn’t denied

This allows the counter to be incremented by other positive integer values, for example, if the “cost” between operations varies, which is useful for consumption-based enforcement.

Database Setup

Limits

The schema for this is quite simple. First, we define a static configuration table for the limits:

CREATE TABLE IF NOT EXISTS limits (
  id INTEGER PRIMARY KEY ASC
  ,name TEXT UNIQUE NOT NULL
  ,maximum INTEGER NOT NULL CHECK ( maximum > 0 )
  ,period_minutes INTEGER NOT NULL CHECK ( period_minutes >= 1 )

  ,UNIQUE (name, maximum)
) STRICT;

name uniquely identifies the limit, this should be the limited operation, e.g. send_message
maximum is the maximum number of allowed operations within the period. This is inclusive, so a maximum of 5 allows 5 requests in a period and denies the 6th
period_minutes defines the time between counter resets.

The period can alternatively be expressed in seconds, which is the highest usable time resolution in SQLite.

To add a new limit:

-- allow 5 messages per 2 minutes
INSERT INTO limits(name, maximum, period_minutes)
VALUES ('send_message', 5, 2);

Counters

Next, create a table for the counter instances. These will be lazily populated - if a visitor never executes a given operation, there’s no need to create a row for it:

CREATE TABLE IF NOT EXISTS visitor_limits (
  id INTEGER PRIMARY KEY ASC
  ,visitor_id INTEGER NOT NULL REFERENCES visitors(id)
  ,limit_id INTEGER NOT NULL REFERENCES limits(id)

  ,denied INTEGER NOT NULL DEFAULT 0 CHECK ( denied IN (0, 1) )

  -- set by trigger
  ,count INTEGER CHECK ( count >= 0 )
  ,resets_at INTEGER CHECK ( resets_at > 1680373723 )

  ,UNIQUE (visitor_id, limit_id)
) STRICT;

Note: this references another table visitors, this may not be the case in your application. Use whichever actor identifier fits your model best.

A sanity check is added to resets_at to protect against logic bugs, using a reasonably recent timestamp and making it clear in the DDL that this column will be a unix epoch in seconds.

Triggers

To implement the add and reset logic, two triggers are required:

CREATE TRIGGER IF NOT EXISTS init_visitor_limit_counter
AFTER INSERT
ON visitor_limits
WHEN (NEW.count IS NULL OR NEW.resets_at IS NULL)
BEGIN
  UPDATE visitor_limits
    SET
      count = 1
      ,resets_at = (unixepoch() + (60 * (SELECT period_minutes FROM limits l WHERE l.id = NEW.limit_id)))
      ,denied = 0
  WHERE id = NEW.id;
END;

This runs after the INSERT statement in our final interface query. If there wasn’t a counter for this operation and visitor, we initialize the counter, flag, and first reset timestamp.

For any INSERTs after this, we need to maintain the flag and reset the counter if necessary using a second trigger:

CREATE TRIGGER IF NOT EXISTS update_visitor_limit_counter_breaker
BEFORE INSERT
ON visitor_limits
BEGIN
  -- trip breaker for non-expired
  UPDATE visitor_limits
    SET denied = visitor_limits.count >= (SELECT maximum FROM limits l WHERE l.id = NEW.limit_id)
  WHERE visitor_limits.visitor_id = NEW.visitor_id
    AND visitor_limits.limit_id = NEW.limit_id;

  -- or reset if expired
  UPDATE visitor_limits
    SET
      count = 0
      ,resets_at = unixepoch() + (60 * (SELECT period_minutes FROM limits l WHERE l.id = NEW.limit_id))
      ,denied = 0
  WHERE visitor_limits.visitor_id = NEW.visitor_id
    AND visitor_limits.limit_id = NEW.limit_id
    AND unixepoch() > visitor_limits.resets_at;
END;

This runs before the INSERT to make sure the denied flag that is returned is up to date. In our example, the 6th message should be denied, not the 7th.

Testing

That’s it. We can test this now by running the same interface query repeatedly:

INSERT INTO visitor_limits(visitor_id, limit_id)
VALUES (1, 1)
ON CONFLICT DO UPDATE SET count = count + 1
RETURNING denied, resets_at;

-- send 6 messages in < 2 minutes:

-- Run 1 returns: 0, NULL
-- Run 2, 3, 4, 5 return: 0, 1680441289
-- Run 6 returns: 1, 1680441289

-- ... wait for 2 minutes ...

-- Run 7 returns: 0, 1680441422

As the limit is reached, the counter will continue to increment; which can be leveraged to gain insights into the behavior of limited users. A minor overage is to be expected, but a drastic surge in requests beyond the set limit could signify a bug or the presence of a malicious bot.

You're using the wrong Apple (Silicon) Cores

Sun, 12 Mar 2023 00:00:00 +0000

With the disruptive arrival of the Apple Silicon Macs, the industry saw a push away from TDP-be-damned space heater hardware to more sensible performance-per-watt targets. The Apple M1, M2 and its mobile counterparts all feature two main general purpose core types: performance and efficiency. Intel’s recent offerings are positively stuffed full of efficiency cores, and other players are following suit.

That however poses a challenge for developers. It’s now harder than ever to light up all those transistors and produce efficient software:

Race to sleep: The faster your software runs, the less power it consumes. […] We’re trying to actually engage the entire chip, the entire piece of hardware to do a computation, in order to do that computation as rapidly and effectively as possible.

– Chandler Carruth, “Efficiency with Algorithms, Performance with Data Structures”

This is pretty good advice. However, simply saturating all cores (or threads) your program can access is no longer a good strategy, as modern silicon throws two wrenches in our works:

First, chip design tends to capture common algorithms in hardware, rendering entire classes of general-purpose-CPU implementations obsolete. Whereas you might have previously used optimized libraries for matrix multiplication, or even GPGPU, you’re now better off turning that work over to system frameworks and have them schedule it on specialized cores within the main chip. On Apple Silicon, there are no less than four different tuned hardware implementations for this purpose: CPU-bound ARM NEON, Apple AMX (a semi-secret matrix instruction set), Apple Neural Engine (dedicated few-big cores), and the GPU itself. ANE implementations for deep learning inference outperform implementations using Apple Accelerate (NEON/AMX) by almost 300%!

Matrix multiplication is still a pretty general use case though and more specific co-processors exist for media encoding and decoding, crypto, real-time image processing and more. But what if you do have a workload that isn’t a good fit for hardware acceleration? If your M1 Ultra reports 20 CPU cores, scheduling 20 threads should be able to use “the entire piece of hardware”?

Sure enough, you can do that and you will see 100% utilization across all cores (let’s assume your workload is data-parallel and your algorithm is optimal, so no time wasted on IO and cache swaps). But taking our earlier example of the 20-core M1, your program will run virtually as fast using 20 threads as it would using 16. Why is that?

The difference is caused by the efficiency cores. For normal software, which executes workloads that are both short-lived and small as well as longer-running intense workloads, using all threads isn’t a problem. The easy stuff can be scheduled on efficiency cores (e.g. keeping the UI responsive), while processing can be scheduled on the performance cores. That’s why e.g. Logic allows you to go all in:

But if all your threads are coordinating the same, intense workload, using both efficiency and performance cores will cause the clock-speed of performance cores to be dragged down. The consequences are almost non-existent gains of performance compared to just using the performance cores, and a risk of system lag as the system UI itself and background services are choked by the inappropriate load on the efficiency cores.

Instead, your program should be aware of the number of cores available at the appropriate performance level. On macOS, this can be queried using sysctl:

$ sysctl hw.perflevel0.logicalcpu_max
hw.perflevel0.logicalcpu_max: 8

Here perflevel0 is “performance”, and perflevel1 would be efficiency.

Hypothetical Embeddings Explained

Sat, 11 Mar 2023 00:00:00 +0000

Large Language Models (LLM) can be combined with embeddings to create accurate knowledge retrieval systems. Hypothetical Document Embeddings, or HyDE¹, can be used to provide a question-answering interface for a given database of facts that feels like an LLM-powered chatbot, but without the risk of hallucinations.

In LLM-only systems, such as search engines powered by RLHF-tuned models, there’s a risk of the core model confidently stating wrong facts or making assertions that go beyond or even contradict the available ground truths (e.g. a search index). This problem is more commonly known as hallucination.

There are several ways to engineer prompts to minimize this effect, and combined with instruction fine tuning are generally successful in limiting hallucinations. In systems where fact retrieval is the primary use case though, this problem can be almost entirely eliminated.

HyDE-powered search relies on embeddings, dense vector representations of (in this case) textual data. OpenAI embeddings for example, can effectively encode abstract qualities of short and long-form text (up to ~8,000 tokens per embedding) in ~1600-dimensional vectors. These vectors can then be stored in vector-similarity (VSS) indexes (e.g. HNSW/RediSearch) that facilitate efficient VSS or hybrid VSS and full-text queries.

However, these systems are then limited to retrieving similar facts. A user interface could prompt the user for a fact, which is then embedded and used as a query vector to retrieve the nearest (most similar) facts from the index. This requires the user to already have a possible answer in mind, which is much less intuitive than asking a question.

Instead of embedding the user input directly, a sufficiently trained LLM can be used to generated hypothetical answers, which are in turn embedded and used for retrieval. The LLM will likely hallucinate details for its answers, but since these answers are not relayed to the user as fact, these errors are inconsequential. The embedding is much more sensitive to the structure and language used in the answers, rather than its correctness.

An example prompt (here using LLaMa-65B) may look like this:

Three possible answers to the question 'what are the negative effects of the war on drugs?' are:
1.

with an example output:

Incarcerated people in the U.S. have exceeded two million and the great majority of the people in jail are there for drug related crimes. In many cases, these people don't get to go to school and learn the skills they need to lead healthy, productive lives.
People all over the world die daily, in violent confrontations, related to the trade of illegal drugs.
Millions of people, including children, are sick with diseases caused by using contaminated, illegal drugs. These people are spreading disease to others.

Depending on the performance of the embedding and the difference between the answers, one embedding per answer (then averaged) or one embedding for the entire answer list can be used. For non-fiction fact retrieval (excerpts from books), I found it helpful to include the question itself in the embedding for each HyDE, since this type of content often poses a question only to immediately answer it.

Notes

HyDE was first introduced in Precise Zero-Shot Dense Retrieval without Relevance Labels, where InstructGPT had been fine tuned to generate HyDE. ↩

Nim 2.0 on Apple Silicon

Sun, 15 Jan 2023 00:00:00 +0000

The Nim 2.0 release candidate is out, and includes a whole host of amazing changes and additions. I had been holding off on exploring Nim because I was waiting for ORC and multi-threading to become the default. They have been available for a long time, but defaults matter - it’s what first time and first-time-in-a-while users experience. It forms their opinion of what the language is about.

Installing Nim on Apple Silicon Macs is pretty straight forward. The official installer sets up choosenim and a default stable release:

curl https://nim-lang.org/choosenim/init.sh -sSf | sh

However, both Nim itself and the compiler output will be amd64 (Intel) binaries. To go native, and to benefit from Nim 2 right now, more configuration is required.

Set up Nim 2 natively

We need a source build, and thankfully that’s easy:

git clone https://github.com/nim-lang/Nim
cd Nim
git checkout version-2-0
sh build_all.sh

Yep, that’s it. We don’t have to mess with our PATH either. Instead, we can now point choosenim at the new build:

choosenim <location>/Nim

And confirm the architecture with:

$ nim -v
Nim Compiler Version 1.9.1 [MacOSX: arm64]
Compiled at 2023-01-14
Copyright (c) 2006-2022 by Andreas Rumpf

(2.0 RC self-reports as 1.9.1 as of Jan 15 2023)

To switch back, use choosenim stable.

Native & universal binaries

Building a project with this toolchain will produce an Intel binary. To get a full build on macOS, we have to follow Apple’s guidelines and set the correct compiler and linker targets. In your project’s nimble (nimble init if you don’t have one yet), add a new target:

task release_clang, "Build a production release (macOS)":
  --verbose
  --forceBuild:on
  --cc:clang
  --define:release
  --deepcopy:on
  --cpu:arm64
  --passC:"-flto -target arm64-apple-macos11" 
  --passL:"-flto -target arm64-apple-macos11"
  --hints:off
  --outdir:"."
  setCommand "c", "hello.nim"

-flto is required because clang doesn’t like Nim’s default passing of LTO parameters
note the cpu and -target parameters
the rest are standard release flags

ORC and threads are default now, so no need to enable them here anymore. It’s a good idea to pin the Nim version in the nimble file, in this case to >= 1.9.1.

Running nimble release_clang now gives us a nice native binary:

$ file hello
hello: Mach-O 64-bit executable arm64

If you want to build an Intel binary, add another task and change the target to x86_64-apple-macos10.12. If you need a universal binary, build both the Intel and ARM targets first, then merge them using lipo:

lipo -create -output universal_app x86_app arm_app

BTW: If you’re wondering what the cover image is about - “Nimm 2” (“take two”) is a popular brand of gummy candy in Germany.

Baking Vertex Animations into Alembic Files

Sat, 31 Dec 2022 00:00:00 +0000

This continues the series on baking dynamics. Last time, we looked at baking rigid body dynamics into FBX f-curves, for rendering in applications that don’t support dynamics (such as mo2).

In that post, I mentioned that baking soft-body dynamics is a whole lot harder. There are several reasons for that:

Soft body deformations rely on per-vertex transformations
Apart from Alembic, there are no standard ways of transporting point-based animations between 3D software
Even then, some simulations (e.g. fluids) may change the vertex and thus poly count, something barely any software can deal with

This problem isn’t new, and the industry had come up with two methods to address this:

Portable dynamics bake caches (take a look at the previous post for an explanation of dynamics baking). This is a way of transporting the internal bake cache between applications without “rendering” the transformations to a mesh. Notable software includes e.g. ChronoSculpt, which can work with three major cache formats (incl. Alembic)
The brute force method: Bake a new mesh for every single frame. This is supported by many apps as “OBJ sequences” or “mesh sequences”, notably AE Element3D. As the name implies, the export format is a folder full of single OBJ files, one for each tick (a frame). The benefit of this method is the wide compatibility: everything supports OBJs (though not necessarily sequences), and changing poly counts are a non-issue since we exchange the entire mesh every tick.

So mesh sequences address a similar problem to Alembic, but how can we convert between them? I needed to do this recently, in order to transport soft-body physics baked in Cheetah to Marmoset Toolbag 4 for rendering. Cheetah supports nothing - it stores its bake cache in the main scene file but the format is undocumented, and there isn’t any support for exporting Alembics. Toolbag on the other hand only supports Alembic, but not mesh sequences.

The first problem was easy enough to address, and I won’t go into much detail because you’re likely not using Cheetah and are here to learn how to do the conversion from sequences to Alembic. In short, there’s both an import and an export script for OBJ sequences for Cheetah, created by forum users.

Baking and exporting leaves us with a folder full of OBJs:

Now we need to convert those into one Alembic file. I’ve searched a while for a simple way to do this, but I’m afraid there isn’t one. We have to involve another piece of software, but thankfully it’s free: good old Blender.

Users of Blender have tried to solve this problem in the past, and a few have had success with slightly abusing the Stop Motion OBJ plugin, though that workaround seems to no longer work in Blender 3.

Enter Barnaba Mikułowski, who graciously provided a plugin-free conversion setup based on a geometry node network in Blender. I’ve adapted his original file to be a more generic template, and you can download it here (3 MB).

Open the Blender scene (I used Blender 3, I recommend you do the same). Click to highlight the “Import Sequence Here” collection and enable visibility if required.

Go to File -> Import -> Wavefront (.obj) – don’t use the legacy option

Navigate to your mesh sequence folder and click-shift-select all files that are part of the sequence (make sure the naming produces proper ordering!)

Click import, you should now see all OBJs imported as children of the import collection, in proper order, and rendered in the viewport.

To clean up the viewport, untick the box on the collection object to re-exclude it from the viewport. In the bake network, confirm the input is set to the import collection. Adjust the Blender animation timeline to fit your imported sequence (start and end frames can be set at the top right of the keyframe panel, and FPS can be set in the output properties which can be found behind the printer icon in the vertical icon bar on the right between the viewports and the inspector). Click the play button to preview your animation and check everything’s in order.

Now select (!) only the bake network in the object tree and go to File -> Export -> Alembic (.abc). Change the scene options if required, but make sure “Only: Selected Objects” is ticked!

Now, the file can be imported into any package that supports Alembic: