\documentclass[12pt]{article}
\usepackage[T2A,OT1]{fontenc}
\usepackage[default]{cantarell}
\usepackage[a4paper, top=20mm, bottom=20mm, left=20mm, right=20mm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[russian, english]{babel}
\usepackage{tabu}
\usepackage{hyperref}
\usepackage{parskip}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage[normalem]{ulem}
\usepackage{float}
\floatstyle{boxed}
\restylefloat{figure}
\usepackage{setspace}
\onehalfspacing
\author{Artyom Bologov \href{mailto:cl-submodules@aartaka.me}{(email)}}
\date{\today}
\title{Common Lisp Dependency Vendoring with Submodules}
\makeatletter
\def\endenv{\expandafter\end\expandafter{\@currenvir}}
\makeatother
\begin{document}
\maketitle

\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{./assets/cl-submodules.png}

So there are half a dozen of Common Lisp package / system / project managers:

\begin{itemize}\item \href{https://quicklisp.org}{Quicklisp} (with Ultralisp distribution)
\item \href{https://clpm.dev}{CLPM}
\item \href{https://github.com/ocicl/ocicl}{OCICL}
\item \href{https://github.com/fukamachi/qlot}{Qlot}
\item \href{https://github.com/fosskers/vend}{Vend}
\item \href{https://github.com/CodyReichert/qi}{Qi}
\end{itemize}

These mostly fall into two categories:

\begin{enumerate}\item Package/project managers fetching things from the Internet right into the running Lisp image and not touching the project itself.
\item Project-local package managers that include all the dependencies into the project tree.
\end{enumerate}

Project-local solutions bloat the checkout size and slow down repository cloning.
Online ones require network connection.
I’m not satisfied with either of the trade-offs, and I think I have a solution to that.

Okay, I’ll need a more involved problem statement than that.
The features I’m looking for in my dependency management journey are:

\begin{itemize}\item Fully local build options
\item CLI/UNIX-friendliness, especially for running tests
\item Reproducibility, of a weaker kind—Lisp deps that don’t change underneath me
\item Not needing bloated Swiss knife binaries just to manage dependencies
\item Option to use both local deps and Quicklisp / Ultralisp / CLPM / whatever
\item Custom REPLs with dependencies pre-loaded
\item Not reinventing the wheel
\end{itemize}

\section*{Submodules} \label{submodules}

Git submodules are this kind of solution.

\begin{itemize}\item They allow the library developer to pin dependencies precisely
\item The library user only needs to fetch the dependencies they require, when they need them
\item There’s no need to install any heavyweight tools like Roswell
\item Repository size doesn’t grow unboundedly for those that don’t want it
\item It’s CI-friendly and well-integrated with the UNIX tooling
\end{itemize}

So my setup is:

\begin{itemize}\item Make a project/library and write the code interactively using Quicklisp for dependencies.
\item Once the library is more or less stable dependency-wise, run a 100 LoC
\href{https://codeberg.org/aartaka/lisp-config/src/branch/main/deps.lisp}{dependency-resolving script}
using Quicklisp metadata.
\item Run all the commands it suggests (it doesn’t run anything itself, by design!)
\item Add some leftover deps that Quicklisp doesn’t have or that are special in some other way.
\href{https://github.com/atlas-engineer/nyxt/blob/master/.gitmodules}{(I tend to use Nyxt’s .gitmodules to get dependency locations (seems like working on Nyxt left an imprint, huh?))}
\item And then set \verb|CL_SOURCE_REGISTRY| to the path with dependencies (usually repository root) when I need to run tests or a custom REPL.
\end{itemize}

Stripped-down project Makefile (yes, I use Make for Lisp projects) then looks like:

\begin{figure}[h!]\begin{verbatim}
LISP ?= sbcl
LISP_FLAGS ?= ...

export CL_SOURCE_REGISTRY ?= $(PWD)//

executable:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:make "system")' --eval '(uiop:quit)'

.PHONY: tests
tests:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:test-system "system")' --eval '(uiop:quit)'

.PHONY: repl
repl:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:load-system "system")'
\end{verbatim}\caption{Makefile for submodule vendoring approach}\end{figure}

Things to note:

\begin{description}\item[exported/shared CL_SOURCE_REGISTRY with double slash]
\href{https://reddit.com/r/Common_Lisp/comments/14it7ag/comment/jpi25rw}{That’s a vital hack mentioned by Alexander Artemenko on Reddit}.
  This double slash means a given directory is an ASDF “tree” to be searched for systems recursively.
  \href{https://asdf.common-lisp.dev/asdf/Shell_002dfriendly-syntax-for-configuration.html}{Here’s the syntax explanation in ASDF manual}.

\item[--eval '(require "asdf")' --eval '(asdf:load-system "system")']
All the bootstrapping is basically two expressions, all reliant on ASDF.
  Given that \verb|CL_SOURCE_REGISTRY| is set up properly, all the dependencies are there.
  Build, test, and launch REPLs all you want.
  No Quicklisp needed.

\item[repl]
REPL is the same call to \verb|$LISP|, but without the \verb|--eval '(quit)'|
  It’s both \verb|rlwrap| compatible and has all the necessary libraries accessible.

\item[Swank/Slynk integration]
Easy to support by
\href{https://codeberg.org/aartaka/cl-minifloats/src/commit/802b3be917125f8210b9d990f0075ebad30e9d22/cl-minifloats.asd#L32}{adding two more ASDF subsystems} and
\href{https://codeberg.org/aartaka/cl-minifloats/src/commit/802b3be917125f8210b9d990f0075ebad30e9d22/makefile#L19}{running them from the Makefile}.
\end{description}

\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{./assets/cl-submodules-benefits.png}

This setup gives me:

\begin{itemize}\item Ability to clone the project without dependencies for use with Quicklisp—\verb|git clone project-url|
\item Ability to clone the project \emph{with} all the dependencies—\verb|git clone --recursive project-url|
\item Ability to get only a subset of dependencies I need (say, for libraries not on Quicklisp yet)—\verb|git submodule update --init -- deps/something| or \verb|git submodule update --init -- deps/| to update them all
\item Global reuse of dependencies with custom ASDF registries (as in <code>.config/common-lisp/source-registry.conf.d/asdf.conf</code>)
\item Option to run tests from CLI and on CI with all the dependencies accessible to them
\item Precisely pinned dependencies possibly going fresher than Quicklisp
\item \href{run:customize-repl}{Custom REPLs} in the dev environment
\item Total Quicklisp / CLPM / OCICL independence, with the only hard project dependency being ASDF (and Quicklisp project repositories at vendoring time, but that’s minor and easy to have locally)
\item An option to use Quicklisp etc. nonetheless, just do <code>make CL_SOURCE_REGISTRY=path/to/systems//</code> and your dependencies are replacing the submodule ones
\end{itemize}

What not to like about it?

\section*{The Problem With ~/common-lisp} \label{home-common-lisp}

Thanks to Konrad Hinsen for making me think about that!
One scenario where this submodule-driven workflow might break is when you clone the repo into \verb|~/common-lisp|.
And then there’s another version of one of the deps in there, provoking a version clash.

Cloning into \verb|~/common-lisp| is a bad practice that should be eradicated in favor of properly compartmentalized registries.
But alas.
Anyway, if you clone into \verb|~/common-lisp| or <code>~/.local/share/common-lisp/source/</code>, then do a non-recursive clone instead.
If you cloned the library there, you likely have all the dependencies available already.
In the very same \verb|~/common-lisp|, no doubt.
So no need for submodules.

But better not use \verb|~/common-lisp| at all, global state bad.

\par\noindent\rule{\textwidth}{0.4pt}

To see how the whole setup around submodules looks, you can check out
\href{https://codeberg.org/aartaka/quickproject-template}{my Quickproject template}
and its
\href{https://codeberg.org/aartaka/quickproject-template/src/branch/main/makefile}{makefile} and
\href{https://codeberg.org/aartaka/quickproject-template/src/branch/main/system.asd}{.asd file}.


\par\noindent\rule{\textwidth}{0.4pt}
\href{https://creativecommons.org/licenses/by/4.0}{CC-BY 4.0} 2022-2026 by Artyom Bologov (aartaka,)
\href{https://codeberg.org/aartaka/pages/commit/a91befa}{with one commit remixing Claude-generated code}.
Any and all opinions listed here are my own and not representative of my employers; future, past and present.
\end{document}