.TH cl-submodules 7
.SH NAME
cl-submodules \- Common Lisp Dependency Vendoring with Submodules
.SH AUTHOR
Artyom Bologov
.SH SYNOPSIS
Submodules give you the flexibility to fetch the dependencies, or not. And they enable more granular reproducible builds. Use submodules!
.SH TEXT


.P
So there are half a dozen of Common Lisp package / system / project managers:

.P
*  
.UR https://quicklisp.org
Quicklisp
.UE
 (with Ultralisp distribution)
.P
* 
.UR https://clpm.dev
CLPM
.UE
.P
* 
.UR https://github.com/ocicl/ocicl
OCICL
.UE
.P
* 
.UR https://github.com/fukamachi/qlot
Qlot
.UE
.P
* 
.UR https://github.com/fosskers/vend
Vend
.UE
.P
* 
.UR https://github.com/CodyReichert/qi
Qi
.UE

.P
These mostly fall into two categories:

.P
* Package/project managers fetching things from the Internet right into the running Lisp image and not touching the project itself.
.P
* Project-local package managers that include all the dependencies into the project tree.

.P
Project-local solutions bloat the checkout size and slow down repository cloning.
Online ones require network connection.
I’m not satisfied with either of the trade-offs, and I think I have a solution to that.

.P
Okay, I’ll need a more involved problem statement than that.
The features I’m looking for in my dependency management journey are:

.P
* Fully local build options
.P
* CLI/UNIX-friendliness, especially for running tests
.P
* Reproducibility, of a weaker kind—Lisp deps that don’t change underneath me
.P
* Not needing bloated Swiss knife binaries just to manage dependencies
.P
* Option to use both local deps and Quicklisp / Ultralisp / CLPM / whatever
.P
* Custom REPLs with dependencies pre-loaded
.P
* Not reinventing the wheel

.SH Submodules

.P
Git submodules are this kind of solution.

.P
* They allow the library developer to pin dependencies precisely
.P
* The library user only needs to fetch the dependencies they require, when they need them
.P
* There’s no need to install any heavyweight tools like Roswell
.P
* Repository size doesn’t grow unboundedly for those that don’t want it
.P
* It’s CI-friendly and well-integrated with the UNIX tooling

.P
So my setup is:

.P
* Make a project/library and write the code interactively using Quicklisp for dependencies.
.P
* Once the library is more or less stable dependency-wise, run a 100 LoC

.UR https://codeberg.org/aartaka/lisp-config/src/branch/main/deps.lisp
dependency-resolving script
.UE
using Quicklisp metadata.
.P
* Run all the commands it suggests (it doesn’t run anything itself, by design!)
.P
* Add some leftover deps that Quicklisp doesn’t have or that are special in some other way.

.UR https://github.com/atlas-engineer/nyxt/blob/master/.gitmodules
(I tend to use Nyxt’s .gitmodules to get dependency locations (seems like working on Nyxt left an imprint, huh?))
.UE
.P
* And then set \fBCL_SOURCE_REGISTRY\fP to the path with dependencies (usually repository root) when I need to run tests or a custom REPL.

.P
Stripped-down project Makefile (yes, I use Make for Lisp projects) then looks like:

.P
.in +4n
.EX
LISP ?= sbcl
LISP_FLAGS ?= ...

export CL_SOURCE_REGISTRY ?= $(PWD)//

executable:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:make "system")' --eval '(uiop:quit)'

.PHONY: tests
tests:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:test-system "system")' --eval '(uiop:quit)'

.PHONY: repl
repl:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:load-system "system")'
.EE
.in
.P
— Makefile for submodule vendoring approach

Things to note:

.TP
.B exported/shared CL_SOURCE_REGISTRY with double slash

.UR https://reddit.com/r/Common_Lisp/comments/14it7ag/comment/jpi25rw
That’s a vital hack mentioned by Alexander Artemenko on Reddit
.UE
.
  This double slash means a given directory is an ASDF “tree” to be searched for systems recursively.
  
.UR https://asdf.common-lisp.dev/asdf/Shell_002dfriendly-syntax-for-configuration.html
Here’s the syntax explanation in ASDF manual
.UE
.

.TP
.B --eval '(require "asdf")' --eval '(asdf:load-system "system")'
All the bootstrapping is basically two expressions, all reliant on ASDF.
  Given that \fBCL_SOURCE_REGISTRY\fP is set up properly, all the dependencies are there.
  Build, test, and launch REPLs all you want.
  No Quicklisp needed.

.TP
.B repl
REPL is the same call to \fB$LISP\fP, but without the \fB--eval '(quit)'\fP
  It’s both \fBrlwrap\fP compatible and has all the necessary libraries accessible.

.TP
.B Swank/Slynk integration
Easy to support by

.UR https://codeberg.org/aartaka/cl-minifloats/src/commit/802b3be917125f8210b9d990f0075ebad30e9d22/cl-minifloats.asd#L32
adding two more ASDF subsystems
.UE
 and

.UR https://codeberg.org/aartaka/cl-minifloats/src/commit/802b3be917125f8210b9d990f0075ebad30e9d22/makefile#L19
running them from the Makefile
.UE
.


.P
This setup gives me:

.P
* Ability to clone the project without dependencies for use with Quicklisp—\fBgit clone project-url\fP
.P
* Ability to clone the project \fIwith\fP all the dependencies—\fBgit clone --recursive project-url\fP
.P
* Ability to get only a subset of dependencies I need (say, for libraries not on Quicklisp yet)—\fBgit submodule update --init -- deps/something\fP or \fBgit submodule update --init -- deps/\fP to update them all
.P
* Global reuse of dependencies with custom ASDF registries (as in <code>.config/common-lisp/source-registry.conf.d/asdf.conf</code>)
.P
* Option to run tests from CLI and on CI with all the dependencies accessible to them
.P
* Precisely pinned dependencies possibly going fresher than Quicklisp
.P
* \fBcustomize-repl (7)\fP
Custom REPLs in the dev environment
.P
* Total Quicklisp / CLPM / OCICL independence, with the only hard project dependency being ASDF (and Quicklisp project repositories at vendoring time, but that’s minor and easy to have locally)
.P
* An option to use Quicklisp etc. nonetheless, just do <code>make CL_SOURCE_REGISTRY=path/to/systems//</code> and your dependencies are replacing the submodule ones

.P
What not to like about it?

.SH The Problem With ~/common-lisp

.P
Thanks to Konrad Hinsen for making me think about that!
One scenario where this submodule-driven workflow might break is when you clone the repo into \fB~/common-lisp\fP.
And then there’s another version of one of the deps in there, provoking a version clash.

.P
Cloning into \fB~/common-lisp\fP is a bad practice that should be eradicated in favor of properly compartmentalized registries.
But alas.
Anyway, if you clone into \fB~/common-lisp\fP or <code>~/.local/share/common-lisp/source/</code>, then do a non-recursive clone instead.
If you cloned the library there, you likely have all the dependencies available already.
In the very same \fB~/common-lisp\fP, no doubt.
So no need for submodules.

.P
But better not use \fB~/common-lisp\fP at all, global state bad.

================================================================================

.P
To see how the whole setup around submodules looks, you can check out

.UR https://codeberg.org/aartaka/quickproject-template
my Quickproject template
.UE
and its

.UR https://codeberg.org/aartaka/quickproject-template/src/branch/main/makefile
makefile
.UE
 and

.UR https://codeberg.org/aartaka/quickproject-template/src/branch/main/system.asd
.asd file
.UE
.


.SH COPYRIGHT

.UR https://creativecommons.org/licenses/by/4.0
CC-BY 4.0
.UE
 2022-2026 by Artyom Bologov (aartaka,)

.UR https://codeberg.org/aartaka/pages/commit/a91befa
with one commit remixing Claude-generated code
.UE
.
Any and all opinions listed here are my own and not representative of my employers; future, past and present.