Chris von Csefalvay

The awesome thing about language is that, well, we all mostly speak it,
to some extent or another. This gives us an immensely powerful tool to
manipulate transformational tasks. For the purposes of this post, I
consider a transformational task to be essentially anything that takes
an input and is largel intended to return some version of the same
thing. This is not a very precise definition, but it will have to do for
now.

Such models are nothing new. Perhaps the most eye-catching and
ubiquitous of such models are [neural style
transfer](https://www.tensorflow.org/tutorials/generative/style_transfer)
models that take an image and return a version of the same image in a
different style that let you turn a picture of your dog into a Van Gogh
painting [(Gatys, Ecker, and Bethge 2015)]{.citation
cites="gatys2015neural"}. Quite simply put, these models are your
typical generative model, with the difference that it takes two separate
loss definitions: content loss, which is loss of the generated image
vis-a-vis the content reference, and style loss, which is the loss
vis-a-vis the style reference image. A "good" image then is one that
minimises total loss, i.e. it's just as close to your dog as it is to
Van Gogh. Figure 1 outlines this logic.

::: {.no-row-height .column-margin .column-container}
::: {#ref-gatys2015neural .csl-entry}
Gatys, Leon A, Alexander S Ecker, and Matthias Bethge. 2015. 'A Neural
Algorithm of Artistic Style'. *arXiv Preprint arXiv:1508.06576*.
:::
:::

::: {.cell layout-align="center"}
::: {.cell-output-display}
::: {#fig-nst .quarto-figure .quarto-figure-center .anchored}
<figure class="figure">
<div>
<pre class="mermaid mermaid-js" data-label="fig-nst"><code>flowchart TD
    C[&quot;Content image&quot;]
    S[&quot;Style image&quot;]

    G[&quot;Generated image&quot;]

    G --&gt; L[&quot;Loss network&quot;]
    L --&gt; G
    
    C -- &quot;Content loss&quot; --&gt; L
    S -- &quot;Style loss&quot; --&gt; L</code></pre>
</div>
<figcaption>Figure 1: A rough outline of NST.</figcaption>
</figure>
:::
:::
:::

Given a content source image
![](https://latex.codecogs.com/png.latex?%5Cvec%7Bc%7D) and a style
reference image ![](https://latex.codecogs.com/png.latex?%5Cvec%7Bs%7D),
we define the total loss of our generated image
![](https://latex.codecogs.com/png.latex?%5Cvec%7Bg%7D) as

![](https://latex.codecogs.com/png.latex?%0A%20%20%20%20%5Cmathcal%7BL%7D_%7Btotal%7D%20=%20%5Calpha%20%5Cmathcal%7BL%7D_%7Bcontent%7D%20+%20%5Cbeta%20%5Cmathcal%7BL%7D_%7Bstyle%7D%0A)

where ![](https://latex.codecogs.com/png.latex?%5Calpha) and
![](https://latex.codecogs.com/png.latex?%5Cbeta) are hyperparameters
that control the relative importance of the content and style losses.
The content loss for layer ![](https://latex.codecogs.com/png.latex?l)
is defined as

![](https://latex.codecogs.com/png.latex?%0A%20%20%20%20%5Cmathcal%7BL%7D_%7Bcontent%7D(%5Cvec%7Bc%7D,%20%5Cvec%7Bg%7D,%20l)%20=%20%5Cfrac%7B1%7D%7B2%7D%20%5Csum_%7Bi,j%7D%20(%5Cvec%7Bc%7D_%7Bij%7D%5El%20-%20%5Cvec%7Bg%7D_%7Bij%7D%5El)%5E2%0A)

which is basically a simple squared error loss between the feature
vector of the content image and the generated image at layer
![](https://latex.codecogs.com/png.latex?l). The style loss is a bit
more complicated, and is these days typically defined as the Maximum
Mean Discrepancy, which [Li et al. (2017)]{.citation
cites="li2017demystifying"} have shown is essentially equivalent to the
Gram matrix loss, defined as

::: {.no-row-height .column-margin .column-container}
::: {#ref-li2017demystifying .csl-entry}
Li, Yanghao, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. 2017.
'Demystifying Neural Style Transfer'.
<https://arxiv.org/abs/1701.01036>.
:::
:::

![](https://latex.codecogs.com/png.latex?%0A%20%20%20%20%5Cmathcal%7BL%7D_%7Bstyle%7D(%5Cvec%7Bs%7D,%20%5Cvec%7Bg%7D,%20l)%20=%20%5Cfrac%7B1%7D%7B4N_l%5E2M_l%5E2%7D%20%5Csum_%7Bi,j%7D%20(%5Cmathbf%7BG%7D_%7Bij%7D%5El%20-%20%5Cmathbf%7BS%7D_%7Bij%7D%5El)%5E2%0A)

where
![](https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D_%7Bij%7D%5El)
and
![](https://latex.codecogs.com/png.latex?%5Cmathbf%7BS%7D_%7Bij%7D%5El)
are the Gram matrices of the generated reference image and the style
image at layer ![](https://latex.codecogs.com/png.latex?l),
respectively. ![](https://latex.codecogs.com/png.latex?N%5El) is the
number of feature maps in layer
![](https://latex.codecogs.com/png.latex?l) and
![](https://latex.codecogs.com/png.latex?M_l) is the dimensionality
(height times width) of the feature map of layer
![](https://latex.codecogs.com/png.latex?l). There are two fundamental
problems with this.

1.  This works much less well for things that aren't images or at least
    sufficiently similar to images.
2.  The bigger problem is that the style reference is pretty much
    exhaustive. By that, I mean that there isn't much we can convey to
    the model about the style that isn't encapsulated in the style
    reference images. Depending on how semantically apt your model is,
    it may or may not be able to pick up some higher level ideas. It may
    be able to pick up the brush strokes of Van Gogh or the colours of a
    Turner, but it may not be able to paint your characters in the
    semantic context of Van Gogh's time and place.

This is where language comes in. Language is a very powerful tool for
conveying information, and it turns out that if we use language as an
intermediate representation, we can use language models to manipulate
this intermediate representation to our heart's content, using a
relatively informal and rather forgiving language. This is the idea
behind LAIR.

::: {#a-toy-example .section .level2 .page-columns .page-full}
## A toy example {.anchored anchor-id="a-toy-example"}

::: {.page-columns .page-full}
I'm a firm believer in silly toy examples. The sillier, the better. So,
we'll start with the following proposition: can we create a model that
will look at a photo from, say, the front page of our favourite
newspaper, and transpose it into the Warhammer 40k universe? In case
you're unfamiliar, Warhammer 40k is set -- as the name suggests -- in
the 40th millennium, but is a weird mixture of medieval and futuristic
technology. The whole atmosphere is taking the 'Dark Ages' part of the
Middle Ages,^1^ adding spaceships and laser guns, and turning the whole
thing into an absolutely depressing dystopia. It's a lot of fun.

::: {.no-row-height .column-margin .column-container}
^1^ Which may or may not ever have actually existed.
:::
:::

There are a few things we want here: I don't merely want the *visual*
style of the Warhammer 40k universe, I also want the *semantics* -- that
is, I want characters to be transposed into the Warhammer 40k universe.
I want the model to understand that the people in the photo are now
Space Marines, and that the buildings are now Gothic cathedrals. I can't
get neural transfer to that for me, because it does not understand, or
care, about semantics, and does not do semantic transformation. More
importantly, I cannot interact with the 'guts' of neural style transfer
beyond setting the hyperparameters and the source images.

What I can, however, do is to use the language I am mostly most
proficient in -- that is, human language -- to manipulate an
intermediate representation.

::: {.cell layout-align="center"}
::: {.cell-output-display}
::: {#fig-lair .quarto-figure .quarto-figure-center .anchored}
<figure class="figure">
<div>
<pre class="mermaid mermaid-js" data-label="fig-lair"><code>flowchart LR
    subgraph Description
        direction TB
        S[&quot;Source image&quot;] --&gt; D(&quot;Descriptor\ne.g. GPT-4 vision&quot;) --&gt; d[&quot;Description&quot;]
    end

    subgraph Transformation
        direction TB
        t(&quot;Transformer\ne.g. GPT-4&quot;) --&gt; td[&quot;Transformed\ndescription&quot;]
    end

    subgraph Rendering
        direction TB
        r(&quot;Renderer\ne.g. DALL-E&quot;) --&gt; I[&quot;Output\nimage&quot;]
    end

    Description --&gt; Transformation --&gt; Rendering</code></pre>
</div>
<figcaption>Figure 2: A rough outline of LAIR as applied to the toy
example.</figcaption>
</figure>
:::
:::
:::

Figure 2 outlines the basic idea for images. We use a descriptor model
to describe the image, and then use a text-to-text transformer to
transform the description. The transformed description is then rendered
into an image. The descriptor model is typically a vision model, and the
transformer is typically a language model. The renderer would then be a
generative model.

Consider the following source image I downloaded from Pexels:

::: {#fig-source .quarto-figure .quarto-figure-center .anchored}
<figure class="figure">
<p><img
src="https://chrisvoncsefalvay.com/posts/lair/pexels-canva-studio-3153198.jpg"
class="img-fluid figure-img" /></p>
<figcaption>Figure 3: Our source image.</figcaption>
</figure>
:::

Here's the intermediate representation (IR) description created by GPT-4
vision:

> This is an image of four individuals gathered around a laptop,
> suggesting a collaborative work environment. The setting has an ample
> amount of natural light, evident by the bright illumination on the
> subjects and the interior. They appear to be located in a modern
> indoor environment with a large curved window in the background,
> contributing to the light-flooded ambiance.
>
> Starting from the left, the first individual is a man with short, dark
> hair and a light beard, wearing a black t-shirt with a white graphic
> and text. He appears focused, resting his hands on the laptop while
> looking at the screen. To his right is a woman with shoulder-length
> curly hair who seems engaged and smiling, leaning slightly towards the
> laptop. She's wearing a light grey short-sleeved top.
>
> Behind these two and slightly out of focus, two more individuals are
> present. The person in the middle of the composition is a man with
> shoulder-length wavy hair, wearing a black shirt, gazing at the laptop
> screen. To the far right, partially cropped out of the frame, there is
> a woman with glasses, wearing a dark business casual blazer. She seems
> to be observing the laptop's screen with interest, standing slightly
> behind the seated individuals.
>
> The group exhibits a sense of teamwork and focus, with everyone's
> attention directed toward the laptop, situated on what appears to be a
> white table or desk. The body language and expressions suggest a
> positive, collaborative atmosphere, typical of a professional or
> academic team setting.

This is a pretty decent description. Let's grimdark it up by passing it
to our transformer model, which was initialised with the following
prompt:

> Your job is to analyze descriptions of images and change them to
> transpose the image into the dark, futuristic aesthetic of the
> Warhammer 40k universe. You are given a description of a scene, and
> your job is to change that scene according to the instructions, and
> give a detailed description of the changed scene that would allow a
> generative model to create an image.
>
> 1.  Replace clothing items with a mixture of futuristic and mediaeval
>     looking armour.
> 2.  Replace every-day objects held by characters with power swords,
>     chainswords,futuristic looking guns, staffs or magic items.
> 3.  Replace architecture with the monumental, dark architecture common
>     to the Warhammer 40k universe.
> 4.  Include things like monumental gothic spaceships in the image
>     description.

Here's the output:

> This is an image of four battle-hardened individuals gathered around a
> hololith, suggesting a strategic war council. The setting is bathed in
> the cold, harsh light of flickering lumens, casting long, grim shadows
> on the figures and the interior. They appear to be situated in a
> cathedral-like command room of an Imperial Battleship, with a
> monumental, gothic archway in the background, contributing to the
> oppressive ambiance.
>
> Starting from the left, the first individual is an Adeptus Astartes,
> his short, dark hair and light beard visible beneath the rim of his
> power armor helmet. He is clad in a black power armor with white
> Imperial insignia. He appears focused, his gauntleted hands resting on
> the hololith while studying the battlefield projection. To his right,
> a Sororitas, her shoulder-length, curly hair peeking out from beneath
> her wimple, leans towards the holographic display. She's clad in a
> light grey power armor, her face engaging and determined.
>
> Behind these two and slightly out of focus, two more figures loom. The
> figure in the middle of the composition is a man, his shoulder-length
> wavy hair barely visible beneath his Inquisitorial hat, wearing a
> black, ornate, medieval-looking armor, his eyes fixed on the hololith.
> To the far right, partially cropped out of the frame, there is a
> Tech-Priestess, her bionic glasses glinting in the lumen-light, her
> dark, mechanicus robes adding to her awe-inspiring presence. She seems
> to be observing the hololith with keen interest, standing slightly
> behind the armored individuals.
>
> The group gives off a sense of unity and focus, with everyone's
> attention directed toward the hololith, situated on what appears to be
> an aged, white marble table. The body language and expressions suggest
> a tense, collaborative atmosphere, typical of a war council in the
> grim darkness of the far future.

By the Emperor, that's some good stuff. Let's see what DALL-E makes of
it:

::: {#fig-output .quarto-figure .quarto-figure-center .anchored}
<figure class="figure">
<p><img
src="https://chrisvoncsefalvay.com/posts/lair/img-eTqdxb5CYGT2vY5FqNkRJdva.png"
class="img-fluid figure-img" /></p>
<figcaption>Figure 4: Our output image.</figcaption>
</figure>
:::

It's interesting to note what remains and what has changed. In
particular, the relationship of the figures to each other, both
spatially and semantically, as well as their posture, their number
(mostly!) and the general subject matter at a highly semanticised level
was preserved. Instead of a laptop, they're looking at a big holographic
sand table. Note that unlike in NST, we actually have control over what
is preserved and what is not -- that's what our transformer prompt
accomplishes. We've told it to change dress, for instance, but we
haven't told it to make any changes to the overall relationship between
the figures (note how a "collaborative atmosphere", for instance, was
retained word for word).
:::

::: {#whats-the-point .section .level2 .page-columns .page-full}
## What's the point? {.anchored anchor-id="whats-the-point"}

Okay, this was quite fun, but what's the point? Well, the point is that
we can use language as an intermediate representation to manipulate
things relatively easily. This is powerful because language is such an
accessible and forgiving intermediate representation. More importantly,
however, *anything* that can be represented in language can be
manipulated this way.

- In the space of language-to-language, this includes adapting text to
  the needs of special audiences,[Verma, Boland, and Miesenberger
  (2023)]{.citation cites="steinmetz2023developing"} conveying
  information to lay audiences [(Lee, Goldberg, and Kohane
  2023)]{.citation cites="lee2023ai"} and creating secondary explanatory
  materials e.g. statutory explanations [(Blair-Stanek, Holzenberger,
  and Van Durme 2023)]{.citation cites="blair2023can"}.
- For image-to-image transformations, retaining semanticity while
  simpifying visuals and removing clutter may often be useful, e.g. for
  creating procedural visual guidance for medical procedures [(Chen
  2023)]{.citation cites="chen2023endoscopic"}. Often, such images are
  created by hand, but this is a time-consuming process that could be
  automated.
- For code-to-code, language as an intermediate representation allows
  the interjection of desired features into code, e.g. for the purposes
  of code refactoring. Beyond simple code rewriting, this allows a kind
  of opinionated transformation. Often, a target language is not only
  idiomatically different but also has certain other characteristics,
  and this is *a fortiori* the case for DSLs [(Magalhães et al.
  2023)]{.citation cites="magalhaes2023c2taco"}.
- For code-to-text, this allows the creation of documentation from code,
  which is a perennial problem in software engineering. The textual
  intermediate representation allows fine control over the resulting
  documentation.

::: {.no-row-height .column-margin .column-container}
::: {#ref-verma2023bridging .csl-entry}
Verma, A Kumar, S Gavra Boland, and Klaus Miesenberger. 2023. 'Bridging
the Digital Divide for Persons with Intellectual Disabilities: Assessing
the Role of ChatGPT in Enabling Access, Evaluation, Integration,
Management, and Creation of Digital Content'. In *ICERI2023
Proceedings*, 3767--76. IATED.
:::

::: {#ref-lee2023ai .csl-entry}
Lee, Peter, Carey Goldberg, and Isaac Kohane. 2023. *The AI Revolution
in Medicine: GPT-4 and Beyond*. Pearson.
:::

::: {#ref-blair2023can .csl-entry}
Blair-Stanek, Andrew, Nils Holzenberger, and Benjamin Van Durme. 2023.
'Can GPT-3 Perform Statutory Reasoning?' *arXiv Preprint
arXiv:2302.06100*.
:::

::: {#ref-chen2023endoscopic .csl-entry}
Chen, Hao-Wen. 2023. 'Endoscopic Endonasal Skull Base Surgery for
Pituitary Lesions: An AI-Assisted Creative Workflow to Develop an
Animated Educational Resource for Patients and Physicians'. PhD thesis,
Johns Hopkins University.
:::

::: {#ref-magalhaes2023c2taco .csl-entry}
Magalhães, José Wesley de Souza, Jackson Woodruff, Elizabeth Polgreen,
and Michael FP O'Boyle. 2023. 'C2TACO: Lifting Tensor Code to TACO'. In
*Proceedings of the 22nd ACM SIGPLAN International Conference on
Generative Programming: Concepts and Experiences*, 42--56.
:::
:::

The possibilities for using language as an intermediate representation
are endless. LAIR is a powerful paradigm for transformational tasks that
allows us to use language as an intermediate representation to
manipulate things in a way that is both accessible and powerful, and
that allows us to pick and choose *what* part of semanticity we want to
manipulate *versus* what we want to preserve. It's hard to reason about
LAIR's relative performance given that it is not a technique but a
paradigm, and that its focus is not simple style transfer but finely
controlled stylistic and contextual transformation, but even in the
current absence of benchmarks, it is clear that models benefit from
using language as an easily workable and malleable intermediate
representation.
:::

::: {#code .section .level2}
## Code {.anchored anchor-id="code"}

The code for the toy example is available
[here](https://gist.github.com/chrisvoncsefalvay/f3c6e7547350cfae50be0078ad99435f).
:::

::: {#quarto-appendix .default}
::: {.section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
BibTeX citation:
:::

``` {.sourceCode .code-with-copy .quarto-appendix-bibtex}
@online{csefalvay2024,
  author = {Chris von Csefalvay},
  title = {LAIR - {Language} {As} {Intermediate} {Representation}},
  date = {2024-01-06},
  url = {https://chrisvoncsefalvay.com/posts/lair},
  langid = {en-GB}
}
```

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-csefalvay2024 .csl-entry .quarto-appendix-citeas}
Chris von Csefalvay. 2024. "LAIR - Language As Intermediate
Representation." January 6, 2024.
<https://chrisvoncsefalvay.com/posts/lair>.
:::

</div>
:::
:::


The awesome thing about language is that, well, we all mostly speak it, to some extent or another. This gives us an immensely powerful tool to manipulate transformational tasks. For the purposes of this post, I consider a transformational task to be essentially anything that takes an input and is largel intended to return some version of the same thing. This is not a very precise definition, but it will have to do for now.

Messaggi di Rogue Scholar

LAIR - Language As Intermediate Representation