ArtRECONon-Relational Recommendation via Contrastive Optimization

Each artwork and caption is embedded with CLIP. Pretrained CLIP was trained on web image–caption pairs rather than curated museum art, so there’s a domain shift. Full fine-tuning of ViT-B/16 risks overfitting on roughly 58k examples; LoRA trains under 1% of parameters, regularizes implicitly, and fits on a consumer GPU.

For a frozen projection $W \in \mathbb{R}^{d \times k}$, LoRA learns a low-rank update

$A$ is Gaussian-initialized and $B$ is zero, so training starts exactly at the pretrained model and gradually learns a domain-specific perturbation.

Config: $r = 8, \, \alpha = 16, \, p_{\mathrm{dropout}} = 0.05$. Adapters target $\texttt{q\_proj}$, $\texttt{k\_proj}$, $\texttt{v\_proj}$, and $\texttt{out\_proj}$in both the vision and text towers — ~600k trainable parameters out of ~150M (under 0.4%).

Symmetric CLIP contrastive loss. For a batch of normalized image embeddings $x_i$ and text embeddings $t_j$, let $s_{ij} = \tau^{-1} x_i^\top t_j$. The image→text and text→image losses are

The final objective is $\mathcal{L} = \tfrac{1}{2}(\mathcal{L}_{I \to T} + \mathcal{L}_{T \to I})$. Temperature is learned via $\texttt{logit\_scale} = \log(1/\tau)$, clamped at $\log(100)$ to prevent $\tau$ collapse. Base CLIP weights stay frozen otherwise.

• AdamW, lr $10^{-4}$, weight decay 0.01.
• Cosine schedule, 500 warmup steps.
• Batch size 64, 5 epochs.
• Mixed-precision fp16 with gradient scaling.
• Held-out 5% split, recall@5 reported each epoch.

Recall@5 before and after LoRA tuning:

A side-by-side top-5 retrieval — baseline vs. LoRA on the same query — is the clearest qualitative view of the gain.

Limitations: captions are synthesized from metadata, the dataset is modest for contrastive learning, and training uses in-batch negatives rather than hard-negative mining.

The recommendation layer uses the CLIP–LoRA embedding space to update preference state from clicks and split future feed slots between learned taste components and random exploration.

The user state is a set of weighted components $\mathcal{S} = \{(s_i, v_i, k_i)\}$ with $\sum_i s_i = 1$. Each component is either a taste component with a unit-norm vector $v_i$, or the single random component, which has no vector and represents exploration mass.

Let $c$ be the unit-norm embedding of the clicked artwork. Find the closest existing taste vector by cosine similarity,$\sigma^* = \max_i \langle v_i, c \rangle$. If $\sigma^* \ge \tau$, merge the click into that component with mixing rate $\delta$:

Otherwise spawn a new taste component $(\delta, c, \mathrm{taste})$. All other weights are scaled by $1 - \delta$, which keeps the total mass at one.

After the update, drop any component with weight below $\varepsilon$ and renormalize the survivors so the weights still sum to one.

Let $R$ be the total weight on the random component. For a feed of length $N$, give $N_p = \mathrm{round}(N(1 - R))$ slots to personalized items and the remaining $N_r = N - N_p$ to random exploration. Personalized slots are divided across taste components in proportion to their relative weight:

The personalized and random items are then interleaved into the final feed.

Clicks only merge with taste components, so the random weight decays geometrically: $R_n = (1 - \delta)^n$. It drops below $\varepsilon$ once $n > \log\varepsilon / \log(1 - \delta)$. For $\delta = 0.2$ and $\varepsilon = 1/60$, that’s about 19 clicks — after which the feed is fully personalized.

The update rule removes the cold-start setup stage: the system starts with random exploration and learns taste directly from clicks, so every interaction shapes the state immediately.

Spawning new taste components when a click is far from existing ones lets the model carry multiple interests instead of collapsing into one profile. The random component keeps early recommendations exploratory, then decays as taste signal accumulates.