Skip to content

Commit e7e7648

Browse files
committed
add post about ecto preloads
1 parent ce55fad commit e7e7648

File tree

7 files changed

+597
-456
lines changed

7 files changed

+597
-456
lines changed

lib/mix/tasks/new_post.ex

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,6 @@ defmodule Mix.Tasks.NewPost do
44
def run(_args) do
55
Application.ensure_all_started(:personal)
66

7-
{:ok, now} = DateTime.now("America/New_York")
8-
today = DateTime.to_date(now)
9-
date_path_fragment = Calendar.strftime(today, "%Y/%m-%d")
10-
117
title = get_title()
128
url = title |> String.downcase() |> String.replace("'", "") |> String.replace(~r/\W+/, "-")
139
tags = get_tags()
@@ -27,7 +23,7 @@ defmodule Mix.Tasks.NewPost do
2723
2824
"""
2925

30-
path = "posts/#{date_path_fragment}-#{url}.md"
26+
path = "drafts/#{url}.md"
3127

3228
File.write(path, post_body)
3329

output/index.html

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ <h1>andy@andyleclair.dev$><span class="blink">_</span></h1>
2121
<h2 class="text-xl">Blog!</h2>
2222
<ul>
2323
<li>
24+
2024-11-15 - <a href="posts/2024/11-15-how-to-do-a-preload-good.html">How To Do A Preload Good</a>
25+
</li><li>
2426
2024-10-28 - <a href="posts/2024/10-28-live-fridge.html">Live Fridge</a>
2527
</li><li>
2628
2024-09-11 - <a href="posts/2024/09-11-opengl-part-3.html">OpenGL Part 3</a>

output/posts/2024/09-09-gltest.html

Lines changed: 174 additions & 174 deletions
Large diffs are not rendered by default.

output/posts/2024/09-10-opengl-part-2.html

Lines changed: 96 additions & 96 deletions
Large diffs are not rendered by default.

output/posts/2024/09-11-opengl-part-3.html

Lines changed: 181 additions & 181 deletions
Large diffs are not rendered by default.
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
<!doctype html>
2+
<html>
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<link rel="stylesheet" href="/assets/app.css">
7+
<script type="text/javascript" src="/assets/app.js"></script>
8+
9+
<title>How To Do A Preload Good</title>
10+
<meta name="description" content="Avoiding N+1 queries with Ecto&#39;s preload">
11+
12+
</head>
13+
14+
<body class="bg-nor-easter text-smurf-blood">
15+
<div class="flex h-60 min-h-screen flex-col items-center">
16+
<header class="bg-bludacris p-10 my-4 lg:mt-10 lg:mb-14">
17+
<h1>andy@andyleclair.dev$><span class="blink">_</span></h1>
18+
</header>
19+
<main class="relative grow min-h-96 flex-1 p-4">
20+
21+
<article class="prose lg:prose-xl prose-pre:bg-codebg">
22+
<h1>How To Do A Preload Good</h1>
23+
<h3>Avoiding N+1 queries with Ecto&#39;s preload</h3>
24+
<h3><a href="https://www.youtube.com/watch?v=nhWlJLP6QyM">Related Listening</a></h3>
25+
<p class="text-smurf-blood">Posted on 2024-11-15</p>
26+
<p>
27+
I recently had <a href="https://twitter.com/andyleclair/status/1857112936101134588">this exchange</a> on Twitter about using Ecto’s <code class="inline">Repo.preload</code> and I wanted to describe
28+
the way that we handle this at Appcues. Obviously everyone has their opinions, but this has served us very well for years, and we’ve never posted anything about it!</p>
29+
<p>
30+
Hopefully this can help somebody out there.</p>
31+
<h2>
32+
The Problem</h2>
33+
<p>
34+
So, if you do something like <code class="inline">Repo.get(queryable) |&gt; Repo.preload(:association)</code>, you’re going to get a query for the original record, and then a query for each of the associated records. This is the classic N+1 query problem, and it’s not good.</p>
35+
<p>
36+
How do you solve it? More functions!</p>
37+
<h2>
38+
The Solution</h2>
39+
<pre><code class="makeup elixir"><span class="kd">def</span><span class="w"> </span><span class="nf">get_thing</span><span class="p" data-group-id="7940493633-1">(</span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">opts</span><span class="w"> </span><span class="o">\\</span><span class="w"> </span><span class="p" data-group-id="7940493633-2">[</span><span class="p" data-group-id="7940493633-2">]</span><span class="p" data-group-id="7940493633-1">)</span><span class="w"> </span><span class="k" data-group-id="7940493633-3">do</span><span class="w">
40+
</span><span class="n">from</span><span class="p" data-group-id="7940493633-4">(</span><span class="n">t</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="nc">Thing</span><span class="p">,</span><span class="w"> </span><span class="ss">where</span><span class="p">:</span><span class="w"> </span><span class="n">t</span><span class="o">.</span><span class="n">id</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">^</span><span class="n">id</span><span class="p" data-group-id="7940493633-4">)</span><span class="w">
41+
</span><span class="o">|&gt;</span><span class="w"> </span><span class="n">preload</span><span class="p" data-group-id="7940493633-5">(</span><span class="n">opts</span><span class="p" data-group-id="7940493633-6">[</span><span class="ss">:preload</span><span class="p" data-group-id="7940493633-6">]</span><span class="p" data-group-id="7940493633-5">)</span><span class="w">
42+
</span><span class="o">|&gt;</span><span class="w"> </span><span class="nc">Repo</span><span class="o">.</span><span class="n">one</span><span class="p" data-group-id="7940493633-7">(</span><span class="n">query</span><span class="p" data-group-id="7940493633-7">)</span><span class="w">
43+
</span><span class="k" data-group-id="7940493633-3">end</span><span class="w">
44+
45+
</span><span class="kd">def</span><span class="w"> </span><span class="nf">preload</span><span class="p" data-group-id="7940493633-8">(</span><span class="n">query</span><span class="p" data-group-id="7940493633-8">)</span><span class="p">,</span><span class="w"> </span><span class="ss">do</span><span class="p">:</span><span class="w"> </span><span class="n">preload</span><span class="p" data-group-id="7940493633-9">(</span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="no">true</span><span class="p" data-group-id="7940493633-9">)</span><span class="w">
46+
</span><span class="kd">def</span><span class="w"> </span><span class="nf">preload</span><span class="p" data-group-id="7940493633-10">(</span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="no">nil</span><span class="p" data-group-id="7940493633-10">)</span><span class="p">,</span><span class="w"> </span><span class="ss">do</span><span class="p">:</span><span class="w"> </span><span class="n">query</span><span class="w">
47+
48+
</span><span class="kd">def</span><span class="w"> </span><span class="nf">preload</span><span class="p" data-group-id="7940493633-11">(</span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="no">true</span><span class="p" data-group-id="7940493633-11">)</span><span class="w"> </span><span class="k" data-group-id="7940493633-12">do</span><span class="w">
49+
</span><span class="n">from</span><span class="w"> </span><span class="n">q</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="ss">preload</span><span class="p">:</span><span class="w"> </span><span class="p" data-group-id="7940493633-13">[</span><span class="w">
50+
</span><span class="ss">:association</span><span class="p">,</span><span class="w">
51+
</span><span class="ss">other_assoc</span><span class="p">:</span><span class="w"> </span><span class="p" data-group-id="7940493633-14">[</span><span class="ss">:sub_assoc</span><span class="p" data-group-id="7940493633-14">]</span><span class="w">
52+
</span><span class="p" data-group-id="7940493633-13">]</span><span class="w">
53+
</span><span class="k" data-group-id="7940493633-12">end</span><span class="w">
54+
55+
</span><span class="kd">def</span><span class="w"> </span><span class="nf">preload</span><span class="p" data-group-id="7940493633-15">(</span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="n">preloads</span><span class="p" data-group-id="7940493633-15">)</span><span class="w"> </span><span class="k" data-group-id="7940493633-16">do</span><span class="w">
56+
</span><span class="n">from</span><span class="w"> </span><span class="n">q</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="ss">preload</span><span class="p">:</span><span class="w"> </span><span class="o">^</span><span class="n">preloads</span><span class="w">
57+
</span><span class="k" data-group-id="7940493633-16">end</span></code></pre>
58+
<p>
59+
If you need to get fancier with it, you can also use a <code class="inline">left_join</code> and get more specific with your preload conditions.
60+
This will allow you to do things like adjust the query based on the associations, like if you’d need to sort based on
61+
say, the index of the association.</p>
62+
<pre><code class="makeup elixir"><span class="kd">def</span><span class="w"> </span><span class="nf">preload</span><span class="p" data-group-id="8303111550-1">(</span><span class="n">query</span><span class="p">,</span><span class="w"> </span><span class="no">true</span><span class="p" data-group-id="8303111550-1">)</span><span class="w"> </span><span class="k" data-group-id="8303111550-2">do</span><span class="w">
63+
</span><span class="n">from</span><span class="w"> </span><span class="n">q</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">query</span><span class="p">,</span><span class="w">
64+
</span><span class="ss">left_join</span><span class="p">:</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">assoc</span><span class="p" data-group-id="8303111550-3">(</span><span class="n">q</span><span class="p">,</span><span class="w"> </span><span class="ss">:thing</span><span class="p" data-group-id="8303111550-3">)</span><span class="p">,</span><span class="w">
65+
</span><span class="ss">left_join</span><span class="p">:</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">assoc</span><span class="p" data-group-id="8303111550-4">(</span><span class="n">t</span><span class="p">,</span><span class="w"> </span><span class="ss">:sub_thing</span><span class="p" data-group-id="8303111550-4">)</span><span class="p">,</span><span class="w">
66+
</span><span class="ss">preload</span><span class="p">:</span><span class="w"> </span><span class="p" data-group-id="8303111550-5">[</span><span class="w">
67+
</span><span class="ss">thing</span><span class="p">:</span><span class="w"> </span><span class="p" data-group-id="8303111550-6">{</span><span class="n">t</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8303111550-7">[</span><span class="ss">sub_thing</span><span class="p">:</span><span class="w"> </span><span class="n">s</span><span class="p" data-group-id="8303111550-7">]</span><span class="p" data-group-id="8303111550-6">}</span><span class="w">
68+
</span><span class="p" data-group-id="8303111550-5">]</span><span class="p">,</span><span class="w">
69+
</span><span class="ss">order_by</span><span class="p">:</span><span class="w"> </span><span class="p" data-group-id="8303111550-8">[</span><span class="ss">asc</span><span class="p">:</span><span class="w"> </span><span class="n">t</span><span class="o">.</span><span class="n">index</span><span class="p" data-group-id="8303111550-8">]</span><span class="w">
70+
</span><span class="p">]</span><span class="w">
71+
</span><span class="k" data-group-id="8303111550-2">end</span></code></pre>
72+
<p>
73+
That’s the basic gist of it. I hope this helps someone out there!</p>
74+
75+
</article>
76+
77+
</main>
78+
<footer class="bg-bludacris p-4 text-center">© Andy LeClair 2024</footer>
79+
</div>
80+
</body>
81+
</html>
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
%{
2+
title: "How To Do A Preload Good",
3+
description: "Avoiding N+1 queries with Ecto's preload",
4+
author: "Andy LeClair",
5+
tags: ["elixir", "ecto"],
6+
related_listening: "https://www.youtube.com/watch?v=nhWlJLP6QyM",
7+
}
8+
---
9+
10+
I recently had [this exchange](https://twitter.com/andyleclair/status/1857112936101134588) on Twitter about using Ecto's `Repo.preload` and I wanted to describe
11+
the way that we handle this at Appcues. Obviously everyone has their opinions, but this has served us very well for years, and we've never posted anything about it!
12+
13+
Hopefully this can help somebody out there.
14+
15+
## The Problem
16+
17+
So, if you do something like `Repo.get(queryable) |> Repo.preload(:association)`, you're going to get a query for the original record, and then a query for each of the associated records. This is the classic N+1 query problem, and it's not good.
18+
19+
How do you solve it? More functions!
20+
21+
22+
## The Solution
23+
24+
```elixir
25+
def get_thing(id, opts \\ []) do
26+
from(t in Thing, where: t.id == ^id)
27+
|> preload(opts[:preload])
28+
|> Repo.one(query)
29+
end
30+
31+
def preload(query), do: preload(query, true)
32+
def preload(query, nil), do: query
33+
34+
def preload(query, true) do
35+
from q in query, preload: [
36+
:association,
37+
other_assoc: [:sub_assoc]
38+
]
39+
end
40+
41+
def preload(query, preloads) do
42+
from q in query, preload: ^preloads
43+
end
44+
```
45+
46+
If you need to get fancier with it, you can also use a `left_join` and get more specific with your preload conditions.
47+
This will allow you to do things like adjust the query based on the associations, like if you'd need to sort based on
48+
say, the index of the association.
49+
50+
```elixir
51+
def preload(query, true) do
52+
from q in query,
53+
left_join: t in assoc(q, :thing),
54+
left_join: s in assoc(t, :sub_thing),
55+
preload: [
56+
thing: {t, [sub_thing: s]}
57+
],
58+
order_by: [asc: t.index]
59+
]
60+
end
61+
```
62+
That's the basic gist of it. I hope this helps someone out there!

0 commit comments

Comments
 (0)