Tag: SSO

If you run Authentik (a good homelab-friendly SSO provider for logins), you may have mounted the Letsencrypt /etc/letsencrypt folder into the Authentik worker as /certs, following the Docker Compose installation instructions. What’s supposed to happen is that Authentik runs a cron job to scan the contents of /certs, recognizes and picks up any certificates along with private keys and imports them for use in the built-in web server. Supposedly, Authentik recognizes the certbot-style folder and can handle it.

A bug in Authentik 2026.5.x causes the certificate_discovery task to spam entries into the authentik_tasks_task table — sometimes ballooning to millions of rows and gigabytes of bloat:

SELECT count(*) FROM authentik_tasks_task
WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery';
  count  
---------
 4899978
(1 row)

SELECT
  schemaname,
  relname AS table_name,
  pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
  pg_size_pretty(pg_relation_size(relid)) AS table_size,
  pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) AS indexes_toast_size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;
 schemaname |                           table_name                            | total_size | table_size | indexes_toast_size 
------------+-----------------------------------------------------------------+------------+------------+--------------------
 public     | authentik_tasks_task                                            | 4705 MB    | 2100 MB    | 2605 MB
 public     | authentik_tasks_tasklog                                         | 355 MB     | 242 MB     | 113 MB
...

Predictably, this ballooned the size of the PostgreSQL database on disk, and created some very unexpected problems for a Docker container that uses overlayfs.

There are related issues identified on GitHub (#19299, #22742) but no one has posted a precise cleanup solution for this. Cleanup worker tasks hang and stall other tasks. Postgres is itself in a dire condition.

Marvel at the size of the tables:

$ docker compose exec postgresql psql -U authentik -d authentik -c "select actor_name, count(*) from authentik_tasks_task group by actor_name ORDER BY count DESC;"

                                      actor_name                                      |  count  
--------------------------------------------------------------------------------------+---------
 authentik.crypto.tasks.certificate_discovery                                         | 4899978
 authentik.core.tasks.clean_expired_models                                            |    8696
 authentik.core.tasks.clean_temporary_users                                           |    7972
 authentik.outposts.tasks.outpost_service_connection_monitor                          |    2945
 authentik.blueprints.v1.tasks.blueprints_discovery                                   |     853
 authentik.blueprints.v1.tasks.clear_failed_blueprints                                |     843
 authentik.admin.tasks.update_latest_version                                          |     630
 authentik.outposts.tasks.outpost_controller                                          |     415
 authentik.enterprise.tasks.enterprise_update_usage                                   |     366
 authentik.sources.oauth.tasks.update_well_known_jwks                                 |     248
...
(25 rows)

The size on disk, too, was humongous:

$ docker compose exec postgresql psql -U authentik -d authentik -c "SELECT actor_name, pg_size_pretty(SUM(pg_column_size(t))) AS total_size, COUNT(*) AS row_count FROM authentik_tasks_task t GROUP BY actor_name ORDER BY SUM(pg_column_size(t)) DESC;"
                                      actor_name                                      | total_size | row_count 
--------------------------------------------------------------------------------------+------------+-----------
 authentik.crypto.tasks.certificate_discovery                                         | 1009 MB    |   4899978

Solution

Before doing anything destructive, take a backup of your VM, or at the very least, of the current contents of the Postgres database.

You may need to stop the Authentik containers, keeping only Postgres running, then clean up the data manually. Rather than pasting commands that use sudo docker compose exec..., start up a Postgres shell with docker compose exec -i postgresql psql -U authentik -d authentik first.

First, figure out how many tasks there are to clean up in the authentik_tasks_task table:

SELECT count(*) FROM authentik_tasks_task WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery';

There are foreign key constraints, so you can’t just drop these rows; you’ve got to delete the other references in other tables first.

Run a sanity check to see how many rows would be deleted if you did a purge of anything older than 14 days:

SELECT
  (SELECT count(*) FROM authentik_tasks_task
     WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
     AND mtime < now() - interval '14 days') AS task_rows,
  (SELECT count(*) FROM authentik_tasks_tasklog
     WHERE task_id IN (SELECT message_id FROM authentik_tasks_task
       WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
       AND mtime < now() - interval '14 days')) AS tasklog_rows,
  (SELECT count(*) FROM authentik_tasks_taskdependency
     WHERE task_id IN (SELECT message_id FROM authentik_tasks_task
       WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
       AND mtime < now() - interval '14 days')
     OR dependency_id IN (SELECT message_id FROM authentik_tasks_task
       WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
       AND mtime < now() - interval '14 days')) AS taskdependency_rows;

If you get a sane output like this, you are ready to go. For safety, use a transaction (start with BEGIN; and, only when all commands have returned, COMMIT;).

BEGIN;

DELETE FROM authentik_tasks_taskdependency
WHERE task_id IN (
    SELECT message_id FROM authentik_tasks_task
    WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
    AND mtime < now() - interval '14 days'
)
OR dependency_id IN (
    SELECT message_id FROM authentik_tasks_task
    WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
    AND mtime < now() - interval '14 days'
);

DELETE FROM authentik_tasks_tasklog
WHERE task_id IN (
    SELECT message_id FROM authentik_tasks_task
    WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
    AND mtime < now() - interval '14 days'
);

DELETE FROM authentik_tasks_task
WHERE actor_name = 'authentik.crypto.tasks.certificate_discovery'
AND mtime < now() - interval '14 days';

COMMIT;

Depending on how many rows remain that are newer than 14 days, you may need to repeat this exercise and just drop rows older than 1 day (keeping some more recent tasks in case for some reason you need to audit more recent events).

You might want to check the Postgres logs too, but when it’s all done, likely a VACUUM FULL or VACUUM ANALYZE is needed to recover the disk space. Gigabytes and gigabytes of disk space.