The Problem in One Sentence

TLS trust store allocations (~4MB) scattered through the heap prevent glibc from returning ~250MB of freed memory to the OS.

Problem Summary

When gnome-software handles search requests from gnome-shell, it allocates memory for:

  • Release history - Application version history and metadata
  • Icons - Application icon images
  • TLS trust stores - gnutls certificate trust lists (via libsoup HTTPS connections)

After the request completes, gnome-software frees the release history and icons, but the TLS trust store allocations remain. These small, scattered allocations prevent glibc from returning memory to the OS, causing the process to grow hundreds of MB larger than necessary.

Root Cause

  1. Each HTTPS request to Flathub triggers a new TLS connection
  2. Each TLS connection causes gnutls to load the system CA trust store (~150-200 certificates)
  3. This creates thousands of small ASN.1 allocations scattered through the heap
  4. When gnome-software frees app data, the trust store allocations remain
  5. These prevent glibc from returning freed memory regions to the OS (heap fragmentation)

Color Coding

Throughout the diagrams:

  • 🟢 Green = Release history allocations
  • 🔵 Blue = Icon allocations
  • 🔴 Red = TLS trust store allocations (gnutls)
  • ⬜ Gray = Free memory

Visual Explanation

Phase 1: Initial State (Idle)

Phase 1: Idle State

The process starts with minimal memory usage - only ~50 MB RSS.

Phase 2: Handling Search Request

Phase 2: Search Request

During a search request, gnome-software allocates memory for:

  • Release history (green) - Application metadata and version information
  • Icons (blue) - Application icon images
  • TLS trust stores (red) - gnutls certificate trust lists for HTTPS connections to Flathub

Notice how TLS trust store allocations are interspersed with application data throughout the heap regions. The RSS grows to 280 MB.

Phase 3: After Cache Clear (30s timeout)

Phase 3: After Cache Clear

After the cache clear timeout, gnome-software frees the release history and icons, but the TLS trust store allocations remain.

The Problem: Each heap region still contains TLS allocations, preventing glibc from returning the entire region to the OS via madvise(MADV_DONTNEED) or sbrk().

The trust store allocations are only ~4-6 MB total, but they pin ~250 MB of heap regions in memory! The RSS only drops to 260 MB instead of returning to ~50 MB.

Call Stack

The trust list allocations occur when downloading icons:

gs_icon_download (gs-remote-icon.c:265)
  → soup_session_send (soup-session.c:3264)
    → soup_connection_connect (soup-connection.c:865)
      → new_tls_connection (soup-connection.c:626)
        → g_tls_connection_gnutls_initable_init (gtlsconnection-gnutls.c:207)
          → g_tls_connection_get_database (gtlsconnection.c:504)
            → g_tls_database_gnutls_populate_trust_list (gtlsdatabase-gnutls.c:590)
              → gnutls_x509_trust_list_add_system_trust (certs.c:384)
                → p11_index_replace_all (index.c:727)
                  → asn1_der_decoding (decoding.c:1627)
                    → _asn1_add_single_node (structure.c:55)