{"id":15565,"date":"2021-01-21T16:19:46","date_gmt":"2021-01-21T15:19:46","guid":{"rendered":"https:\/\/daniel.haxx.se\/blog\/?p=15565"},"modified":"2021-02-24T22:36:31","modified_gmt":"2021-02-24T21:36:31","slug":"more-on-less-curl-memory","status":"publish","type":"post","link":"https:\/\/daniel.haxx.se\/blog\/2021\/01\/21\/more-on-less-curl-memory\/","title":{"rendered":"More on less curl memory"},"content":{"rendered":"\n

tldr: curl uses 30K of dynamic memory for downloading a large HTTP file, plus the size of the download buffer.<\/em><\/p>\n\n\n\n

Back in September 2020 I wrote about my work to trim curl allocations done for FTP transfers<\/a>. Now I’m back again on the memory use in curl topic, from a different angle.<\/p>\n\n\n\n

This time, I learned about the awesome tool pahole<\/a>, which can (among other things) show structs and their sizes from a built library – and when embracing this fun toy, I ran some scripts on a range of historic curl releases to get a sense of how we’re doing over time – memory size and memory allocations wise.<\/p>\n\n\n\n

The task I set out to myself was: figure out how the sizes of key structs in curl have changed over time<\/strong>, and correlate that with the number and size of allocations done at run-time. To make sure that trimming down the size of a specific struct doesn’t just get allocated by another one instead, thus nullifying the gain. I want to make sure we’re not slowly degrading – and if we do, we should at least know about it!<\/p>\n\n\n\n

Also: we keep developing curl at a fairly good pace and we’re adding features in almost every release. Some growth is to be expected and should be tolerated I think. We also keep the build process very configurable so users with particular needs and requirements can switch off features and thus also gain memory.<\/p>\n\n\n\n

Memory sizes in modern computing<\/h2>\n\n\n\n

Of course systems are growing every year and machines ship with more and more ram, which also goes for the smallest machines. But there are still a vast amount of systems out there with limited memory capabilities that want good Internet transfers as well. Also, by keeping sizes down, it allows applications and systems to scale better: a 10% decrease in size can<\/em> imply a 10% increase in number of possible parallel transfers. curl, and especially libcurl, is still today in 2021 frequently used on machines with limited amounts of available memory. Sometimes in the few megabytes of ram range.<\/p>\n\n\n\n

Fixed configuration<\/h2>\n\n\n\n

In my tests I did for this I used the exact same configuration and build config for all versions tested. The sizes and behavior will vary greatly depending on config, but I tried to use a fairly complete and typical build to see how code and memory use is for “most” users. I ran everything on my x86_64 Debian Linux dev machine. My focus is on curl versions from the last 3-4 years. I figured going back to ancient times won’t help here.<\/p>\n\n\n\n

Key structs<\/h2>\n\n\n\n

struct Curl_easy<\/code> – this is the “easy handle”, what is allocated by curl_easy_init()<\/a> and is the anchor for every transfer done with libcurl, no matter which API you’re using. An application creates one of these for each concurrent transfer it wants to do or keep around. Some applications allocate hundreds or even thousands of these.<\/p>\n\n\n\n

struct Curl_multi<\/code> – this is the “multi handle”, allocated with curl_multi_init()<\/a>. This handle is created by applications as a holder of many concurrent transfers so applications typically do not have a very large amount of these.<\/p>\n\n\n\n

struct connectdata<\/code> – this is an internal struct that isn’t visible externally to applications. It is the holder of connection related data for a connection to a specific server. The connection pool curl uses to handle persistent connections will hold a number of these structs in memory after the transfer has completed, to allow subsequent reuse. The size of the connection pool is customizable. A busy application doing lots of transfers might end up with a sizeable number of connections in the pool, so the size of this struct adds up.<\/p>\n\n\n\n

Dynamic allocations<\/h2>\n\n\n\n

In early curl history, the download and upload buffers for transfers were part of the Curl_easy struct, which made it fairly large.<\/p>\n\n\n\n

In curl 7.53.0 (February 2017) the download buffer was turned dynamically sized and is since then allocated separately. Before that transition, curl 7.52.0 had a Curl_easy struct that was 36584 bytes, which included both the download and the upload buffers. In 7.58.0 the size was down to 21264 bytes since the download buffer was then allocated separately and was then also allowed to be done much larger than the previously set 16KB fixed size.<\/p>\n\n\n\n

The 16KB upload buffer was moved out of the Curl_easy handle in the 7.62.0 release (October 2018) to be done on demand – which of course especially benefits everyone who doesn’t do uploads at all… The size of this struct was then down to 6208 bytes.<\/p>\n\n\n\n

In curl 7.71.0 we also made the download buffer allocated on demand, and immediately freed after the transfer completes. This makes applications that keep handles around for reuse use significantly <\/em>less memory. Applications are generally encouraged to keep the handles around to better facilitate connection reuse.<\/p>\n\n\n\n

Struct size development<\/h2>\n\n\n\n

Curl_easy<\/h3>\n\n\n\n

The size in bytes of struct Curl_easy<\/strong> the last few years:<\/p>\n\n\n\n

 7.52.0 36584\n 7.58.0 21264\n 7.61.0 21344\n 7.62.0 6208\n 7.63.0 6216\n 7.64.0 6312\n 7.65.0 5976\n 7.66.0 6024\n 7.67.0 6040\n 7.68.0 6040\n 7.69.0 6040\n 7.70.0 6080\n 7.71.0 6448\n 7.72.0 6472\n 7.73.0 6464\n 7.74.0 6512<\/pre>\n\n\n\n
Current git: 5272<\/strong> bytes (-19% from last release). With this, the struct is smaller than it has ever been before<\/em>.<\/p>\n\n\n\n
How we made this extra reduction? Primarily I noticed how we had a DoH related struct in the handle by default, which was turned into on-demand allocation<\/a>. DoH is still rare and that data only needs to be allocated during the name resolving phase.<\/p>\n\n\n\n
Curl_multi<\/h3>\n\n\n\n
The size in bytes of struct Curl_multi<\/strong> the last few years has remained very stable and it keeps being very small. Notable is that when we removed pipelining support in 7.65.0 it took away 96 bytes from this struct.<\/p>\n\n\n\n
 7.50.0 384\n 7.52.0 384\n 7.58.0 480\n 7.59.0 488\n 7.60.0 488\n 7.61.0 512\n 7.62.0 512\n 7.63.0 512\n 7.64.0 512\n 7.65.0 416<\/span>\n 7.66.0 416\n 7.67.0 424\n 7.68.0 432\n 7.69.0 416\n 7.70.0 416\n 7.71.0 416\n 7.72.0 416\n 7.73.0 416\n 7.74.0 416<\/pre>\n\n\n\n
Current git: 416<\/strong> bytes.<\/p>\n\n\n\n
With this, we’re smaller than we were in the beginning of 2018.<\/p>\n\n\n\n
connectdata<\/h3>\n\n\n\n
The size in bytes of struct connectdata<\/strong>. It’s been both up and down.<\/p>\n\n\n\n
 7.50.0 1904 \n 7.52.0 2104 \n 7.58.0 2112 \n 7.59.0 2112 \n 7.60.0 2112 \n 7.61.0 2128 \n 7.62.0 2152 \n 7.63.0 2160 \n 7.64.0 2160 \n 7.65.0 1944 \n 7.66.0 1960 \n 7.67.0 1976 \n 7.68.0 1976 \n 7.69.0 2600 <\/span>\n 7.70.0 2608 \n 7.71.0 2608 \n 7.72.0 2624 \n 7.73.0 2640 \n 7.74.0 2656<\/pre>\n\n\n\n
Current git: 1472<\/strong> bytes (-44% from last release)<\/p>\n\n\n\n
The size bump in 7.69.0 was the insertion of a new struct for state data when doing SOCKS connections non-blocking, and the corresponding decrease again for the pending release is the removal of the buffer<\/a> from that struct. With this, we’re down to a size we had a very long time ago.<\/p>\n\n\n\n
Run-time memory use<\/h2>\n\n\n\n
To make sure that we don’t just move memory to other on-demand buffers that we need to allocate anyway, I ran a script with a lot of curl versions and counted the number of allocations needed and the peak amount of memory allocated. For a plain 512MB download over HTTP from localhost. The counted allocations were only the ones done by curl code<\/strong> (malloc, calloc, realloc, strdup etc).<\/p>\n\n\n\n
Number of allocations<\/h3>\n\n\n\n
There are many reasons to allocate memory and while we want to keep the number down, lots of factors of course needs to be taken into account.<\/p>\n\n\n\n
In the list below you’ll see that clearly we had some mistake in 7.52.0 and perhaps some more versions, as it did over 32,000 allocations. The situation was fixed in or before 7.58.0 and I haven’t bothered to go back to check exactly what it was.<\/p>\n\n\n\n
 7.52.0 32883 <\/span>\n 7.58.0 82 \n 7.59.0 82 \n 7.60.0 82 \n 7.61.0 82 \n 7.62.0 86 \n 7.63.0 87 \n 7.64.0 87 \n 7.65.0 82\n 7.66.0 101 \n 7.67.0 107 \n 7.68.0 111 \n 7.69.0 113 \n 7.70.0 113 \n 7.71.0 99 \n 7.72.0 99 \n 7.73.0 96 \n 7.74.0 96<\/pre>\n\n\n\n
Current git: 96<\/strong> allocations.<\/p>\n\n\n\n
We do more allocations than some years back, but I think it is still within a reasonable growth. <\/p>\n\n\n\n
Peak memory allocations<\/h3>\n\n\n\n
Here’s some developments to look closer at!<\/p>\n\n\n\n
If we start out looking at the oldest versions in my test, we can see that they’re sub 100KB allocated – but we need to take into account the fact that back then we used a fixed 16KB download buffer. In curl 7.54.1 we bumped the default buffer size the curl tool uses to 100K which in the table below is visible in the 7.58.0 allocation.<\/p>\n\n\n\n
 7.50.0 84473 \n 7.52.0 85329 \n 7.58.0 174243 <\/span>\n 7.59.0 174315 \n 7.60.0 174339 \n 7.61.0 174531 \n 7.62.0 143886 <\/span>\n 7.63.0 143928 \n 7.64.0 144128 \n 7.65.0 143152 \n 7.66.0 168188 \n 7.67.0 173365 \n 7.68.0 168575\n 7.69.0 169167\n 7.70.0 169303 \n 7.71.0 136573\n 7.72.0 136765 \n 7.73.0 136875 \n 7.74.0 137043<\/pre>\n\n\n\n
Current git: 131680<\/strong> bytes.<\/p>\n\n\n\n
The gain in 7.62.0 was mostly the removal of the default allocation of the upload buffer, which isn’t used in this test…<\/p>\n\n\n\n
The current size tells me several things. We’re at a memory consumption level that is probably at its lowest point in the last decade – while at the same time having more features and being better than ever before. If we deduct the download buffer we have 29280<\/strong> additional bytes allocated. Compare this to 7.50.0 which allocated 68089<\/strong> bytes on top of the download buffer!<\/p>\n\n\n\n
If I change my curl to use the smallest download buffer size allowed by libcurl (1KB) instead of the default 100KB, it ends up peaking at: 30304<\/strong> bytes. That’s 44% of the memory needed by 7.50.0.<\/p>\n\n\n\n
In my opinion, this is very<\/em> good.<\/p>\n\n\n\n
It might also be worth to reiterate that this is with a full featured libcurl build. We can shrink even further if we switch off undesired features or just go tiny-curl<\/a>.<\/p>\n\n\n\n
I hope this goes without saying, but of course all of this work has been done with the API and ABI still intact.<\/p>\n\n\n\n
Graphs?<\/h2>\n\n\n\n
You know I like graphs<\/a>, but for now I decided this blog post and analysis was enough. I’m going to think about how we can perhaps get this info somehow floated on a more regular and automated way in the future. Not sure it is worth spending a lot of effort on though.<\/p>\n\n\n\n
Reproduce<\/h2>\n\n\n\n
  1. build curl with the --enable-debug<\/code> option to configure. Don’t use the threaded resolver – I use the c-ares one, because it otherwise breaks the memdebug system.<\/li>
  2. Run your command line with tracing enabled and then run memanalyze on the log:<\/li><\/ol>\n\n\n\n
    #!\/bin\/sh\nexport CURL_MEMDEBUG=\/tmp\/curlmem.log\n.\/src\/curl -v localhost\/512M -o \/dev\/null\n.\/tests\/memanalyze.pl -v \/tmp\/curlmem.log<\/pre>\n\n\n\n
    To get the struct sizes, just run pahole on the static libcurl lib after the build:<\/p>\n\n\n\n
    pahole -s lib\/.libs\/libcurl.a > sizes.txt<\/pre>\n\n\n\n
    Credits<\/h2>\n\n\n\n
    The photo was taken by me, in Siem Reap, Cambodia. “A smaller transport”<\/p>\n\n\n\n
    Updates<\/h2>\n\n\n\n
    After the initial posting of this article I optimized the structs even further so the numbers have been updated since then to reflect the state of what’s in git a week later.<\/p>\n","protected":false},"excerpt":{"rendered":"
    tldr: curl uses 30K of dynamic memory for downloading a large HTTP file, plus the size of the download buffer. Back in September 2020 I wrote about my work to trim curl allocations done for FTP transfers. Now I’m back again on the memory use in curl topic, from a different angle. This time, I … Continue reading More on less curl memory<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":5,"featured_media":15610,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[33],"class_list":["post-15565","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-curl","tag-curl-and-libcurl"],"_links":{"self":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/15565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/comments?post=15565"}],"version-history":[{"count":54,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/15565\/revisions"}],"predecessor-version":[{"id":15829,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/15565\/revisions\/15829"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media\/15610"}],"wp:attachment":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media?parent=15565"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/categories?post=15565"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/tags?post=15565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}