{"id":15772,"date":"2021-02-16T11:39:10","date_gmt":"2021-02-16T10:39:10","guid":{"rendered":"https:\/\/daniel.haxx.se\/blog\/?p=15772"},"modified":"2021-02-16T11:39:10","modified_gmt":"2021-02-16T10:39:10","slug":"transfers-vs-connections-spring-cleanup","status":"publish","type":"post","link":"https:\/\/daniel.haxx.se\/blog\/2021\/02\/16\/transfers-vs-connections-spring-cleanup\/","title":{"rendered":"Transfers vs connections spring cleanup"},"content":{"rendered":"\n

Warning: this post is full of libcurl<\/a> internal architectural details and not much else.<\/p>\n\n\n\n

Within libcurl there are two primary objects being handled; transfers and connections. The transfers objects are struct Curl_easy<\/span><\/code> and the connection counterparts are struct connectdata<\/span><\/code>.<\/p>\n\n\n\n

This is a separation and architecture as old as libcurl, even if the internal struct names have changed a little through the years. A transfer is associated with none or one connection object and there’s a pool with potentially several previously used, live, connections stored for possible future reuse.<\/p>\n\n\n\n

A simplified schematic picture could look something like this:<\/p>\n\n\n\n

\"\"
One transfer, one connection and three idle connections in the pool.<\/figcaption><\/figure>\n\n\n\n

Transfers to connections <\/h2>\n\n\n\n

These objects are protocol agnostic so they work like this no matter which scheme was used for the URL you’re transferring with curl.<\/p>\n\n\n\n

Before the introduction of HTTP\/2 into curl, which landed for the first time in September 2013 there was also a fixed relationship that one transfer always used (none or) one connection and that connection then also was used by a single transfer. libcurl then stored the association in the objects both ways. The transfer object got a pointer to the current connection and the connection object got a pointer to the current transfer.<\/p>\n\n\n\n

Multiplexing shook things up<\/h2>\n\n\n\n

Lots of code in libcurl passed around the connection pointer (conn<\/span><\/code>) because well, it was convenient. We could find the transfer object from that (conn->data<\/span><\/code>) just fine.<\/p>\n\n\n\n

When multiplexing arrived with HTTP\/2, we then could start doing multiple transfers that share a single connection. Since we passed around the conn<\/span><\/code> pointer as input to so many functions internally, we had to make sure we updated the conn->data<\/span><\/code> pointer in lots of places to make sure it pointed to the current driving transfer.<\/p>\n\n\n\n

This was always awkward and the source for agony and bugs over the years. At least twice I started to work on cleaning this up from my end but it quickly become a really large work that was hard to do in a single big blow and I abandoned the work. Both times.<\/p>\n\n\n\n

Third time’s the charm<\/h2>\n\n\n\n

This architectural “wart” kept bugging me and on January 8, 2021 I emailed the curl-library<\/a> list to start a more organized effort to clean this up:<\/p>\n\n\n\n

Conclusion: we should stop using ‘conn->data’ in libcurl<\/p>

Status: there are 939 current uses of this pointer<\/p>

Mission: reduce the use of this pointer, aiming to reach a point in the future when we can remove it from the connection struct.<\/p><\/blockquote>\n\n\n\n

Little by little<\/h2>\n\n\n\n

With the help of fellow curl maintainer Patrick Monnerat<\/strong> I started to submit pull requests that would remove the use of this pointer.<\/p>\n\n\n\n

Little by little we changed functions and logic to rather be anchored on the transfer rather than the connection (as data->conn<\/span><\/code> is still fine as that can only ever be NULL or a single connection). I made a wiki page to keep an updated count of the number of references. After the first ten pull requests we were down to just over a hundred from the initial 919 – yeah the mail quote says 939 but it turned out the grep pattern was slightly wrong!<\/p>\n\n\n\n

We decided to hold off a bit when we got closer to the 7.75.0 release so that we wouldn’t risk doing something close to the ship date that would jeopardize it. Once the release had been pushed out the door we could continue the journey.<\/p>\n\n\n\n

Gone!<\/h2>\n\n\n\n

As of today, February 16 2021, the internal pointer formerly known as conn->data<\/span><\/code> doesn’t exist anymore in libcurl and therefore it can’t be used and this refactor is completed. It took at least 20 separate commits to get the job done.<\/p>\n\n\n\n

I hope this new order will help us do less mistakes as we don’t have to update this pointer anymore.<\/p>\n\n\n\n

I’m very happy we could do this revamp without it affecting the API or ABI in any way. These are all just internal artifacts that are not visible to the outside.<\/p>\n\n\n\n

One of a thousand little things<\/h2>\n\n\n\n

This is just a tiny detail but the internals of a project like curl consists of a thousand little details and this is one way we make sure the code remains in a good shape. We identify improvements and we perform them. One by one. We never stop and we’re never done. Together we take this project into the future and help the world do Internet transfers properly.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n","protected":false},"excerpt":{"rendered":"

Warning: this post is full of libcurl internal architectural details and not much else. Within libcurl there are two primary objects being handled; transfers and connections. The transfers objects are struct Curl_easy and the connection counterparts are struct connectdata. This is a separation and architecture as old as libcurl, even if the internal struct names … Continue reading Transfers vs connections spring cleanup<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":5,"featured_media":14784,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[33],"class_list":["post-15772","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-curl","tag-curl-and-libcurl"],"_links":{"self":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/15772","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/comments?post=15772"}],"version-history":[{"count":21,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/15772\/revisions"}],"predecessor-version":[{"id":15794,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/posts\/15772\/revisions\/15794"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media\/14784"}],"wp:attachment":[{"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/media?parent=15772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/categories?post=15772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/daniel.haxx.se\/blog\/wp-json\/wp\/v2\/tags?post=15772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}