Death by a thousand slops

I have previously blogged about the relatively new trend of AI slop in vulnerability reports submitted to curl and how it hurts and exhausts us.

This trend does not seem to slow down. On the contrary, it seems that we have recently not only received more AI slop but also more human slop. The latter differs only in the way that we cannot immediately tell that an AI made it, even though we many times still suspect it. The net effect is the same.

The general trend so far in 2025 has been way more AI slop than ever before (about 20% of all submissions) as we have averaged in about two security report submissions per week. In early July, about 5% of the submissions in 2025 had turned out to be genuine vulnerabilities. The valid-rate has decreased significantly compared to previous years.

We have run the curl Bug Bounty since 2019 and I have previously considered it a success based on the amount of genuine and real security problems we have gotten reported and thus fixed through this program. 81 of them to be exact, with over 90,000 USD paid in awards.

End of the road?

While we are not going to do anything rushed or in panic immediately, there are reasons for us to consider changing the setup. Maybe we need to drop the monetary reward?

I want us to use the rest of the year 2025 to evaluate and think. The curl bounty program continues to run and we deal with everything as before while we ponder about what we can and should do to improve the situation. For the sanity of the curl security team members.

We need to reduce the amount of sand in the machine. We must do something to drastically reduce the temptation for users to submit low quality reports. Be it with AI or without AI.

The curl security team consists of seven team members. I encourage the others to also chime in to back me up (so that we act right in each case). Every report thus engages 3-4 persons. Perhaps for 30 minutes, sometimes up to an hour or three. Each.

I personally spend an insane amount of time on curl already, wasting three hours still leaves time for other things. My fellows however are not full time on curl. They might only have three hours per week for curl. Not to mention the emotional toll it takes to deal with these mind-numbing stupidities.

Times eight the last week alone.

Reputation doesn’t help

On HackerOne the users get their reputation lowered when we close reports as not applicable. That is only really a mild “threat” to experienced HackerOne participants. For new users on the platform that is mostly a pointless exercise as they can just create a new account next week. Banning those users is similarly a rather toothless threat.

Besides, there seem to be so many so even if one goes away, there are a thousand more.

HackerOne

It is not super obvious to me exactly how HackerOne should change to help us combat this. It is however clear that we need them to do something. Offer us more tools and knobs to tweak, to save us from drowning. If we are to keep the program with them.

I have yet again reached out. We will just have to see where that takes us.

Possible routes forward

People mention charging a fee for the right to submit a security vulnerability (that could be paid back if a proper report). That would probably slow them down significantly sure, but it seems like a rather hostile way for an Open Source project that aims to be as open and available as possible. Not to mention that we don’t have any current infrastructure setup for this – and neither does HackerOne. And managing money is painful.

Dropping the monetary reward part would make it much less interesting for the general populace to do random AI queries in desperate attempts to report something that could generate income. It of course also removes the traction for some professional and highly skilled security researchers, but maybe that is a hit we can/must take?

As a lot of these reporters seem to genuinely think they help out, apparently blatantly tricked by the marketing of the AI hype-machines, it is not certain that removing the money from the table is going to completely stop the flood. We need to be prepared for that as well. Let’s burn that bridge if we get to it.

The AI slop list

If you are still innocently unaware of what AI slop means in the context of security reports, I have collected a list of a number of reports submitted to curl that help showcase. Here’s a snapshot of the list from today:

  1. [Critical] Curl CVE-2023-38545 vulnerability code changes are disclosed on the internet. #2199174
  2. Buffer Overflow Vulnerability in WebSocket Handling #2298307
  3. Exploitable Format String Vulnerability in curl_mfprintf Function #2819666
  4. Buffer overflow in strcpy #2823554
  5. Buffer Overflow Vulnerability in strcpy() Leading to Remote Code Execution #2871792
  6. Buffer Overflow Risk in Curl_inet_ntop and inet_ntop4 #2887487
  7. bypass of this Fixed #2437131 [ Inadequate Protocol Restriction Enforcement in curl ] #2905552
  8. Hackers Attack Curl Vulnerability Accessing Sensitive Information #2912277
  9. (“possible”) UAF #2981245
  10. Path Traversal Vulnerability in curl via Unsanitized IPFS_PATH Environment Variable #3100073
  11. Buffer Overflow in curl MQTT Test Server (tests/server/mqttd.c) via Malicious CONNECT Packet #3101127
  12. Use of a Broken or Risky Cryptographic Algorithm (CWE-327) in libcurl #3116935
  13. Double Free Vulnerability in libcurl Cookie Management (cookie.c) #3117697
  14. HTTP/2 CONTINUATION Flood Vulnerability #3125820
  15. HTTP/3 Stream Dependency Cycle Exploit #3125832
  16. Memory Leak #3137657
  17. Memory Leak in libcurl via Location Header Handling (CWE-770) #3158093
  18. Stack-based Buffer Overflow in TELNET NEW_ENV Option Handling #3230082
  19. HTTP Proxy Bypass via CURLOPT_CUSTOMREQUEST Verb Tunneling #3231321
  20. Use-After-Free in OpenSSL Keylog Callback via SSL_get_ex_data() in libcurl #3242005
  21. HTTP Request Smuggling Vulnerability Analysis – cURL Security Report #3249936

13 thoughts on “Death by a thousand slops”

  1. I’m not sure how successful it would be, but could you use the reputation system in the opposite way? I.e. when someone has submitted x amount of verified issues they are then eligible to receive a bounty?

    It might just be the worst of all cases of course.

    1. Seems like multiple separate reputation gates would be needed. Only offering bounties via the official channel on hackerone to users with high reputation. And people who discovered genuine exploits, but don’t have a high reputation account, could get somebody else to vouch for them (to discourage the slop problem simply being moved, the people who can vouch wouldn’t do so for total strangers).

  2. If people are sure in their bug – reproduced it themselves, then you could charge them small amount before submission and then give back that amount + bounty. The problem here is that all of these are not reproducible. So if a person generates AI bug report and validates it – they could pay and receive a reward. Or if they thought they validated it but in reality didn’t – then they would lose money, educating public not to believe in AI reports.

  3. Charging small fee upon a bug submit for new users without enough reputation sounds like a sane way to filter out bad reports. The fee can be directed to fund fixing of real issues or returned if it’s a valid report.

    As an adult IT professional, I see no other universal way to battle increasing low effort AI slop. The entry bar must be raised and money is a good entry bar for the cheap low effort AI slop

  4. I had a look at some of these AI-generated submissions that you’ve linked. On one hand I think you’re being overly polite and respectful in these discussions. When you press for the details and you are not receiving them, you should probably close the bug immediately.

    On the other hand, in the world where machines are taking over, it’s important to behave like a human, so kudos for that.

    1. True, you should close it immediately if it smells of AI slop, then give them a few weeks; if they have something to say, they will reply to it, and in case it is valid and you made a mistake, you can always change the status of the report.

    2. this isn’t really machines taking over so much as the wave of eternal september reaching foss’s shores

  5. You are way to polite. When you see the telltale signs of ChatGPT (em dash, bullet points, bold font, words like “delve”), just close the ticket and stop reading. It’s a mistake to give the benefit of the doubt to every single opened ticket especially if it’s from a new account. Just bin them and forget they ever existed. There’s no reason not to. In those listed tickets, there are numerous examples of you guys noticing stuff that’s 100% hallucinated by AI and yet you keep talking to them. Of course it’s taking you hours per week, you’re choosing to.

  6. Hi Daniel,

    I am following your situation with AI slop very closely. To my understanding the cost to generate a (slopy) security report has become very cheap and HackerOne as a platform makes it quite easy to submit it to you without going through quality gates.

    Does it makes sense to increase the cost of submitting and generating a security report? I think of CI-pipline-like quality gates that check for arbitrary checks. Some proof-of-work checks that aim to increase the cost for slop reports.

    Kind regards,

    Janik

  7. There was a time we were signing gpg keys for people we knew. Eventually you would “trust” a key because several people you knew would trust them.

    Maybe it’s time to get something similar: a decentralized trust model. You would pretty much toss requests from people who you or your peers wouldn’t “trust”.

    If someone reset their identities (ie their keys), they would start from scratch. If you see someone that has been dropped and now has a different key with similar trust signs, you drop the people who ‘trusted’ that person.

    In a nutshell, works like social karma chain. Would be a little difficult for people to get in the field, but that’s ok as they will need to ‘build’ trust over time just like in every other area.

    Anyway, software development is not the only area being negatively affected by ai sloppiness. For recruiting for example, we can barely trust the person we’re interviewing isn’t using AI addons during the interview process. The solution seems to tend on rely more on human interactions: in person interview and/or internal referrals from people we trust – which is counter intuitive given than technology is pushing us to do things the way we used to do 10 years ago.

  8. You could use Bitcoin to put a money firewall to stop slop.
    Then its purely tehnical, no banks and other friction.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.