Understanding the OnlyFans API for a Private Scraping Project

[thecodergus]

Hello everyone,

I'm currently working on a personal project that involves web scraping, and I'm interested in understanding more about the OnlyFans API. I have a very strong background in Python and have done some web scraping on HTML content before, but this is my first time working with an authenticated API like OnlyFans.

I understand that the use of private or undocumented APIs can sometimes be against the terms of service, so I want to make sure I'm approaching this in the right way. I'm looking to create a private and personal scraper, not for commercial use or distribution.

Does anyone have experience with this API or know how it functions? I'm particularly interested in how the authentication process works. Any tips, relevant articles, forum threads, or other resources would be greatly appreciated.

Please note that I'm not looking for proprietary information or anything that would violate OnlyFans' terms of service. I'm just trying to understand the general workings of the API to better inform my project.

Thank you in advance for your help!

Best regards, Gus.

[taux1c]

One of the big things that you should keep in mind is that these apis are always updated and every time they are updated you have to pretty much reverse engineer the site again to update it. In the case of onlyfans they were rotating some different types of code at random which broke most scrapers. However; after I began using logic to test and search for names they didn't do that as much. However; they did eventually change the whole api.

This being said, you can learn a lot from reviewing the code in highlights.py that I re-designed with the logic. Sites generally use ajax to run api calls in the background so you have to find the ajax call in the code and make that call with the requests module then capture the returned output then iter that content.

example something like response = requests.get('https://site.com/api/posts?x=500&key=asdf02389ds&s=23jlsdgh')

You could in the example above find all of the information in the response object. To see what all is in there you could do something like
print(dir(response))

or you could use json = response.json (if there is json data)

or html = resposne.content

The list of things you can do with this goes on but the big thing is finding that api url and making the api call with requests just like you would with a regular scraper.

For authentication, you can use something like insomnia to make life a bit easier. You could then just use the inspect pannel to pull the request from the network tab and just copy that for curl and post in insomnia. You can then use that as a little sandbox to make calls to the site and see what is required and what isn't. Test to see if post and get work or just one, etc.

[thecodergus]

Thank you for your valuable insights and suggestions. I want to share with you that I did indeed notice a specific pattern with the "sign" header field. The middle part changes with each request, and I hypothesized that there's a client-server validation using a timestamp-based signature, possibly combined with some cryptographic algorithm (maybe SHA265 or another) and key.

I decided not to delve deeper into this complex mechanism, especially considering the ever-changing nature of these systems. Instead, I opted to create a bot using Selenium, even adding an intelligent bot agent to handle captchas. It's been an interesting journey, and I've managed to achieve most of my goals, though there are still some limitations with video scraping that I haven't tackled yet.

I really liked your tip about using tools like Insomnia and focusing on the API calls. It's cool to have this kind of technical chat with someone who gets it. Sharing information like this not only helps us individually but also fosters creativity and growth within the community. Let's keep this collaboration going!

Best, Gus.

[taux1c]

You are correct, it uses time and user id, session id, and other stuff if I remember correctly then makes a checksum. (This is generated for every request so in my new version I am looking at building I am making a decorator to build the sign along with a make request function.

Selenium is AMAZING! I really would like to learn more about it but I just haven't had the time or interest of late. I have messed with it a bit just to open and close, visit urls, etc. My plan was to have selenium open a window that you put your username and password in (to sign into the onlyfans site) and then it would auto grab your cookies, x-bc, session, etc. then write that to the json and import to the config as normal.

Absolutely, glad that it helped! Community and communication is key!

[taux1c]

Side note, I pulled this from the code. If you ever want to try your hand at the calls this is the code to create the sign (if it hasn't changed).

def create_sign(link, headers):
    """
    credit: DC and hippothon
    """
    content = read_request_auth()

    time2 = str(round(time.time() * 1000))

    path = urlparse(link).path
    query = urlparse(link).query
    path = path if not query else f"{path}?{query}"

    static_param = content['static_param']

    a = [static_param, time2, path, headers['user-id']]
    msg = "\n".join(a)

    message = msg.encode("utf-8")
    hash_object = hashlib.sha1(message)
    sha_1_sign = hash_object.hexdigest()
    sha_1_b = sha_1_sign.encode("ascii")

    checksum_indexes = content['checksum_indexes']
    checksum_constant = content['checksum_constant']
    checksum = sum(sha_1_b[i] for i in checksum_indexes) + checksum_constant

    final_sign = content['format'].format(sha_1_sign, abs(checksum))

    headers.update(
        {
            'sign': final_sign,
            'time': time2
        }
    )
    return headers

[Tayluk]

hey, could you drop us a message. We'll pay for your knowledge here... :)

[thecodergus]

At first I still don't have a good idea of how the API handles and separates true signatures from false ones, but I appreciate the prepared code and I will measure and play to test ideas that may occur to me.

[taux1c]

The big thing to remember is that the api is always changing and any public code that they find that works with their api is then used or at least taken into consideration for future site updates. I was going to change some of my code and make it private but then that turns into a whole other can of worms. I just thought, you know what. I could hide the logic I use to do some of this and it wouldn't be so easy to break it.

[oaklabsdev]

A bit late but - I'm building a tool where OF data is core. I reaaaalllllyyy want to use reverse API, but you've made it clear that OF is constantly interfering with that method, which would be a nightmare in production with tons of a tool users.

I guess I already know the answer - use smart, robust scraping through Selenium or similar for more stable, long-term data extraction. Just curious if you have any thoughts on this decision since December haha

[FebX23]

When dealing with the OnlyFans API or any similar authenticated service, you're essentially stepping into a realm where access is locked behind authentication mechanisms. With your Python background, you're in a good position to tackle this. Here are the key points:

• Authenticated APIs require specific credentials. For OnlyFans, this likely means handling tokens or session keys that prove your access rights. You’d use these in your headers or requests to authenticate your scraping bot.

• Legal and Ethical Considerations: Super important to stress this—make sure your project doesn't clash with OnlyFans' terms. Unauthorized scraping can be a legal and ethical minefield.

• Practical Approach: If you're exploring how authentication works without stepping over any lines, consider tools like Postman for testing API requests. For a Python-focused approach, libraries like requests or httpx are your best friends for handling web requests, including authenticated ones.

Since you mentioned looking for tools, CrawlBase could be a resourceful tool for scraping projects, offering solutions that might navigate around common scraping challenges. However, always double-check the terms of service to ensure compliance.

Remember, the key to working with any API is understanding its documentation and respecting its usage policies.