Automating myself out of an unloved project with Python, n8n and Telegram

In 2020, I built my first side project. I "scratched my own itch" and started selling it, and since then, the project has earned me over $15,000. However, just a few months after its release, I was so overwhelmed by maintenance work that I considered shutting it down. Instead of killing the goose that lays (small) golden eggs, I decided to automate the project.

In this presentation, I will share how I used the fair-code licensed tool n8n to automate interactions with users. By connecting custom Python scripts, a web scraper, and integrating a Telegram bot, I created a system for fast and easy interactions with my project.

I'll also discuss the highs and lows of building a side project. As a software developer, I always saw a project with recurring revenue as the "holy grail." After putting in the initial effort to build and launch it, I thought I could sit back and watch the money roll in. Not quite. Building the project is just the beginning—you'll also need to handle customers, combat fraud, and manage the challenges that can quickly lead to burnout.

Transcript

1. Automating myself out of an unloved project with Python, n8n and Telegram Sebastian Witowski
2. 2020 was a weird year
3. 2020 was a weird year
4. 2020 was a weird year
5. Robinhood's Active Users chart source: https:/ /www.sec.gov/Archives/edgar/data/1783879/000162828021013986/robinhoods-1a1.htm 2020 was a weird year
6. My 2020 was special
7. My 2020 was special
8. My 2020 was special
9. My 2020 was special
16. collect payments
17. collect payments published articles
18. collect payments published articles promoted my scripts
19. My first sale!
22. I started getting A LOT of emails
23. I started getting A LOT of emails •How does it work?
24. I started getting A LOT of emails •How does it work? •Can I get a free access to your scripts?
25. I started getting A LOT of emails •How does it work? •Can I get a free access to your scripts? •Can you add feature X?
26. I started getting A LOT of emails •How does it work? •Can I get a free access to your scripts? •Can you add feature X? •Should I buy/sell stock Y?
27. I started getting A LOT of emails •How does it work? •Can I get a free access to your scripts? •Can you add feature X? •Should I buy/sell stock Y?
28. Chargebacks
29. Fraud
30. Fraud
31. Fraud
32. New job Day-trading
37. Turning the autopilot on
38. What to automate? •Giving access
39. What to automate? •Giving access •Removing access
40. What to automate? •Giving access •Removing access •Giving a 7-day trial access
41. How?
42. How? vs.
43. How?
44. Giving access to new subscribers Gumroad New subscription Welcome email Gumroad Ping n8n Webhook Python code
45. from pyppeteer import launch from pyppeteer.browser import Browser from pyppeteer.page import Page Log in
46. from pyppeteer import launch from pyppeteer.browser import Browser from pyppeteer.page import Page # I've removed my username and any other confidential information MAIN_URL = "https: // www.tradingview.com/u/<my-username>/" LOGIN_URL = "https: // www.tradingview.com/accounts/signin/" USERNAME = os.environ["TV_USERNAME"] PASSWORD = os.environ["TV_PASSWORD"] async def login(page: Page): await page.goto(LOGIN_URL, {"waitUntil": "networkidle2"}) await page.click('button[name="Email"]') await page.waitForSelector("input[name=id_username]") await page.type("input[name=id_username]", USERNAME) await page.type("input[name=id_password]", PASSWORD) submit_button = await page.xpath(" / / button[contains(., 'Sign in')]") await submit_button[0].click() await asyncio.sleep(3) await page.goto(MAIN_URL, {"waitUntil": "networkidle2"}) Log in
47. async def add_access(script: str, user: str, trial: bool = False): """Handle the logic for adding access to a script for a given user.""" browser, page = await launch_browser() await login(page) resp = await check_access_json(page, script, user) ... Add access
48. async def check_access_json(page: Page, script: str, user: str): """Check access to a script for a given user and return a JSON response.""" payload = { "pine_id": INDICATOR_IDS[script], "username": user, } return await post_request(page, "https: / / www.tradingview.com/pine_perm/list_users/", payload) Helpers for sending POST requests async def post_request(page: Page, url: str, params: dict) -> dict: """Evaluate a JavaScript code with POST request in the context of the current page.""" form = "let formData = new FormData();\n" for key, value in params.items(): form += f"formData.append('{key}', '{value}');\n" payload = "() => {\n" + form payload += f'return fetch("{url}",' payload += '{"credentials": "include", "mode": "cors", "method": "POST", "body": formData}' payload += ").then(res = > res.json());}" return await page.evaluate(payload)
49. async def add_access(script: str, user: str, trial: bool = False, date: Optional[str] = None): """Handle the logic for adding access to a script for a given user.""" browser, page = await launch_browser() await login(page) resp = await check_access_json(page, script, user) if results := resp["results"]: if "expiration" not in results[0]: output = f"User '{user}' already has non-expiring access to script {script}!" await browser.close() logging.info(output) return output else: # Remove existing temporary access logging.info(f"User '{user}' has temporary access to script {script}") resp = await remove_access_json(page, script, user) if resp["status"] != "ok": error = f"Something went wrong when removing access to {script} from user {user}: {resp}" await browser.close() logging.info(error) return error logging.info("Temporary access REMOVED") if trial: end_date = datetime.date.today() + datetime.timedelta(days=8) expiration_str = f"{end_date.isoformat()}T23 : 59 : 59.999Z" elif date: try: end_date = datetime.datetime.strptime(date, "%Y%m%d").date() except ValueError: error = f"Invalid end date format: '{date}'! It should be like this: 20211231" await browser.close() logging.info(error) return error expiration_str = f"{end_date.isoformat()}T23 : 59 : 59.999Z" else: expiration_str = None resp = await add_access_json(page, script, user, expiration_str) if resp.get("status") == "ok": until = f"until {expiration_str}" if expiration_str else "non-expiring" output = f"Access to script {script} ADDED for user {user} [{until}]" await browser.close() logging.info(output) return output else: error = f"Something went wrong when adding access to {script} for user {user} [{expiration_str}]]: {resp}" await browser.close() logging.info(error) return error Add access
50. async def add_access(script: str, user: str, trial: bool = False, date: Optional[str] = None): """Handle the logic for adding access to a script for a given user.""" open_browser() login() if user_has_access(): return "user already has access" remove_trial_access() if trial: grant_access_until(date.today() + 7) return "access granted for 7 days" elif date: grant_access_until(date) return f"access granted until {date}" else: grant_permanent_access() return "permanent access granted" Add access (pseudocode)
51. CLI import asyncio import click from .api import add_access, remove_access @click.group() def cli(): pass @cli.command() @click.argument("user") @click.argument("script") @click.argument("until", required=False) def add(user, script, until): trial = False expiration = None if until: if until == "t": trial = True elif until.isnumeric(): expiration = until result = asyncio.run(add_access(script, user, trial, expiration)) click.echo(result) if __ name __ = = " __ main __ ": cli()
52. CLI import asyncio import click from .api import add_access, remove_access @click.group() def cli(): pass @cli.command() @click.argument("user") @click.argument("script") @click.argument("until", required=False) def add(user, script, until): trial = False expiration = None if until: if until == "t": trial = True elif until.isnumeric(): expiration = until result = asyncio.run(add_access(script, user, trial, expiration)) click.echo(result) @cli.command() @click.argument("user") @click.argument("script") def remove(script, user): result = asyncio.run(remove_access(script, user)) click.echo(result) if __ name __ = = " __ main __ ": cli()
53. Full workflow
54. Full workflow
55. Creating a new Telegram bot
56. Notifications at every step of the workflow
57. Deployment
58. Deployment $5/month VPS server
59. Deployment $5/month VPS server + n8n Docker container https://docs.n8n.io/hosting/installation/server-setups/digital-ocean/
60. Deployment $5/month VPS server + n8n Docker container https://docs.n8n.io/hosting/installation/server-setups/digital-ocean/ + reverse proxy
62. Locally I could use whatever Python package I wanted. But how do I install Python packages for n8n on a remote server?
63. Custom Docker image
64. Custom Docker image # Dockerfile FROM n8nio/n8n # Install python/pip ENV PYTHONUNBUFFERED=1 RUN apk add -- update -- no-cache python3 && ln -sf python3 /usr/bin/python RUN python3 -m ensurepip RUN pip3 install - - no-cache -- upgrade pip setuptools # Install chromium RUN apk -U add chromium udev ttf-freefont COPY requirements.txt . # Install Python libraries RUN python -m pip install -r requirements.txt # Copy remaining files (cache busts here) COPY . . # ENDPOINT command will be used from the original n8n image
65. Automatically build docker image in Gitlab # .gitlab-ci.yml build: stage: build image: docker:24.0.7 services: - docker:24.0.7-dind variables: DOCKER_HOST: tcp: // docker:2375/ DOCKER_TLS_CERTDIR: "" script: - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" -- password-stdin "$CI_REGISTRY" - docker build -t "$CI_REGISTRY_IMAGE" "$CI_PROJECT_DIR" - docker push "$CI_REGISTRY_IMAGE"
66. Automatically update the docker image on VPS
67. Automatically update the docker image on VPS + + +
68. docker-compose.yml # docker-compose.yml version: "3" services: n8n: image: registry.gitlab.com/<username-redacted>/<reponame-redacted> restart: always ports: - '0.0.0.0 : 5678 : 5678' environment: - N8N_BASIC_AUTH_ACTIVE=true - N8N_BASIC_AUTH_USER - . .. volumes: - ${DATA_FOLDER}:/home/node/.n8n watchtower: image: containrrr/watchtower volumes: - /var/run/docker.sock:/var/run/docker.sock - /root/.docker/config.json:/config.json
69. I modify the Python code and test it locally I push to Gitlab Automatically triggered when I push new code Automatic check every 5 minutes Gitlab builds a new Docker image Watchtower pulls the new Docker image and restarts the containers
70. Gumroad New subscription Welcome email Gumroad Ping n8n Webhook > run CLI command New subscription bot Access granted bot Email sent bot
71. Trials
72. Trials n8n Trial form Airtable spreadsheet Airtable trigger > run CLI command Trial added bot
73. Trial form: Allows requesting access multiple times People: Request access multiple times
74. Trials n8n Trial form Airtable spreadsheet Jotform spreadsheet Airtable trigger Jotform trigger > run CLI command Trial added bot
75. Cancelled subscriptions
76. Cancelled subscriptions "Gumroad Ping" only works for new subscriptions.
77. Cancelled subscriptions "Gumroad Ping" only works for new subscriptions. For handling cancelled subscriptions, I would need to use full Gumroad API.
78. Cancelled subscriptions "Gumroad Ping" only works for new subscriptions. For handling cancelled subscriptions, I would need to use full Gumroad API. Granting access to new subscribers is urgent.
79. Cancelled subscriptions "Gumroad Ping" only works for new subscriptions. For handling cancelled subscriptions, I would need to use full Gumroad API. Granting access to new subscribers is urgent. Removing access isn't.
80. Cancelled subscriptions "Gumroad Ping" only works for new subscriptions. For handling cancelled subscriptions, I would need to use full Gumroad API. Granting access to new subscribers is urgent. Removing access isn't. I didn't have to automate removing access. I just had to make it convenient.
81. Talking back to my Telegram bot
82. Talking back to my Telegram bot
83. Talking back to my Telegram bot
85. Finally, a passive income!
86. How much maintenance work my project needs?
87. I haven't touched the source code for almost 2 years! How much maintenance work my project needs?
88. Lessons learned
89. Lessons learned • Building something that others will pay for is hard. Maintaining it is harder
90. Lessons learned • Building something that others will pay for is hard. Maintaining it is harder • Set boundaries
91. Lessons learned • Building something that others will pay for is hard. Maintaining it is harder • Set boundaries • Put a small obstacle to filter out the worst customers (e.g. a trial form)
92. Lessons learned • Building something that others will pay for is hard. Maintaining it is harder • Set boundaries • Put a small obstacle to filter out the worst customers (e.g. a trial form) • Maintaining a project that you no longer use is much harder
93. Lessons learned • Building something that others will pay for is hard. Maintaining it is harder • Set boundaries • Put a small obstacle to filter out the worst customers (e.g. a trial form) • Maintaining a project that you no longer use is much harder • You don't need the best automation tool to start with
94. https://switowski.com/blog/ More details can be found on my blog
95. https://github.com/switowski/blog-resources/tree/master/web-automation Repository with all the code for this presentation (not the real code behind my project):
96. Attributions • Broken heart: https:/ /www.midjourney.com/jobs/198a7150-9cc0-476b-b4ae-019189cbf755 • Software developer icon: https:/ /thenounproject.com/icon/software-developer-5124258/ • Software icon: https:/ /thenounproject.com/icon/software-7662616/ • Hammer icon: https:/ /thenounproject.com/icon/hammer-7887761/ • Money bag icon: https:/ /thenounproject.com/icon/money-bag-7905935/ • User icon: https:/ /thenounproject.com/icon/global-businessperson-5124234/ • Sweet passive income sign: https:/ /www.midjourney.com/jobs/cce38d20-898d-41f7-aab3-296a773cae13 • Not-so-sweet passive income: https:/ /www.midjourney.com/jobs/337cd30c-01d2-4c1b-9acd-a23899d24f69 • Fraud: https:/ /www.midjourney.com/jobs/0fb8e516-8dcb-4fec-a2ac-23d24c487759 • Turning on the autopilot: https:/ /unsplash.com/photos/pilot-controlling-airplane-dashboard-_nshY51PXHg Most of the drawings were generously provided by https:/ /undraw.co
97. Questions? Website: switowski.com Slides: https://speakerdeck.com/switowski Code: github.com/switowski/blog-resources/tree/master/web-automation