Server kills half way process of crawler build through python. Need help to find a solution

Hello, I can help you resolve the process kill issue on your linux. I will also create cronjob to run the crawler automatically on schedules. I have completed lots of similar project before for other clients. Let’s have a quick chat on this project to discuss the deadlines. Looking Forward to working with you. Thanks

Hello, Nice meet you! I have read your project requirements and then I am sure I can complete that project. I can help you. Thank you.

Hi mate, Sounds like the process is killed by kernel oom (out-of-memory), suggest looking into syslog for 'oom' string - it'll give some insights. Happy to help you with your both tasks, will need ssh access to your Linux box.

Hi, there. I read your project description carefully and I am interested in your proposal. I'm a Python and Web Scraping developer and I can help you quickly. Could you send me your code? I will check. If you hire me, you'll get a perfect result. Please contact me and let's discuss the details. Best regards. Andrey

Hi. I'm very interested in your description and as a Senior developer, I can complete your task perfectly and will be my best for you and provide the successful results that you want. Please contact me quickly for discussing your task in more detail. Let's scale your business together at a large pace. Looking forward to working with you. Thanks.

⭐⭐⭐I can start now and I am confident that I can do it ⭐⭐⭐ I have 10 years of experience in this field and I have these skills (Linux, Web Scraping, Google Sheets, Pythonand Google Apps Scripts) so just check my reviews and please have a look on my profile. let us discuss in chat

Hi Mr, I have been working with scrapping in many different projects and facing similar situations. Im sure I can help you with this problem. Kind regards.

Hi, Can You provide more details about that script or can you send it to me? What kind of linux is it?

Hi There, I am an expert python engineer with skills including Google Sheets, Linux, Python, Web Scraping and Google Apps Scripts. Please contact me to discuss more regarding this project. Thank you

Hi there, I am Ahmed a Python developer, I have read yoyr project requirements and I understand that you are facing a probelm with your process getting killed automatically by the server. I could solve that, and we will make a cron routine for that crawler... contact me over chat, and we will talk abouth the details ...

This project seems like a two task project: Task 1: Schedule running the crawler using crontab. This is fairly easy but on modern Linux servers it's probably better to use systemd units and timers for this. The reason being that it could capture logs better and restart the crawler on failure. Task 2: Fix the crawler being killed. This is likely more difficult if the solution isn't something simple like a program bug. If the server is killing your program because it's using too much memory or CPU then you might need to redesign your program's data structures a bit. I could tell you what you need to do but you may need to open another job for this if it's not an easy fix.