Hi,
This task is a component of a larger API. There is no interface design with this and we'll be testing with a Rest client.
My application needs to tail multiple log files (like from /var/log) and write that data into mongodb. There is some pre-processing that take place for each line and I would like to manage that in the database. The pre-processing is things like changing names of fields before insert, setting category based on conditiions, or parsing non-json strings. Also, log file line number needs to be saved in the event of an error or shutdown.
The package I would like to use is [login to view URL]
I have so far mongoose schemas for the logging source (the file) and the log itself. After starting a private conversation, I will share those out.
Requirements:
-Ability to tail multiple log files and start tailing a new one without restarting the application
-Ability to retrieve a list of available logs and extract a sample of log file (this is used on the front end for adding a logging source)
-Ability to store information about the logging source such as name, description, rules, etc...
-Pre-processing of log based on basic rules
Thanks,
Hello,
Thanks for posting this requirement.
My understanding is:-
1. Using PHP you want an API to be written, which will accept file name and then last n lines of those files would be saves in MongoDb using some preprocessing for each line.
Kindly correct me if I am wrong.
Thanks
Dear Prospect Hiring Manager.
Thank you for giving me a chance to bid on your project. i am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned
I have checked your requirements.i have right skills to work on this assignment
my award = superb result = happy client. In a good partnership, good results happen. Good cooking makes good eating!BWe consider our client as our partner.
can u provide your email or sky-pe etc for further discussion about the project.
I am ready to discuss with you
with best Regards
Hello,
I have worked extensively on these kind of applications before and mongo db also. But my expertise mostly lies in java and scala. If its okay with you, would it be fine to go with scala programming language. Its completely new and very famous these days, and lot better performant than Nodejs. If that suits you, please revert back.
Hello,
an interesting project indeed. I just want to know are those log files that you are using generated by your application, or some 3rd party app. If so the better solution would be to modify the logging adapter, if possible, to insert the data directly in the DB. This way we could avoid tailing of the files as these operations are expensive.
I would like to see your schema and to explain how this parsing of log entries would be done.
BR,
Vladimir
Hi! I can help.
Watch -> Tail -> Parse -> Model -> Save -> API
Are there multiple log file formats or just one?
Would there be an API call to list available logs, watch log, and also stop watching?