We need a parser that is run from crontab and parses the newest files from the source directory.
About 80% of the project is completed.
The parser works, but has no error handling and the file name is hardcoded.
The attached script has sensitive data removed, but is otherwise functional.
Old project tasks to complete:
1. Read the specification file into a MySQL table. (one time, not automated task) (will be sent after proj accepted)
2. Script writes the file name of the parsed file into a third table. If any errors occurred during the parse, then it writes a binary flag and dumps errors into a BLOB/TEXT.
Add one field as a flag that is false if no errors and true if errors.
The error messages should be human readable.
Required error cases are: 1. file incomplete (all rows are fixed length), 2 row/field contains data that is the wrong type and 3. directory/file inaccessible/unreadable etc.
Currently only #3 is implemented in the attached script.
If possible, #2 should be implemented using warnings from mysql. e.g trying to cast hex/alphanumeric into integer.
The script uses the DBI database interface. This is the most reliable choice for our platform, but unfortunately its warning and error handing sucks.
We are not interested in swapping it out for something else.
If it is impossible to capture warnings from the DBI writes, then our fallback plan is to read how many rows have been written and the sum of one of the accounting fields. These 2 numbers are compared to the result of the read file. If they match, everything is OK.
New tasks:
3. We receive 2 files of identical format. The script should do a loop and read both files. The data in the files are written to different tables, but errors are written to the same log table.
4. Add command line argument that display the proposed tables scheme and another that displays debug/verbose and usage. Normally the script is run from crontab, so it should only print to STDOUT or STDERR if there is an actual error. Crontab logging is complementary to the error logging into the mysql log table (which is my primary way of checking for errors). Except for DB connect errors which can not be logged to mysql.
5. Add deterministic ID for the log and data file because our master-master setup prevents the use of auto-incremented indexes.
Lets use a massive index with contingency for up to 99999 row per day.
Row 55 of [login to view URL] will be 15021200055.
For the "log" table, use 1502121 for cdr_xxx and 1502122 for in_cdr_xxx.
Lets use the date from the first row in the file, we assume that there will never be 2 days of data in any file.
6. Add creation of the log table to sub create_table.
Bid range = 50 - 100 EUR
Bid deadline is Thursday 05/03/2012 midnight EET
I've made hundreds of such scripts in my life (mostly processing some CDR files from platforms in the GSM world, either reported by some platforms or directly extracted from network traffic sniff like SS7 or TCP/IP).
Hello. More 20 years programming experience.
Regards.
---------------------------------------------------------------------------------------------------------------------------------------------------
Hello there,
I am an excellent Perl programmer. I am also a good communicator and a proactive one at that. I would also be available through Skype on the time zones comfortable for you. Hope to know more from you on this project.
Regards,
ClearKruti
Thu, 05 Mar 2015 05:05:48 +0530
If on windows platform, error log from running the script may help.
Please advise on MySQL version and perl -v:config_args
Test script can be tried to check for needed updates
Hi, expert programmer and web/data scraper here with over 19 years experience in programming and RDBMS - please see my reviews.
I'm using Perl for this kind of jobs.
I'm able to finish fast.
I have good exposure on Perl, my present work experience is exactly the same. so i can finish the project as early i can if requirements are clear from your end.
I am working in same kinf of the enviroment on APNS/GCM notification server .In which we are doing similar kind of work with the user notifcation records stored in the database in mysql and also creating the CDR and logs for the same processing.
I think i can delivery this project in cost effective and quality manner once we getting in more detail discussion for the project requirement and considerations.