Parse Exchange Server Logs -- 2

Here below follows an example of the IIS logs for Exchange Server. There is one log file per day. The files can be big (Approx 20Mb is normal). The goal of this project is to parse the logs and extract the date and time, username, device name, device type, IP address (which is all contained in the longer log lines - last one in the example below). After extracting this information I then want to be able to report on it in a meaniful way. I'd like an expanding tree view (can be HTML page) with levels as listed below: 1. Username 2. Device Name (Device Type) 3. First Request time = xx:xx Last Request time = x:xx 4. Requests per hour 00:00 01:00 02:00 03:00 ...... 23:00 xx xx xx xx (xx = This shows the total number of requests in that hour) (Note, each username may have multiple devices which is why we need to seperate as some may be switched on all the time and others are used at specific times) At the top should have a start and end date selector (to select the range of dates to report on). The default should be yesterday (as both start and end date). There should be an option to expand all branches of the tree in one click. Example log: 2020-05-31 00:00:24 [login to view URL] POST /bosh/bosh/ - 443 - [login to view URL] - 404 0 2 46 2020-05-31 00:00:27 [login to view URL] POST /bosh/bosh/ - 443 - [login to view URL] - 404 0 2 15 2020-05-31 00:00:29 [login to view URL] POST /bosh/bosh/ - 443 - [login to view URL] - 404 0 2 31 2020-05-31 00:00:31 [login to view URL] POST /Microsoft-Server-ActiveSync/[login to view URL] Cmd=Ping&User=[login to view URL]%5Cusername&DeviceId=androidc316345793&DeviceType=Android&Log=V141_Fid:1_Sk:1648320286_Sst380_SsCmt380_BR1_BPR0_Fid:2_Sk:90664543_Sst22_SsCmt22_BR1_BPR0_Fid:459_Sk:1428453710_Sst24_SsCmt24_BR1_BPR0_LdapC1_LdapL16_RpcC60_RpcL156_Hb480_Rto1_Cpo9974_Fet480009_S1_Mbx:[login to view URL] _Throttle0_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f0%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5F7869507b-dfc2-4e1c-9e80-a89db9cef848%2cNorm_ 443 [login to view URL]\username [login to view URL] Android/9-EAS-2.0 200 0 1236 495384 When parsing the logs, I recommend to do this into a mysql database. The program will have a place to specify the log folder. It will search for newer logs since the last time. As the current day log will only be partial until midnight is reached it should always re-parse the last log file in full (since the last parse event) until the log file for the next day exists (then we know all logs have been parsed fully). Presentation should look nice and use some colour where needed if it helps. Thought should be given to optimise the code so it doesn't take too long to parse the logs and generate a report.