Statistical Analysis Module

The project has to do with determining measurement anomalies. A system periodically measures a parameter and provides a double value. The task is to determine whether this value is an anomaly or a normal value. To accomplish this task, we have a number of historical data points from measurements, around 600 double values per parameter. Most of the values are small, but there are some large values. To detect the anomalies, we want to build the disctribution for these 600 points, meaning how many points have the value between N and N+1; then devide by the total number of points to get probability of getting a value in this range. Then for each new incoming value we get the probability and if it is below some probability then it is an anomaly. Simple example: if we have the set 1,1,1,1,5, we can clearly see that 5 happens only in 20% cases, but 1 happens in 80% cases. So , we know that 5 is an anomaly. See the "Deliverables" section for more information ## Deliverables What needs to be done is: - Create an object that will be the Anomaly Detector. - Develop two public methods: double [login to view URL] (string param_name, double current_value) bool [login to view URL](string param_name, double current_value) - When a method is called, you need to check if the statistics has already been collected and probabilities are cashed. IF not, then call the external method double[] GetParameterData( string param_name ) build the probability distribution given the range size ( visually, it is a bar histogram). Note, that some ranges may have no historical values ( see the picture ). In this case, approximation should be used to add the missing points. store it in SQL2000 database based on parameter name (all db operations must use SPs) mark, that the statistics for this parameter has been analysed and next time GetParameterData does not need to be called. take the current value, see within which range it falls. And return the appropriate value to the calls. ( GetProbability, IsAnomaly ) That is it for the main development project. Bonuses: - Show the histogram bar chart using the free library ZedGraph. - Build the memory cache for distribution values stored in database so that if the distribution is already in memory, the SPs do not have to be called. <HR> For your information here are the answers to a few question that I received regarding the project. Let me know if it is clear. If you think something below is outside the scope of the project for which you made the bid please let me know or change your bid. Thank you 1. Potentially unlimited number of parameters, but the usual number is between 1 and 40. The set does change from time to time, meaning, some parameters disappear from the set and there are some new once. 2. The check whether the statistics is collected could be implemented by retrieving a value from the database. This value could be a column in the same table as the parameter name column. When the statistics is calculated , this column is set to the datetime of the calculation. If we want to reset the statistics, we will reset the values (we will do this manually, no need to code anything here) 3. For missing range appoximation: Any reasonable method is ok for aproximation, (left + right)/2 is just fine. If more than one is missing, (left + right)/n where n is the number of missing parts plus one. 4. The output: We need a library; the app that displays the statistics could be a WinForms app. 5. We do not have enough processing power to recalculate on each new incoming point, there are many and they come frequently. We plan to do the recalculation if it is necessary by resetting the flag mentioned in 2. 6. We do want to sore the values in DB and NOT in XML file. Please ask more questions if something is not clear. ## Platform Win XP Visual Studio .NET 2005 C# MS SQL Server 2000 * * *