[{"id":328,"title":"Algorithms learning for large scale facilities","sha":"9a52fa2122dac91321c355073bcc25a3","user_id":null,"state":"published","body":"A large scale facility can be described as an object producing, as an output, datasets `D_i`, that scientists analyse to obtain results `R_i`. The ideal data analysis trajectory for an experiment is thus `D_i--\u003eR-i--\u003eP_i`, where `P_i` denotes the desired output: a publication.\r\n\r\nMost of the times, something goes wrong along the way: there is no time to analyse the data, the analysis is harder than expected... To speed up the process and make it more efficient, I suggest using a new approach, including machine learning methods. Once developed, it could be applied to various industries. \r\n\r\nWith time, a large set of algorithms `A_ij` is developed to analyse the data collected at the considered large scale facility. I propose to profile the algorithms, translating them to a high-level formal language, so to create a platform that \r\n\r\n- If the data collected during an experiment is similar to a previous dataset, suggests in a concise format which steps to follow.\r\n\r\n- If the experiment is new, it searches how similar data have been treated, and uses machine learning techniques to suggest a possible data analysis approach. \r\n\r\nThe system would help to speed up data analysis both for well developed and new techniques.","created_at":"2016-08-04T08:44:54.916Z","updated_at":"2016-08-16T06:00:24.570Z","zenodo_id":145365,"doi":"http://dx.doi.org/10.5281/zenodo.60165","subject":null,"tags":["machinelearning","large scale facilities","scientific programming","data analysis","data-manipulation"],"vote_count":0,"view_count":1,"score":0.1,"deleted":false,"muted":false,"tweeted":true,"attachment_file_name":null,"attachment_content_type":null,"attachment_file_size":null,"attachment_updated_at":null},{"id":478,"title":"Detecting the delimiter in CSVs that lie","sha":"f3e6d0c680c88caf54c1e4b0ae995768","user_id":null,"state":"published","body":"File extensions for data sharing sometimes lie about their contents. \r\n\r\nHere is an algorithm to infer the actual delimiter of a CSV, TSV or any related format:\r\n\r\n- Assume that alpha-numeric characters (A-Z, a-z, 0-9) and the period/full stop (.) are cannot be delimiters.\r\n- Begin with input text _a_.\r\n- Store a short sample of _a_ as _b_, by copying _n_ lines from the input text _a_.\r\n- Rank every character that appears in _b_ by frequency, creating candidate delimiters _c_\r\n- For every candidate delimiter _d_ in _c_, split each line in _b_. If every line has the same number of splits, _d_ is the delimiter.\r\n\r\nAn implementation of this idea is available [on Zenodo](https://zenodo.org/record/1323186) and within the [Python Package Index](https://pypi.org/project/detect-delimiter/).","created_at":"2018-07-29T08:12:03.204Z","updated_at":"2020-10-18T13:32:41.085Z","zenodo_id":1402269,"doi":"https://doi.org/10.5281/zenodo.1402269","subject":null,"tags":["text processing","data_exchange","data-manipulation"],"vote_count":2,"view_count":1,"score":2.1,"deleted":false,"muted":false,"tweeted":true,"attachment_file_name":null,"attachment_content_type":null,"attachment_file_size":null,"attachment_updated_at":null}]