[{"id":478,"title":"Detecting the delimiter in CSVs that lie","sha":"f3e6d0c680c88caf54c1e4b0ae995768","user_id":null,"state":"published","body":"File extensions for data sharing sometimes lie about their contents. \r\n\r\nHere is an algorithm to infer the actual delimiter of a CSV, TSV or any related format:\r\n\r\n- Assume that alpha-numeric characters (A-Z, a-z, 0-9) and the period/full stop (.) are cannot be delimiters.\r\n- Begin with input text _a_.\r\n- Store a short sample of _a_ as _b_, by copying _n_ lines from the input text _a_.\r\n- Rank every character that appears in _b_ by frequency, creating candidate delimiters _c_\r\n- For every candidate delimiter _d_ in _c_, split each line in _b_. If every line has the same number of splits, _d_ is the delimiter.\r\n\r\nAn implementation of this idea is available [on Zenodo](https://zenodo.org/record/1323186) and within the [Python Package Index](https://pypi.org/project/detect-delimiter/).","created_at":"2018-07-29T08:12:03.204Z","updated_at":"2020-10-18T13:32:41.085Z","zenodo_id":1402269,"doi":"https://doi.org/10.5281/zenodo.1402269","subject":null,"tags":["text processing","data_exchange","data-manipulation"],"vote_count":2,"view_count":1,"score":2.1,"deleted":false,"muted":false,"tweeted":true,"attachment_file_name":null,"attachment_content_type":null,"attachment_file_size":null,"attachment_updated_at":null}]