[{"id":478,"title":"Detecting the delimiter in CSVs that lie","sha":"f3e6d0c680c88caf54c1e4b0ae995768","user_id":null,"state":"published","body":"File extensions for data sharing sometimes lie about their contents. \r\n\r\nHere is an algorithm to infer the actual delimiter of a CSV, TSV or any related format:\r\n\r\n- Assume that alpha-numeric characters (A-Z, a-z, 0-9) and the period/full stop (.) are cannot be delimiters.\r\n- Begin with input text _a_.\r\n- Store a short sample of _a_ as _b_, by copying _n_ lines from the input text _a_.\r\n- Rank every character that appears in _b_ by frequency, creating candidate delimiters _c_\r\n- For every candidate delimiter _d_ in _c_, split each line in _b_. If every line has the same number of splits, _d_ is the delimiter.\r\n\r\nAn implementation of this idea is available [on Zenodo](https://zenodo.org/record/1323186) and within the [Python Package Index](https://pypi.org/project/detect-delimiter/).","created_at":"2018-07-29T08:12:03.204Z","updated_at":"2020-10-18T13:32:41.085Z","zenodo_id":1402269,"doi":"https://doi.org/10.5281/zenodo.1402269","subject":null,"tags":["text processing","data_exchange","data-manipulation"],"vote_count":2,"view_count":1,"score":2.1,"deleted":false,"muted":false,"tweeted":true,"attachment_file_name":null,"attachment_content_type":null,"attachment_file_size":null,"attachment_updated_at":null},{"id":888,"title":"Extending run-length encoding to support arbitrary byte streams with hexadecimal notation","sha":"6c83218ba44c176d99322e8d69699079","user_id":null,"state":"published","body":"Conventionally, run-length encoding is restricted to a fixed alphabet. Because the encoded form is often transmitted over human-readable media, this alphabet is usually restricted to printed characters. That makes run-length encoding unsuitable for transmitting arbitrary byte strings.\r\n\r\nTo address this, run-length encoding can be extended by converting the numerical value of each byte to its hexadecimal form. Hexadecimal numbers are often prefixed with `0x`. We can replacing `0` with the actual count, in decimal, to allow for parsing to be fast and unambiguous. \r\n\r\nConsider the input string `Hello`. The following two lines illustrate the different representations in conventional run-length encoding and the extended variant proposed here:\r\n\r\n- `1H     1e     2l    1o`\r\n- `1x48 1x65 2x6c 1x6f`\r\n\r\nAn example implementation in Python:\r\n\r\n```python\r\ndef run_length_encode(text, delimiter=''):\r\n  count = 1\r\n  hex_digits = [hex(ord(x)) for x in text]\r\n  encoded_fragments = []\r\n  \r\n  for (i, current) in enumerate(hex_digits):\r\n    try:\r\n      next = hex_digits[i+1]\r\n    except IndexError:\r\n      next = None\r\n    \r\n    if current != next:\r\n      fragment = str(count) + current[1:]\r\n      encoded_fragments.append(fragment)\r\n      count = 0\r\n    \r\n    count += 1\r\n     \r\n  return delimiter.join(encoded_fragments)\r\n```","created_at":"2021-10-27T22:58:25.126Z","updated_at":"2021-10-28T06:04:26.503Z","zenodo_id":5606102,"doi":"https://doi.org/10.5281/zenodo.5606102","subject":null,"tags":["computer science"],"vote_count":0,"view_count":1,"score":0.1,"deleted":false,"muted":false,"tweeted":true,"attachment_file_name":null,"attachment_content_type":null,"attachment_file_size":null,"attachment_updated_at":null},{"id":981,"title":"Average difference entropy","sha":"98f1536ea2cca26dac907b958904b7a5","user_id":null,"state":"published","body":"Calculating Shannon entropy on octet streams can be confusing, especially with those who lack a background in mathematics. It's also not necessary in many cases. A more intuitive calculation, which is also easier to implement, involves calculating the average difference.\r\n\r\nAn informal description of the algorithm:\r\n\r\n- Find the absolute difference of each octet, as a numeric value, against its successor. Pad with 0 values where the calculation is impossible.\r\n- Exclude the left-most and right-most values.\r\n- Take the average of those differences.\r\n\r\nAn example implementation in Python:\r\n\r\n    def entropy(data, encoding=None):\r\n        if not isinstance(data, bytes):\r\n            data = bytes(data, encoding=encoding)\r\n        data = list(data)\r\n        A = [0] + data\r\n        B = data + [0]\r\n        diff = [abs(B[i] - A[i]) for i in range(len(data))][1:-1]\r\n        return sum(diff) / len(diff)\r\n\r\n","created_at":"2022-06-13T00:55:54.431Z","updated_at":"2022-06-15T06:00:38.223Z","zenodo_id":6644371,"doi":"https://doi.org/10.5281/zenodo.6644371","subject":null,"tags":["entropy","computer science","strings"],"vote_count":0,"view_count":1,"score":0.1,"deleted":false,"muted":false,"tweeted":false,"attachment_file_name":null,"attachment_content_type":null,"attachment_file_size":null,"attachment_updated_at":null}]