xan: vanhemmat versiot ja versiohistoria

Features
- Support for .mtx (Matrix Market) files.
- Adding geometric_mean & harmonic_mean aggregation functions.
Fixes
- Fixing xan rename with non-comma delimiters.
- Fixing xan from -f=parquet not converting timestamp columns.
- Fixing open-ended moonblade slicing.
- Fixing xan parallel (cat|map) not working with -R/--run.
- Fixing xan parallel -R/--run not spawning checker thread.
- Fixing xan cat rows (-I|-U) -S/--source-column.
Performance
- Improving performance of xan freq & xan p freq.
Quality of Life
- Better moonblade error messages when values are not yet materialized.
- Better xan count -H/--human-readable.

Breaking
- Renaming xan from --path to --root.
- Renaming xan separate -T/--txt to -L/--lines.
Features
- Adding xan count -c/--check-alignment & -H/--human-readable.
- Adding xan sort -e -z/--compress.
- Adding xan sample -g -S/--sorted.
- Adding xan from -f=(json|ndjson) --model.
- Adding xan cat rows -I/--intersection & -U/--union.
- Adding xan top -g -S/--sorted.
- Adding the sort, dedup, flatten & flat_map moonblade function.
- Adding support for list & map columns when using xan from -f=parquet.
Fixes
- Fixing text wrapping across tool, especially xan flatten -w & xan flatten -F.
- Fixing xan select -e & xan map plural clause flattening given list.
- Fixing xan from -f=parquet not emitting correct column names for list & map columns.
Performance
- Introducing a fast path for xan sort -e when input fits in a single chunk.
- Faster xan from -f=ndjson & -f=parquet.
- Amortizing xan sample allocations.
- Improving performance of xan freq & xan p freq.
Quality of Life
- Better record size estimation for xan sort -e.
- Better xan flatten -c when string width cannot be computed correctly (because of emojis typically).

Breaking
- Bumping MSRV to 1.85.0 and edition 2024.
- Overhauling how xan scrape takes its inputs. It now targets HTML files on disk by default now.
- xan plot --density-scale now defaults to log.
- xan freq -X/--approx-algo & xan p freq -X/--approx-algo now default to heavy-keeper.
- Dropping xan eval -S/--serialize. The default behavior of the command is now to output the serialized value.
Features
- Adding xan search -x/--pattern-file.
- Adding xan cat --glob, xan merge --glob, xan parallel --glob.
- Adding the hostname moonblade function.
- Adding xan scrape --paths, --path-column, --docs, --docs-column, -D/--stdin-doc & --glob.
- Adding xan from -f=(json|ndjson) --path <path>.
- Adding xan from -f toml & -f raw.
- Adding xan spark --hide-all & --repeat-x-axis.
Fixes
- Fixing moonblade string with bstring equality.
- Fixing percentages shown by xan spark -P.
- Fixing xan complete correctness in presence of duplicate values.
- Fixing xan spark -c discretization & legend.
Performance
- Improving performance of xan scrape.
- Improving performance of xan from -f ndjson.
Quality of Life
- Better legends & axis for xan spark.
- Highlighting empty labels when using xan heatmap.

Breaking
- Stopping to serialize moonblade lists either as joined by some separator or JSON. This was awkard, error-prone & potentially lossy. Use the join function manually to format output when required.
- As per previous point, dropping xan scrape --sep.
- Dropping implicit unary function calls in moonblade pipelines. This feature was not well-known, confusing (an indentifier, could be understood as a call in a pipeline, only if not in first position...), and mostly useless now that moonblade has had a proper dot operator.
- xan plot -A/--aggregate does not take an expression anymore but has an automatic selection of two modes: sum and mean. It should also be faster.
- Renaming the index function as row_index for clarity.
- xan agg -C/--along-columns & -M/--along-matrix & xan groupby -C/--along-columns & -M/--along-matrix will not map current column index to the result of the index() function. The col_index() can be now used instead for this very purpose.
- xan window -g/--groupby does not require the file to be sorted anymore. This means using -g/--groupby will now require the whole file to be buffered into memory by the command. The old behavior can still be used through the -S/--sorted flag, thus aligning the xan window command with the rest of the tool.
- row_index will now error if the expression has no concept of row index, instead of returning nothing.
- xan parallel -z/--compress now take the desired compression (either gzip or zstd).
- Retiring the xan grep command in favor of xan search -Z/--fast-parser.
- xan tokenize --keep short flag becomes -k instead of -K to harmonize with other commands.
- Retiring the xan flatmap command in favor of xan explode -e.
- Retiring the xan fuzzy-join command in favor of a consolidated xan join command.
- Changing xan from -f txt -c <name> default to line instead of value.
- Renaming xan join -L/--prefix-left & -R/--prefix-right short flags to -l & -r respectively to avoid colliding with the added -R/--reverse flag that can be used for merge joins.
- Dropping xan plot -B/--bars. It never worked very well and its use-case will be redirected to xan spark.
- Changing xan heatmap --width short flag from -w to -W so that adding a -H/--height flag remain consistent and avoids clashing with -h/--help.
- Dropping xan heatmap --show-gradients in favor of xan help gradients.
- Renaming xan search -A/--all flag to --every-column for clarity and avoid clash with -A/--after-context.
- Dropping xan sort -U/--unstable. It was never used and the performance boost it supposedly provides cannot be observed.
Features
- Adding xan parallel --dont-chunk.
- Adding nullary col, col_index & header variants, to work with expression applied in series to multiple columns at once.
- Adding prev_col & next_col functions.
- Adding xan (search|filter) -B/--before-context & -A/--after-context.
- Adding xan window -O/--overwrite.
- Adding xan map -C/--along-columns.
- Adding xan window -C/--along-columns.
- Adding xan cat rows --raw, -P/--preprocess & -H/--shell-preprocess.
- Improving xan select DSL star selectors. You can now do stuff like vec_*_count, *[1], vec_*[1] etc.
- xan p -H/--shell-preprocess now works on Windows.
- Adding native zsh completions (@apcamargo).
- Adding xan dedup --u32.
- Adding xan explode -e/--evaluate, -f/--evaluate-file, --pad & -k/--keep.
- xan to npy is now able to stream.
- Adding xan parallel top & xan top -p/--parallel, -t/--threads.
- Adding xan network edgelist --range.
- Adding xan network nodelist.
- Adding the xan run command.
- Adding xan view --name.
- Adding xan join -S/--sorted, -R/--reverse & -N/--numeric.
- Adding xan parallel --run & xan cat rows --run.
- Adding xan to md -l/--limit.
- Adding the xan spark command.
- Adding xan stats -R/--report, --color, --cols, --sep.
- Adding xan (freq|p freq) -X/--approx-algo.
- Adding xan plot -D/--density-gradient, --density-scale, --hide-legend, --hide-x-axis, --hide-y-axis, --hide-all & -Q/--square.
- xan separate will now a

Fixes
- Fixing xan sort --check -n & xan dedup --check -n printed report.
- Fixing xan parallel cat -nS.
- Fixing CSV parsing rare edge case panics.
- Fixing commands relying on zero-copy CSV parsing for performance.
Performance
- Improving performance of xan flatten, xan view, xan to & xan network.
- Faster xan network -f nodelist in some cases.

The temporal update.
Breaking
- xan select -n will not error anymore on empty inputs and, generally, empty files should not trigger selection errors when using commands with -n/--no-headers.
- xan heatmap -C/--cram becomes a flag accepting either auto, always or never.
- Dropping -C short flag for xan sort --cells (it could be confused with --columns or --check).
- Completely overhauled how datetimes work in moonblade.
- xan separate will not trim splitted values with some modes by default anymore.
- Dropping xan network --stats in favor of -f stats.
- -D becomes short flag for xan network --degrees instead of --disjoint-keys.
- xan separate --capture-groups is dropped in favor of -c/--captures & -C/--all-captures.
- Renaming xan search --breakdown shortflag to -b to allow for future -B/--before-context.
Features
- Adding xan matrix count & xan matrix adj.
- Adding front_coding window function.
- Timestamp support with xan plot -LT.
- Adding xan rename -n/--no-headers support for -p/--prefix & -x/--suffix.
- Adding xan from -f parquet (requires the parquet feature).
- Adding xan to latex.
- Adding xan top -L/--lexicographic.
- Adding xan heatmap flags: -w/--width, -F/--fill, -a/--align, -U/--unit, -Z/--show-normalized, -A/--ascii, -l/--label & -v/--values.
- Adding new gradients to xan heatmap.
- Adding range & repeat moonblade functions.
- Adding xan sort --columns.
- Adding xan view -T/--tee.
- Adding now, fractional_days, to_timezone, to_local_timezone, with_timezone, with_local_timezone, without_timezone, to_timestamp, to_timestamp_ms, from_timestamp, from_timestamp_ms, span, date & time moonblade functions.
- Better type inference with xan stats, and the type & types aggregation functions, now including more types for temporal values (zoned_datetime, datetime, date & time).
- Adding xan input -T/--tolerant.
- Adding xan separate --trim.
- Adding xan grep -B/--before-context & -A/--after-context.
- Adding xan network -f=components, -S/--simple, --union-find, --minify & --sample-size <n>.
- Adding xan plot --timezone.
- Adding xan hist --log shorthand flag for --scale=log.
- Adding log_dist sparkline column to xan stats -q output.
- Adding dist & log_dist aggregation functions.
- Adding xan search -L/--levenshtein <k> & -D/--damerau-levenshtein <k>.
Fixes
- Fixing xan separate automatic column prefix extraction.
- Fixing xan heatmap -n.
- Fixing xan heatmap --repeat-headers --cram always not repeating x-axis legend.
- Fixing correctness of xan plot -T and increase resolution to microseconds.
- Fixing moonblade column-related functions returning incorrect results wrt -n/--no-headers.
- xan search should now properly error when handling invalid utf-8 in relevant modes.
- Fixing xan search -iR & xan search -i --replacement-column.
Performance
- Improving performance of xan complete, xan top, xan plot -T & xan hist.
- Improving overall performance of xan network.
- Slightly optimizing xan vocab by allowing needless heap allocation & indirection.
- Improving performance and memory usage of xan separate.
Quality of Life
- Adding proper help to xan heatmap.

Features
- Adding xan bisect.
- Adding xan flatten -N/--non-empty.
- Adding the soundex, refined_soundex & phonogram moonblade functions for phonetic encoding.
Fixes
- Fixing xan to (md|html) --no-headers.
- Fixing xan plot -R/--regression-line.
Quality of Life
- Adding xan to markdown as an alias for xan to md.
- xan flatten & xan view will stop masquerading trimmed empty cells as empty.

Breaking
- Changing how xan separate generates default column names.
- xan from -f=(json|ndjson|jsonl) will now emit column in input order by default.
- Changing xan to -B/--buffer-size to --sample-size to harmonize flag names with xan from.
Features
- Adding the xan complete command.
- Adding an optional unit to ceil, floor, round & trunc moonblade function. E.g. floor to nearest decade: floor(year, 10).
- Adding basename & dirname moonblade functions.
- Adding parse_py_literal moonblade functions. Useful to deal with files dubiously serialized using pandas.
- Adding xan view --repeat-headers=(auto|always|never).
- Adding xan view --reveal-whitespace=(auto|always|never).
- Adding --color support to XAN_VIEW_ARGS.
- Adding xan from -f json --sample-size -1 to sample the whole file.
- Adding xan from -f json --single-object.
- Adding xan from --sort-keys.
- Adding xan to (json|ndjson|jsonl) --sample-size -1 to sample the whole file.
- Adding xan to (json|ndjson|jsonl) --strings flag.
- Adding xan separate --prefix.
- Adding xan heatmap -C short flag for --cram.
- Adding xan heatmap --repeat-headers.
- Adding rank, cume_dist, percent_rank and ntile window functions.
- Adding xan help --color.
Fixes
- Fixing xan select -ne incorrectly emitting headers.
Quality of Life
- xan view -p will not print bottom header anymore by default.
- xan view will not reveal problematic whitespace if output is not colored anymore, by default.
- Better xan hist error messages and help.
- Testing more file name variants when searching for a .gzi index.

Fixes
- Fixing xan freq --groupby incorrectly unescaping group cells.
- Fixing help related to xan pivot & xan unpivot.
- Upgrading simd-csv to get safety fixes.

The SIMD update.
Breaking
- Bumping MSRV to 1.83.0.
- Dropping xan plot -Y/--add-series. It is now possible to select multiple columns as <y> in xan plot <x> <y> instead.
- Dropping the -C/--force-colors flag in flatten, heatmap, hist, plot and view in favor of the more standardized and flexible --color=(auto|never|always) flag.
- xan join will now automatically drop joined columns from one the files when it is obviously safe to do so.
- xan behead & xan rename do not normalize the output anymore to be as fast as possible.
- The new SIMD CSV parser might not deal with CSV irregular cases the same way rust-csv did. In any case, xan input will still continue to use rust-csv.
- xan slice -B/--byte-offset & xan slice -A/--accumulate are now mutually exclusive.
- xan input has been overhauled.
- Dropping xan count --sample-size.
- Overhauling xan fixlengths to accept streams by shifting default from double-pass read to buffering the whole stream into memory.
- xan plot --x-scale log & --y-scale log are now natural log. Use log10 for the base10 log as before.
- Dropping xan reverse -m/--in-memory flag. Behavior is now automatically detected.
- Dropping xan shuffle -m/--in-memory flag. Loading the file into memory is now the default. The xan shuffle -e/--external flag has been added if
you want the old default behavior.
- xan bins now outputs <empty> values instead of <nulls>.
- Overhauling xan bins. The default is now to find nice boundaries for the bins. Use -e/--exact to revert to the old behavior. The default number of bins is now 10, and won't use Freedman-Diaconis rule by default. A -H/--heuristic flag has been added if you want to automatically select a suitable number of bins.
Features
- Adding xan flatten -F/--flatter.
- xan pivot can now target multiple columns.
- Adding the xan grep command for fast but coarse filtering.
- Adding xan search -f/--flag.
- Adding xan map -F/--filter.
- xan search -B/--breakdown now consolidates the results when multiple patterns have a same name.
- Adding xan flatten --row-separator.
- Adding xan flatten --csv.
- Adding xan headers --color.
- Adding the xan join <columns> <input1> <input2> arity as a convenience when joined column names are the same in both inputs.
- Adding xan join -D/--drop-key=(none|both|left|right).
- Adding xan fuzzy-join -D/--drop-key=(none|both|left|right).
- Adding xan plot -A/--aggregate.
- Adding support for plural selection clauses in both xan select -e & xan map e.g. xan map 'full_name.split(" ") as (first_name, last_name).
- Adding xan search -P/--add-pattern.
- Adding xan groupby -M/--along-matrix.
- Adding xan groupby -T/--total.
- Adding support for .ndjson & .jsonl files. Those are considered as headless TSV files with null byte quoting so you can easily use them with xan commands.
- Adding out-of-the-box support for .vcf, .sam, .bed, .gtf & .gff2 files.
- Adding a xan cat cols alias to xan cat columns.
- Adding zstd support.
- Adding earliest & latest moonblade functions.
- Adding xan dedup -f/--flag.
- Adding -k short flag for xan dedup --keep-duplicates, and -C short flag for xan dedup --choose.
- Adding xan fixlengths -H/--trust-header.
- Adding xan separate.
- Adding full log scale support to xan plot.
- Adding xan hist --scale.
- xan window is now able to run total aggregations.
- Adding thousands_sep, comma and significance kwargs to numfmt moonblade function.
Fixes
- Fixing xan dedup --check bug where the first record was ignored.
- Fixing xan hist -D when a same date is found multiple times.
- Fixing xan from -f xls datetime conversion.
- Fixing xan flatten & xan view when column names contain line breaks.
- Fixing invalid argument parsing error being printed to stdout instead of stderr.
- Fixing xan progress SIGINT corrupting output.
- Fixing xan enum -A/--accumulate.
- Fixing xan from -f tar when tarball archive is not gzipped.
- Fixing min & max moonblade function when passing a list of numbers.
- Fixing xan flatten -H edge cases.
- Fixing command

Breaking
- xan partition now normalizes filenames to lowercase to correctly deal with case-insensitive filesystems. xan partition also gets a related -C/--case-sensitive flag.
Features
- Adding all and any moonblade higher-order functions.
- Allowing moonblade printf function to be called with lists.
- Adding -f/--evaluate-file flag to map, filter, flatmap & transform commands.
- Adding xan map -O/--overwrite.
Fixes
- Fixing xan top -T/--ties edge case.
- Fixing broken pipe panics for some commands.
- Dropping remnant dbg! macro when reading files in reverse.
Performance
- Using jemallocator for musl builds.
Quality of Life
- Better moonblade printf function error messages.