================================================================
TOP OF LIST:

* windows port and testing
  o useful warnings from: go test miller/src/...
  o parser-experiments out of miller/go entirely -- ?

* regexes
  o finish stats1 -r
  o regex captures ...
    - rename -r
  ! separate DSL/non-DSL string-literal handlers

* remaining verbs:
  o bar format-values
  o nest reshape merge-fields stats2

* remaining functions:
  o new columns-to-arrays and arrays-to-columns for stan format
  o regextract regextract_or_else
  o system

* something getoptish ... -xyz -> -x -y -z ...

! localtime functions
* strptime w/ ...00.Z -> error
* strptime/strftime experiments ...
  * verb sec2gmtdate
    - leaves non-numbers as-is -- ?
    - check sec2gmt as well -- ?
  * functions:
    sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime
    sec2localtime dhms2fsec dhms2sec fsec2dhms fsec2hms hms2fsec hms2sec
    localtime2sec gmt2sec strftime strftime_local strptime strptime_local
  ! strptime:
    strptime("1970-01-01T00:00:00.Z", "%Y-%m-%dT%H:%M:%SZ")
    (error)

! --nojvstack; default --jvstack; code it up ...

* json null

* smart fmtnum
  - poor-person's initial version: lld/llx -> d/x; lf/le/lg -> f/e/g ...
  - fmtnum
  - ofmt
  - no other callsites?!?!?!
  - note hexfmt for non-ints -- especially floats -- ?!?

* smarter head-stop etc https://github.com/johnkerl/miller/issues/291

* port verb merge-fields
* port verb stats2

* in-place processing
* prepipe
  - gzin; bz2in ...

* fixed-width pprint input-scanner

* commenting passes ...

* help-refactor:
  o audit for DEFAULT_FOOs @ 
  o audit for '-z {zzz}'
  o audit for consistent usage style

! reconsider OFMT -- %.6f is not really OK when inputs have far more digits.
* new columns-to-arrays and arrays-to-columns for stan format
* reg-test: diff the *.err files?
* single-line JSON for DKVP/CSV/etc ...

* small:
  o rmd ex1 even simpler -- commarect
  o sg/mktags more explicit comment.
  o justWroteEmptyLine -> pprint too ...
  o nidx @ 10-min
  o text-put @ 10-min
  o xls note (issue #396) @ file-format docs
  o 'string or int' -> 'string, int, or array thereof' throughout

* https://segment.com/blog/allocation-efficiency-in-high-performance-go-services/

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* c/go both:
  o https://brandur.org/logfmt is simply DKVP w/ IFS = space (need dquot though)
  o https://docs.fluentbit.io/manual/pipeline/parsers/ltsv is just DKVP with IFS tab and IPS colon
* do some profiling every so often
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
emitp/emitf:

* case-c-dsl-redirects.sh
  o nesting level is wrong:
    mlr head -n 2 then put -o csv -q 'emitp > stderr, $*' s

* ./reg-test/cases/case-c-dsl-indirect-oosvar-assignments.sh
  _:t:u=1 vs t:u=1 on emitp @*

* check more
  ./reg-test/cases/case-c-dsl-absent-empty.sh
  ./reg-test/cases/case-c-dsl-break-continue.sh
  ./reg-test/cases/case-c-dsl-for-oosvar-loops.sh
  ./reg-test/cases/case-dsl-functional-tests.sh
  ./reg-test/cases/case-c-dsl-indirect-oosvar-assignments.sh
  ./reg-test/cases/case-c-dsl-lashed-emitp-singles.sh
  ./reg-test/cases/case-c-dsl-mapvar-emit.sh
  ./reg-test/cases/case-c-dsl-oosvar-assignments.sh
  u/rr ./reg-test/cases/case-c-dsl-type-predicates.sh

* new emitx punctuation-syntax -- decide x 4
  o to support (from C impl):
    x MD_TOKEN_ALL
    x md_fcn_or_subr_call
    x md_indexed_local_variable
    x md_oosvar_keylist
    k MD_TOKEN_FULL_OOSVAR
    k MD_TOKEN_FULL_SREC
    k md_map_literal
    k md_nonindexed_local_variable
* double-check all examples at
  https://miller.readthedocs.io/en/latest/reference-dsl.html#emit-statements
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

! se-q reads

* stats1: actualize -S and -F -- ?

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! JSON null needs to be passed through as-is ... somehow ... new mlrval type??
  o https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions
  o "null can be added to any value, and returns the other value unchanged."
  o "the length of null is 0"
  o note json types: string, number, boolean, null, object, array.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! smart fmtnum w/ state-machine on chars, and casts, etc -- decouple from raw sprintf
  o use:
    - %
    - 0 - " " + "'"
    - number for min field width
    - "." and precision width
    - s   d i o u x X   e E f F g G
    - "%%"
  o need a parser for validity, and a mapper to Go-builtin formats
  o need to pull out the printed type & use that for casting/ignoring. examples:
    - %d int: ok
    - %d float: int(x) then ok
    - %d string: convert to %s
    - %f int: float(x) then ok
    - %f float: as is
    - %f string: convert to %s
    - %s int: convert to %d
    - %s float: convert to %f
    - %s string: ok
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! smarter head-stop etc https://github.com/johnkerl/miller/issues/291
  o done-reading back-channels in stream.go
    https://gobyexample.com/non-blocking-channel-operations
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

* UDF nexts:
  o more functions (see below)
  !! strlen for UTF-8; wat
  ! regex captures
  o strmatch https://github.com/johnkerl/miller/issues/77#issuecomment-538790927
  o DSL sort function https://github.com/johnkerl/miller/issues/77#issuecomment-321916921

* bash completion script https://github.com/johnkerl/miller/issues/77#issuecomment-308247402
  https://iridakos.com/programming/2018/03/01/bash-programmable-completion-tutorial#:~:text=Bash%20completion%20is%20a%20functionality,key%20while%20typing%20a%20command.

* sliding-window averages into mapper step (C + Go)
* stats1 rank

* double-check rand-seeding
  o all rand invocations should go through the seeder for UT/other determinism

* comment-handling
  - delegator for CSV ...

! quoted NIDX
  - how with whitespace regex -- ?
! quoted DKVP
  - what about csvlite-style -- ? needs a --dkvplite ?
! pprint emit on schema change, not all-at-end.

* mlr -f / -F flags
  o others in mlrcli_parse.go

* more mlrtut
  https://onepointzero.app/posts/miller-command-line-csv-file-processing/

* go 1.16
  o https://golang.org/doc/go1.16
  o io/ioutil deprecation: ioutil.ReadFile -> os.ReadFile

================================================================
CATEGORIES:

! sort functions for online help
! port all I/O formats:
  o TSV/ASV reader
  o XTAB reader
  o TSV/ASV writer
  o markdown writer

* more I/O:
  !!! golang encoding/csv ignores blank lines -- i need those for csvlite schema-change
    detection -- needs an input-stream delegator -- ?
  o --allow-ragged-csv-input|--ragged
  o non-lite DKVP reader/writer
  o comment-filtering-composed io.Reader
  o make a pprint reader w/ width-detect

* widen DSL coverage
  o err-return for array/map get/put if incorrect types ... currently go-void ...
    ! the DSL needs a full, written-down-and-published spell-out of error-eval semantics
  o profile mand.mlr & check for need for idx-assign
    -> most definitely needed
  o multiple-valued return/assign -- ?
    - array destructure at LHS for multi-retval assign (maps too?)

----------------------------------------------------------------
DEFER:
* parser-fu:
  o iterative LALR grok
- jackson notes
- gocc .txt/.go for simple grammars
o find/bookmark/grok rob's lexer slides
o iterate on a parser-generator with JSON config file
no need to bootstrap a parser for the parser-generator language

----------------------------------------------------------------
GOCC UPSTREAMS:

* support "abc" (not just 'a' 'b' 'c') in the lexer part

----------------------------------------------------------------
NITS/NON-IMMEDIATE:

* "Miller: " prefixes on all errors.New
* libify errors.New callsites for DSL/CST
* record-readers are fully in-channel/loop; record-writers are multi with in-channel/loop being
  done by ChannelWriter, which is very small. opportunity to refactor.
* address all manner of xxx and TODO comments
* throughout -- 'map' used too much -- 'mapping/mapper' -> 'transform' or something?
* AST insertions: make a simple NodeFromToken & have all interface{} be *ASTNode, not *token.Token
* mlr --help-for w/ stdout redirect for manpage -- ?
* mlr verb -h -> stdout & exit 0
* cst printer with reflect.TypeOf -- ?
* godoc ...
* mlrdoc false && 4, true || 4 because of short-circuiting requirement
* update whyc.html with efficiency notes from go/README.md
? makefile for build-dsl: if $bnf newer than productionstable.go
* I/O perf delta between C & Go is smaller for CSV, middle for DKVP, large for JSON -- debug
* neaten/error-proof:
  o mlrmapEntry -> own keys/mlrvals -- keep the kcopy/vcopy & be very clear,
    or remove. (keeping pointers allows nil-check which is good.)
  o inrec *types.Mlrmap is good for default no-copy across channels ... needs
    a big red flag though for things like the repeat verb (maybe *only* that one ...)
! clean up the AST API. ish! :^/
* doc: no boolish truthy falsy
* json:
  d thorough UT for json mlrval-parser including various expect-fail error cases
  d doc re no jlistwrap on input if they want get streaming input
  d UT JSON-to-JSON cat-mapping should be identical
  d JSON-like accessor syntax in the grammar: $field[3]["bar"]
  d flatten/unflatten for non-JSON I/O formats -- maybe just double-quoted JSON strings -- ?
    - make a force-single-line writer
    - make a jsonparse DSL function -- ?
  d other formats: use JSON marshaler for collection types, maybe double-quoted
  o research gocc support
  o maybe a case for hand-roll
* ICE on scan of 0xffffffffffffffff
* support 0b0101 et al.? mlrdoc if so
? dsl/ast.go -> parsing/ast.go? then, put new-ast ctor -> parsing package
  o if so, update r.mds
* keep or get rid of $[[[ ]]] ... ?
* 1-up note from mlrval_collections -> doc6
  o substr -- not backward-compatible
  o example of key-stringify to get truly awkish associative-array behavior
  o also $1 vs $[1] -- the former 'sticks' with input-data ordering.
  o moral: miller is a name-indexed tool as its primary purpose. positional
    indexing is supported but it can be confusing.
* mlrdoc: there should have always been a >>> and >>>=, and now there are
! head early-out -- cite the relevant issue & fix the right way ...
* relnotes: label b,i,x vs x,i,b change
* doc note 'array' is now a keyword
* dump is now redundant? print @*
* double-check dump CR-terminators depending on expression type
* good example of wording for why/when to make a breaking release:
  https://webpack.js.org/blog/2020-10-10-webpack-5-release/
* unset, unassign, remove -- too many different names. also assign/put ... maybe stick w/ 2?
* make truncate() UTF8-friendly
* document array-unset as shiftback
! error if UDF has same name as built-in
* huge commenting pass
* profile mlr sort
* go exe 17MB, wut. try to discover. (gocc presumably but verify.)
* fill-down make columns required. also, --all.
* check triple-dash at mlr fill-down-h ; check others
* doc that multi-arg print interleaves spaces; they can strcat to avoid
* clean up unused exitCode arg in sort/put usage.
  o also document pre/post conditions for flag and non-flag usages x all mappers
* need to fix up ;; and start-with-; handling in the grammar so multiple put -f/-e work right :^/
? emit @x or emit x -- should make k/v pairs w/ "x" & value -- ? check against C impl
i emitp/emitf -- note for-loops didn't appear until 4.1.0 & emits are much older (emitp 3.5.0).
  if i were starting clean-slate, i'd have had just a single `emit`.
* doc one-variable for is over keys for maps, elements for arrays
* asserting_{type}: os.Exit(1) -> return nil, err flow?
* test put/filter w/ various combinations of -s/-e/-f
* more text examples in mlr-put doc
* mt_void keep-or-not .......
  o check dispo matrices
  o if keep, need careful MT_VOID at from-string constructor -- ? or not ?
  o comment clearly regardless
* window.mlr, window2.mlr -> doc somewhere
* bitwise_and_dispositions et al should not have _absn for collections -- _erro instead
* fmtnum: auto-replace %l_, %ll_ -> no l/ll for backwards compatibility, and comment accordingly
* mlrdoc for new go-only verbs
* ast-parex separate mlr auxents entrypoint?
* port u/window*.mlr from mlrc to mlrgo (actually, fix mlrgo of course)
* doc: substr in inferred-numeric fields: https://github.com/johnkerl/miller/issues/290.
  o xref to 1-up note.
* line/column caret at parse-error messages -- would require some GOCC refactoring
  in order to get the full DSL string and the line/number info into the same method
* csvlite rd/wr: comment for USV/ASV too. no need for escaping then.
* comment schema-change supported only in csvlite reader, not csv reader
* for-multi: C semantics 'k1: data k2: a:x v:1', sigh ...
* neaten mlr gap -g (default) print
! write out thorough min/max/cmp cases for all orderings by type
* silent zero-pass for-loops on non-collections:
  o intended as a heterogenity feature ...
  o consider a --errors or --strict mode; something
* note about non-determinism for DSL print/dump vs record output-stream now ...
* mlrdoc the ?? and ??? operators
* document --jvstack is now the default; --no-jvstack
* doc about put -S/-F cannot make sense anymore
* put/filter updates:
  o filter statement needn't come last anymore: '$u = 1; NR > 5; $v = 2'
  o put 'NR > 3' and put 'filter NR > 3' are the same.
    more generally, bare-boolean and filter bare-boolean are synonymous -- 'filter' DSL keyword
    is legacy.
  o mlr put and mlr filter are identical except:
    - mlr put accepts zero statements like 'mlr put ""'; mlr filter requires at least one.
    - mlr filter 'filter NR > 2' is a syntax error <---- rethink even that .....
* [[...]] / [[[...]]]:
  o ....
  o ....
  o ....
  o put '$array = [1,2,3,[4,5]]' is a syntax error unfortunately; need '$array = [1,2,3,[4,5] ]'
i https://en.wikipedia.org/wiki/Delimiter#Delimiter_collision
* doc: string index/slice access on RHS
* substr(7,5) -- "" or error?
* write up re auto-deepen logic -- foo[NR] -- should create a map not an array unless already array
  o idea is arrays are created with explicit array literals, then extended as such
  o otherwise preserve map semantics for backward/awkish compatbility
  o arrays for JSON
  o doc clearly that these are different:
    begin {
      foo = []
    }
    foo[NR] = $foo # auto-extends a array with indices 1,2,3,...
    --versus--
    foo[NR] = $foo # auto-creates and auto-extends a map with keys "1","2","3",...
* reorder locations of get/put/remove methods in mlrval/mlrmap
* grep out all error message from reg-test outputs & doc them all & make sure index-searchable at readthedocs
* short 'asserting' functions (absent/error); and/or put --strict or somesuch
* function metadata: auto-sort on mlr -f?
* --x2b @ help-doc .go; etc
? remove flagSet x all -- ? for consistency?
* "mlr" -> os.Args[0] throughout the codebase
* emitx later: 'emit([a,b,c],d,e,f)' for SR-conflict issues

* go doc ...
  o export GOPATH=$(pwd); go doc types.Mlrval
  o go get golang.org/x/tools/cmd/godoc
    go get golang.org/x/tools/cmd/vet
  o brew install go
    brew upgrade go
  o https://golang.org/lib/godoc/analysis/help.html
  o export GOPATH=$(pwd); godoc -http=:6060
    http://localhost:6060/pkg/miller/

* x flagSet. can't be supported everywhere, so don't confuse the user by
  supporting it some places and not others.

* maybe split Context into varying & non-varying -- separate structs entirely

* idea: records as mlrmap -> mlrval?
  o reduce $* copy ...
  o opens the door to some (verb-subset) truly arbitrary-JSON processing ...

* mlr --opprint put $[[1]] = $[[2]]; unset $["a"] ./reg-test/input/abixy
  o squint at pointer-handling
  o output varied after flatten-mods

* join
  > clean up VERBOSE in joiner-files
  > joinBucketKeeper & joinBucket need to be privatized
  > rewrite join-bucket-keeper.go entirely
  > also needs UT per se (not just regression)
* cli-doc --no-auto-flatten and --no-auto-unflatten
* note (fix? doc?) flatten of '$x={}' expands to nothing. not invertible.
* doc: --jvquoteall --jknquoteint no-ops
* back-incompat:
  mlr -n put $vflag '@x=1; dump > stdout, @x'
  mlr -n put $vflag '@x=1; dump > stdout @x'
* parex print regtest -- what about new ast-node types?
* all case-files could use top-notes
* doc-note: 'emit > stdout, @foo' and 'emit @foo' are not the same:
  o the former allows put --oxxx
  o the former is not necessarily in sync with the output record stream
* dev-note on why `int` not `int64` -- processor-arch & those who most need it get it
* document tee -p
* doc auto-flatten/auto-unflatten -- incl narrative from mlrcli_parse.go
* doc6: default flatsep is now "." not ":" in keeping with JSON culture
? allow [[...]] / [[[...]]] at assignment LHS

* why no flagSet:

	// Unlike other transformers, we can't use flagSet here. The syntax of 'mlr
	// sort' is it needs to take things like 'mlr sort -f a -n b -n c', i.e.
	// first sort lexically on field a, then numerically on field b, then
	// lexically on field c. The flagSet API would let the '-f c' clobber the
	// '-f a', while we want both.

	// Unlike other transformers, we can't use flagSet here. The syntax of 'mlr put'
	// and 'mlr filter' is they need to be able to take -f and/or -e more than
	// once, and Go flags can't handle that.

* doc re multi-load: can't '$x >' and '3' in separate -f anymore. no worries.
* doc re two-arg flatten
* doc md5/sha1/sha256/sha512

* readeropts/writeropts/readerwriteropts -> cliutil funcs
  o then put into join.go, put.go, & repl
* surface the REPL at:
  o go/README.md
  o man mlr
  o readthedocs (once mlr6 is there ...)
* mlr inp parse error failstring retback?
* https://blog.golang.org/go1.13-errors
* split REPL lines on ';' -- ?
* tilde-expand for REPL load/open: if '~' is at the start of the string, run it though 'sh -c echo'
* doc shift/unshift as using [2:] and append
? ctx invars -> ptr w/ cmt
? string/array slices on assignment LHS -- ?
* beyond:
  o support 'x[1]["a"]' etc notation in various verbs?
  o sort within nested data structures?
  o array-sort, map-key sort, map-value sort in the DSL?
  o closures for sorting and more -- ?!?
  o or maybe just use UDFs ...
* optimize MlrvalLessThanForSort
  o mlr --cpuprofile cpu.pprof --from ~tmp/big sort -f a -nr x then nothing
  o GOGC=1000 mlr --cpuprofile cpu.pprof --from /Users/kerl/tmp/huge sort -f a -nr x then nothing
  o wc -l ~/tmp/big
    1000000 /Users/kerl/tmp/big
  o wc -l ~/tmp/huge
    10000000 /Users/kerl/tmp/huge
* optimize MlrvalGetMeanEB et al.
* data-copy reduction wup:
  o literal-type nodes -- now zero-copy
  x modify Evaluate to return pointer -- too much copying
  o wup for it was the binary-operator node, w/ the '*', that broke w/ no-output-copy & fibo UT
  o bonus: return MlrvalSqrt(MlrvalDivide(input1, input2))
  o type-gated mv -- should use passed-in storage slot -- ?
  o nice narrative write-up w/ the C stack-allocator problem, Go non-solution,
    profilng methods, GC readings/findings, before-and-after CST data structures,
    final perf results.
  o next round of data-copy reduction:
    - $z = min($x, $y) -- needs to return pointer to x or y
    o $z = $x + $y -- needs to have space for sum, and return pointer to it
    o therefore type BinaryFunc func(input1, input2 *Mlrval) *types.Mlrval
      > have the function z-allocate outputs when needed
      > the outputs must be on the stack, not statically allocated, to make them re-entrant
        and OK for recursive functions
      > var output types.Mlrval w/ field-setters, rather than return &Mlrval{... all of them ...}
    - then IEvaluable: Evaluate(state *runtime.State) *types.Mlrval
    - invalidate CopyFrom
    - check for under/over copy at Assign
    - global *ERROR / *ABSENT / etc
* for i, e in range c optimization -- always *copies* e
  o try and benchmark/compare ...
  o lots of array-of-pointer stuff, this is totally fine
  o take care w/ copying (non-pointer) mlrvals though

* more copy-on-retain for concurrent pointer-mods
  o make a thorough audit, and warn everywhere
  o either do copy for all retainers, or treat inrecs as immutable ...
  o 'this.recordsAndContexts.PushBack(inrecAndContext)' idiom needs copy everywhere ...

* consider -w/-W to stderr not stdout -- ?
* doc6 re warnings
* -W to mlr main also -- so it can be used with .mlrrc
* push/pop/shift/unshift functions
