CUE field types: what's available & what's missing

[jpluscplusm]

Right now, CUE makes 3 types of field available. To quote the spec:

A field is a definition if its identifier starts with # or _#.
A field is hidden if its identifier starts with a _.
All other fields are regular.

Here's a table mapping each of those types to some key characteristics:

	Regular fields	Definitions (#)	Hidden fields (_)	Hidden definitions (_#)
Visible in cue export	✅	❌	❌	❌
Accessible across package boundaries	✅	✅	❌	❌
Recursively closed	❌	✅	❌	✅

Table markdown for copy/paste in replies

| | Regular fields | Definitions (`#`)| Hidden fields (`_`) | Hidden definitions (`_#`)|
| ---:|:---:|:---:|:---:|:---:|
| Visible in `cue export` | ✅ | :x: | :x: | :x: |
| Accessible across package boundaries | ✅ | ✅ | :x: | :x: |
| Recursively closed | :x: | ✅ | :x: | ✅ |

This discussion is here to ask:

• are there any additional field types that users would benefit from CUE making available?

And to help answer that:

• are there any distinguishing characteristics that should be added to the above table?

[jpluscplusm]

In https://cuelang.slack.com/archives/CLT4FD7KP/p1694086198707659 I proposed:

I'd like a CUE field type to exist that:

• isn't visible in exported output (like a hidden field isn't)
• is visible across package boundaries (like a definition is)
• doesn't recursively close its contents (like a hidden field doesn't)

The use case for this new type was for it to act as a non-recursively-closed, cross-package "function parameter" type.

This would extend the table in the discussion's opening post as follows:

	Regular fields	Definitions (#)	Hidden fields (_)	Hidden definitions (_#)	New type (%)
Visible in cue export	✅	❌	❌	❌	❌
Accessible across package boundaries	✅	✅	❌	❌	✅
Recursively closed	❌	✅	❌	✅	❌

A nuance which I don't /think/ is completely captured in this table is the effect of combining the new type with a definition. I.e. is Baz closed, in the following example?

#Foo: {
  %Bar: {
    Baz: {
       a: int
       b: bool
     }
   }
}

In other words, does the new type "block" (for want of a better word) recursive closedness implied by its container? I think that the answer needs to be "yes, it stops any 'inherited' closedness, but without recursively opening the fields it contains". If this wasn't the case, then the new type wouldn't be usable as a cross-package function parameter type where the function itself was declared as a definition. Or, rather, the new type would be indistinguishable from a definition when used in this situation, and might as well not exist.

[fgalassi]

// this works

#definition: {
foo: string
...
}

def1: #definition & {
foo: "foo"
bar: "bar"
}

[jpluscplusm]

Thanks for commenting, but I'm not quite sure what point you're making!

Are there useful CUE field types you think are missing ... or something else?

[fgalassi]

My point is that the triple dot at the end of the definition makes it behave exactly like the % type you're talking about, AFAIC

[jpluscplusm]

The ... only permits additional fields inside the scope in which it's specified. It doesn't keep the structure open recursively, as would be the case with the proposed % type.

Here's an example of where that's a problem, building on your example:

exec cue vet file.cue
-- file.cue --
#Def: {
	foo: string
	A: {
		B: {
			C: string
		}
	}
	...
}

def1: #Def & {
	foo: "foo"
	bar: "bar"
	A: B: X: "wibble"
}

Running this with testscript gives the following:

> exec cue vet file.cue
[stderr]
def1.A.B.X: field not allowed:
    ./file.cue:4:6
    ./file.cue:11:7
    ./file.cue:14:8
[exit status 1]
FAIL: /tmp/testscript2646488578/foo.txtar/script.txtar:1: unexpected command failure

[myitcv]

Following some discussion with @rogpeppe last week, I think the requirement/option space here is better expressed as the cross-product between the following:

• Visibility modifier: hidden (_), package (TBC), none
• Closedness modifier: definition (#), none
• Optionality modifier: optional (?), required (!), none

The gap identified in this discussion being that a package visibility modifier is required.

[the-nurk]

i'm a fan of __ (double under) instead of % just for ease of use and lack of special character clutter on the screen. also more pythonic to catch some newbies. also also, trying to relate what is, in my mind, 'resetting recursive closedness' to modulo arithmetic and it's not working.

might i actually recommend another orthogonal specifier that might be worth the extra syntax clutter to be used for '%'? max depth/dimensionality of a struct/list. more related to finite bucket counting imho.

but honestly i'd just like struct.MaxDepth(uint) and list.MaxDims(). this would yield/need sparse structs and maps with more memory control and kind of fit in (i think) as cue's unique way of handling memory via abstracted syntax and semantics. just limit depth of recursion in syntax and ensure your fields aren't packing tons of non-packed structs and cue can likely guarantee much more for the user in terms of expressiveness, security, and performance. however, pretty big codebase requirement i'd imagine.

it would operate more like @ than # or _, so decorator pattern?

#a: { 
  %6
  // for purposes of counting depth, for end user simplicity this 'root' level is depth 1, there aren't 7 allowed levels.
  // the 6th level must be concrete non-collection types. so there are 5 levels left?
  // i'm not against 0-based numbering, it might yield some nice syntax with selectors like maybe `a%3[2]` (yields all 2nd elements at depth 3?
  
  #foo: uint & <=5 @tag(foo)
  
  // # (1 + 2) = +3 <=5 :- #ok
  #bar: [string]: {...} %2
  
  // pattern constraint adds +2 levels (each field is a groups of fields).
  // specifier seemingly adds 2 levels to the root #bar, but the middle two overlap logically so only 3 are added? or am i just confusing myself?
  // the uint 2 means after the `:` delimiter, only two more levels are allowed.
  // same as this :-  
  #bar: {...} %3
  
  // (1 + 1 + 3) = + 5 <= 5 :- #ok 
  baz: [string]: [string]: #bar
  
  // can only accept uints
  #noo: [...] %#foo
}

[rogpeppe]

Apologies, but I don't quite see how that relates to this discussion. Perhaps you could elucidate?

[the-nurk]

sorry. simply suggesting what to do with % instead. i'm still very much for the proposal to have this cross-package recursive umbrella for closedness over a field. i just really think it should be __ not %, because % has a better use in the constraint of depth and its use for field closedness crowds the left hand side of a field name with yet more special character complexity. this will likely cause slightly more confusion that compounds over time once query proposal is in and the LHS becomes more useful to use sugar. i'll use whatever is the decision of course.

[the-nurk]

i currently am trying to union these _# structs that should have open structures below where the openness is largely arbitrary and can't be well defined by a constraint. the ability to say 'everything below this field, don't apply the closedness flag of the parent.

maybe i'm misunderstanding what this new type is supposed to be though.