Hints/attributes for properties and types

[KOLANICH]

The idea is to provide compiler and runtime with some info about the purpose of the member. This info can be used either to generate more usable and/or more performant code, or it can be passed to runtime and used by runtime somehow, or can be used by other tools.

seq:
  - id: signature
    hint:
      - signature # informs the implementation that it can scan the blob for the bytes in contents. Informs the compiler to generate offset for this field and to emit a warning if there is no 'contents' attribute
    contents: "signature"
  - id: reserved
    hint:
      - format_version # valid only for anything that can be considered a version: integers, and types of anything that can be considered a version. The components of the type must be used within `if: clauses`. If they are not, the compiler emits a warning. The conditions are used in automatic format changelog generation.
     - format_flavour # like `format version`, but can also be `bytes` and `strings`. The set of flavours is unordered.
      - reserved # informs that this field is not currently meaningful and should not be accessed and is reserved for future extension. Compiler can create the metainformation for languages that support it (for C# [Obsolete], for C++ and C2x: [[deprecated]] )
      - obsolete # informs that this field should not be accessed and us not meaningful in the current and possible versions of the format. Compiler can create the metainformation for languages that support it (for C# [Obsolete], for C++ and C2x: [[deprecated]] )
      - custom:ololo # custom attribute
    type: u4
  - id: port0
    hint: volatile # reread/recalculate on every access. Compiler creates a property.
    type: u4

[GreyCat]

Thanks for the idea! Probably we need to write down which of these "hints" should affect what and see which ones are compatible with which, and then decide on how to implement these. May be most of them should be promoted to full-size members of AttrSpec, i.e. stuff like reserved: true, signature: true, or something like that.

[KOLANICH]

I don't think they should be promoted, they are like just a list of traits each adding an additional optional implementation-defined effect to the property. They don't affect parsing, the old-style code (i mean the one written in get_blob -> parse_into_memory -> use_parsed_objects style) should not be broken by adding/removing them, but performance or other things not breaking compilation can be affected. For example signature hint will be useful only if runtime (compiler can also implement this) has this feature. For example ks-powered binwalk would be able look for fields in definition having signature hint, add signatures in its internal data structures, scan files, when a signature from ks-file is found it seeks back by the offset of that field computed by KSC (that way getting the beginning of the structure, some formats have signature not in the very beginning) and starts parsing. Or KSC can precompile an Aho-Corasick finite automata for matching for languages where it makes sense and embed it into the generated code.

[arekbulski]

I agree with Mikhail on this one. Each hint should be discussed as separate feature and I think most of them would be good candidates for AttributeSpec features. And we already have features that could be considered compiler hints, like -construct-render and some others.

The volatile hint: I propose naming it nocache as volatile is sometimes used as atomic access memory (its a C# keyword, see docs). I also propose making it AttributeSpec, or InstanceSpec, whatever.

I also propose that we add a compiler option so that all AttributeSpec tags, not just the hints part of it, are exposed to the runtime. In that sense, all AttributeSpec fields would be considered compiler hints.

[KOLANICH]

The volatile hint: I propose naming it nocache as volatile is sometimes used as atomic access memory

It was inspired with C++ volatile. I mean if we write a driver using memory for transferring data from and to device we mark variable as volatile to make the compiler not optimize reading/writing out.

And we already have features that could be considered compiler hints, like -construct-render and some others.

What is it? What does it do?

Hints are like meaningful metadata for programming languages like C++ and C#, they don't affect API, they don't affect parsing in ksy language, but they do side effect (in the sense that they can be safely removed and the format must still be parsed correctly) outside of ksy language. It's convenient to have them all in a separate object.

[arekbulski]

What is it? What does it do?

-construct-render is used to allow alternative outputs for the Construct compiler target. There are several types that could be compiled into more than just one Construct type, depending on writers intentions. For example, size: N could be Bytes(N) but it could be Padding(N) and KSY frankly does not make the distinction.

[zolo]

I have a similar need, namely to express that I want std::array generated instead of std::string for (certain) fixed-size buffer items.

[jgerrish]

First, thanks for this wonderful project.

Compiler hints would be useful to me too. I'm working on a language and runtime implementation, and initializing variable length vectors to a certain size, knowing they'll probably be bigger or smaller for a given binary.

People writing KSY files probably have a lot of experience with common sizes for their formats. Adding a hint on what size a vector is likely to be may reduce the number of allocations.

It's a low priority request, but another vote. Maybe the feature is already implemented.

Thanks again for your time.

[KOLANICH]

I guess it'd require the modification of the proposed syntax to allow arguments. I also guess the size should be not in bytes, but in items.

hints:
  - `alloc-count(expr_here)`