Do we need an `MLConstantOperand`?

[a-sully]

The constant() operator is special. Constants are effectively inputs that are known at the time of graph compilation (i.e. build()).

You may ask, why do we need a separate method when we could just pass this constant data as an input()?

Most discussions I've seen so far about constant() (#614 (comment), this thread about the since-abandoned "fill sequence" overload, etc) have been concerned with optimizing performance. There are a number of compile-time optimizations a backend may perform if it knows that some data is constant.

Our experience with CoreML has given us another reason:

The backend requires that a parameter to an operator must be constant

Take conv2d(), for example. It's defined for CoreML here. The bias parameter is defined as:

• bias: const tensor<[C_out],T>

Meanwhile, WebNN allows this to be any MLOperand, as long as it's of the appropriate shape and dtype:

dictionary MLConv2dOptions {
  // ...
  MLOperand bias;
  // ...
};

This appears to be a straightforward plumbing through of DML's interface, which does not require the BiasTensor to be a constant. Neither does the corresponding operator for TFLite. From what I can tell, this seems to be because these frameworks don't have a way to express that some input tensor must be const. The options are either to pass the parameter as the framework's generic representation of a Tensor - which would in practice always(?) be created from a constant() - or to pass the parameter as a 1D array directly. If the parameters may be large (and perhaps unbounded), the former is the more appropriate choice.

To get a sense for whether this is a reasonable hypothesis, I've inspected of all† uses of the affected operators in the WebNN Samples repo repo:

operator.param	Usage in WebNN Samples
batchNormalization.mean	Constant only
batchNormalization.variance	Constant only
batchNormalization.scale	Constant only
batchNormalization.bias	Constant only
conv2d.bias	Constant only
convTranspose2d.bias	Not used
gru.weight	Constant only
gru.recurrentWeight	Constant only
gru.bias	Constant only
gru.recurrentBias	Constant only
instanceNormalization.scale	Constant only††
instanceNormalization.bias	Constant only††
layerNormalization.scale	Not used
layerNormalization.bias	Not used
lstm.weight	Not used
lstm.recurrentWeight	Not used
lstm.bias	Not used
lstm.recurrentBias	Not used
lstm.peepholeWeight	Not used
prelu.slope	Not used

†This list only includes WebNN operators which trivially map to CoreML operators. WebNN operators which need to be in terms of other CoreML operators will be subject to the restrictions of those respective CoreML operators. For example, CoreML doesn't have operators for gruCell or lstmCell, so these operators will need to be implemented in terms of gru and lstm, respectively. These operators will in turn need many of their parameters to be const, as well

††One caller of passes the result of a reshape... but that's only because the sample was written before constant() took an MLOperandDescriptor. The reshape is just assigning dimensions to a constant(). Nowadays we'd just pass the constant() directly

Remarkably, every single instance where one of these params is used in the WebNN Samples, it was created from a constant(). Cool!

Of course, this is not close to a comprehensive list of all models hope to run with WebNN. That being said, if there are no significant known use cases for passing any of these parameters as non-constant tensors - if their non-constness is simply a limitation in the framework and there are no useful reasons to pass non-const tensors - I think there's a reasonable argument that WebNN should require these parameters to be constants. @fwdr could you perhaps provide some more color here? :)

It seems that we have the following options to support each of these operators on CoreML:

1. Require that operator.param must be a constant MLOperand (my tentative preference)
   • Note that it's much easier to relax these restrictions than to try to impose them later
   • This can be done two ways:
     1. Create an MLConstantOperand interface which extends MLOperand, and specify that param takes an MLConstantOperand
     2. Specify that MLOperand has a "kind", as the Chromium implementation already does, and throw a TypeError if not a "constant" kind. This may be confusing to developers
2. Make operator.param a sequence<MLNumber>
   • This doesn't make much sense for anything other than 1-D tensors
3. Decide that this is a quirk of CoreML and fail if operator.param is not a constant only on CoreML
   • Currently this failure happens during build() on Chromium, though we could conceivably make this a synchronous check on the respective builder method, especially if we have defined procedures for querying for backend-specific support (see Allow checking whether operators/types are supported for a backend before creating a graph #463)

Thoughts?

[huningxin]

@a-sully

gru.weight Not used
gru.recurrentWeight Not used
gru.bias Not used
gru.recurrentBias Not used

FYI, these operands (constants only) are used by NSNet2 (noise supression) example: https://github.com/webmachinelearning/webnn-samples/blob/master/nsnet2/nsnet2.js#L51

[a-sully]

My apologies for the error. I've updated the respective rows (they're also all "Constant only")

[a-sully]

Turns out XNNPACK also requires some tensors to be constants - for example prelu.slope. The Chromium implementation currently fails if the slope operand was not created as a constant()

[inexorabletash]

It would be nice to make progress here, as @a-sully notes "it's much easier to relax these restrictions than to try to impose them later"

I'm a fan of the MLConstantOperand approach - it's very straightforward to document (e.g. MDN), specify (just WebIDL, no special case text needed), and implement (bindings code should make it "just work").

I see a 👍 from @huningxin above; @fdwr have you had a chance to think about this?

[a-sully]

Sketched out a PR for MLConstantOperand in #747

[fdwr]

There are a number of compile-time optimizations a backend may perform if it knows that some data is constant.

Indeed, and I like the concept of an operand having a "constness" property, whether it be via a separate class MLConstantOperand, or just an inspectable property on the ordinary MLOperand (e.g. boolean MLOperand::isConstant()) that is set true when returned from constant, and false when generated from another operator. A constness property enables backends to make more informed decisions (not limited to CoreML), but I'm reluctant to require operands like batchNorm to only take constants.

It seems that we have the following options to support each of these operators on CoreML:

Can this be an additional property on the opSupportLimits, besides rank and dataType? My preference is your option 3, except add constant to opSupportLimits, expose an MLOperand::isConstant property, and fail synchronously before build if the backend requires constness. Changing my mind below - I think we should decompose in such cases in the backend.

if their non-constness is simply a limitation in the framework and there are no useful reasons to pass non-const tensors

Or is the reverse true, that mandatory constness is a limitation in a framework, and there are useful reasons to pass dynamic tensors such as for custom weights or as outputs from other operators? DirectML has a means (via DML_GRAPH_NODE_TYPE_CONSTANT) to specify that a tensor is a constant, which can indeed afford optimization opportunities, but it's not required. Accepting dynamic inputs can be useful for cases like convTranspose weights being inflated from a dequantized tensor at runtime (saving significant model space). It sounds like I should scan through all the models on my hard drive to find any cases when dynamic tensors are fed into an operator which CoreML requires to be constant. ⏳

Skimming through the MIL ops, some decisions are unclear to me. Why is MIL conv's weight dynamic while requiring bias to be constant, when there are more interesting preprocessing advantages to knowing that weights are constant? (update - Ningxin points out iOS17 supports dynamic weights, showing that iOS may relax more of these constraints over time) Why is conv's filter weight tensor dynamic, but closely related conv_transpose requires the filter weight to be constant?

Note we're not necessarily blocked for all the ops above when inputs have dynamic values, as multiple operators like normalization.batch_norm can be decomposed into operators that do not require constness, like {add, mul, div, sqrt}. When those input operands are constant, then the built-in CoreML normalization.batch_norm can be used directly, but if non-constant, then we can still achieve dynamic behavior via op emulation. How many other operators can this apply to? conv/convTranspose/gru/lstm seem the toughest ones 🤔.

[inexorabletash]

Can this be an additional property on the opSupportLimits, besides rank and dataType? My preference is your option 3, except add constant to opSupportLimits, expose an MLOperand::isConstant property, and fail synchronously before build if the backend requires constness.

I was initially skeptical of this approach, as it would seem that developers would be likely to develop and test on one platform (e.g. Chromium/Windows/DML) and only discover incompatibilities when users with different systems ran into failures. But if the vast majority of models that we've seen use constant weights anyway then this will be rare in practice.

How smoothly would probing opSupportLimits here and/or responding to a sync build failure integrate into ORT? Is this an expected limitation of EPs already, or would we just rely on errors to propagate up from build?

[philloooo]

Accepting dynamic inputs can be useful for cases like convTranspose weights being inflated from a dequantized tensor at runtime (saving significant model space).

This seems useful, are all of the use cases that need none constant weights derive these weights from actual constants? Or we have use cases that derive none constant weights from inputs?

If all dynamic weights are derived from constants, we can solve this by doing a pre-processing subgraph to generate these constants.

The pre-processing step can be within webnn logic, in such case, we would still need to expose the requirement that these weights need to be constant or are derived from constants. WebNN will detect when a weight is not constant and run a preprocessing subgraph to generate the constants.

The pre-processing step can also be done outside of webnn, in such case we can expose the constness requirement through opSupportLimits, and have ORT (or a javascript library) to implement the logic of preparing the const weights as a preprocessing step. The pros of doing it outside of webnn is:

• Explicit separate the steps of preparing constants and running prediction, the prepared constants can be reused.
• Logic is simpler to implement in js / ORT.

Cons:

• Makes WebNN API harder to use by pushing some logic to user space.
• Requires implementing MLBuffer for constants to make it efficient.