Layers

struct Conv2d{T<:Union{Real, JuMP.VariableRef, JuMP.AffExpr}, U<:Union{Real, JuMP.VariableRef, JuMP.AffExpr}, V<:Integer} <: Layer

Represents 2-D convolution operation.

p(x) is shorthand for conv2d(x, p) when p is an instance of Conv2d.

Dimension conventions (TensorFlow-style NHWC/HWIO):

This follows the conventions of tf.nn.conv2d:

• Input: (batch, height, width, in_channels) — NHWC format, matching TensorFlow's default data_format="NHWC"
• Filter: (filter_height, filter_width, in_channels, out_channels) — HWIO format, matching TensorFlow's filter shape for tf.nn.conv2d
• Output: (batch, out_height, out_width, out_channels) — NHWC format

To convert from PyTorch (OIHW filters, NCHW inputs), use convert_conv_filter_from_pytorch and convert_images_from_pytorch.

To convert from Flux.jl (WHIO filters, WHCN inputs), use convert_conv_filter_from_flux and convert_images_from_flux.

Fields:

• filter

• bias

• stride

• padding

Conv2d(filter)

Convenience function to create a Conv2d struct with the specified filter and zero bias.

struct Flatten{T<:Integer} <: Layer

Represents a flattening operation.

p(x) is shorthand for permute_and_flatten(x, p.perm) when p is an instance of Flatten.

Fields:

• n_dim

• perm

struct Linear{T<:Real, U<:Real} <: Layer

Represents matrix multiplication.

p(x) is shorthand for matmul(x, p) when p is an instance of Linear.

Dimension convention:

• Weight matrix: (in_features, out_features) — note: transposed during forward pass
• Bias: (out_features,)
• Computation: transpose(matrix) * x + bias

This matches the TensorFlow/Keras Dense layer convention. To convert from PyTorch's nn.Linear (which stores weights as (out_features, in_features)), use convert_linear_weights_from_pytorch.

Fields:

• matrix

• bias

struct MaskedReLU{T<:Real} <: Layer

Represents a masked ReLU activation, with mask controlling how the ReLU is applied to each output.

p(x) is shorthand for masked_relu(x, p.mask) when p is an instance of MaskedReLU.

Fields:

• mask

• tightening_algorithm

struct Normalize <: Layer

Represents a Normalization operation.

MaxPool(strides)

Convenience function to create a Pool struct for max-pooling.

struct Pool{N} <: Layer

Represents a pooling operation.

p(x) is shorthand for pool(x, p) when p is an instance of Pool.

Stride convention:

Strides are specified as an N-tuple matching the input dimensions. For 4D input in NHWC format: (1, stride_height, stride_width, 1) (batch and channel strides are typically 1).

Fields:

• strides

• pooling_function

struct ReLU <: Layer

Represents a ReLU operation.

p(x) is shorthand for relu(x) when p is an instance of ReLU.

struct SkipBlock <: Layer

A layer that implements a skip connection pattern, commonly used in residual networks (ResNets).

A SkipBlock takes multiple input tensors and applies a corresponding transformation layer to each input. The outputs of these transformations are then combined through element-wise addition.

When used within a SkipSequential, a SkipBlock processes inputs from the most recent layers in the network. If the block has n layers, it will take the outputs from the last n layers as input, in order. For example, with a 2-layer SkipBlock:

• The second layer processes the output from the immediately preceding layer
• The first layer processes the output from two layers back

Typically used in conjunction with SkipSequential to create networks with skip connections, where the outputs from earlier layers can bypass intermediate layers and be combined with later layer outputs.

Example:

skip_block = SkipBlock([
    Linear(input_size_1, output_size),  # Transform from one path
    Linear(input_size_2, output_size)   # Transform from another path
])

Fields:

• layers

struct Zero <: Layer

Always outputs exactly zero.

conv2d(input, params)

Computes the result of convolving input with the filter and bias stored in params.

Mirrors tf.nn.conv2d from TensorFlow, with strides = [1, params.stride, params.stride, 1].

Dimension conventions:

• Input: (batch, height, width, in_channels) — NHWC format
• Filter: (filter_height, filter_width, in_channels, out_channels) — HWIO format
• Output: (batch, out_height, out_width, out_channels) — NHWC format

Padding:

• SamePadding(): TensorFlow-style SAME padding is used, so output spatial size is (ceil(input_height / stride), ceil(input_width / stride)).
• ValidPadding(): No padding is added.
• Fixed padding, specified as:
  • A single integer, interpreted as padding for both axes
  • A tuple of two integers, interpreted as (y_padding, x_padding)
  • A tuple of four integers, interpreted as (top, bottom, left, right)

Throws

• AssertionError if input and filter are not compatible.

Permute dimensions of array in specified order, then flattens the array.

matmul(x, params)

Computes the result of pre-multiplying x by the transpose of params.matrix and adding params.bias.

matmul(x, params)

Computes the result of pre-multiplying x by the transpose of params.matrix and adding params.bias. We write the computation out by hand when working with JuMPLinearType so that we are able to simplify the output as the computation is carried out.

getoutputsize(input_array, strides)

For pooling operations on an array, returns the expected size of the output array.

getpoolview(input_array, strides, output_index)

For pooling operations on an array, returns a view of the parent array corresponding to the output_index in the output array.

getsliceindex(input_array_size, stride, output_index)

For pooling operations on an array where a given element in the output array corresponds to equal-sized blocks in the input array, returns (for a given dimension) the index range in the input array corresponding to a particular index output_index in the output array.

Returns an empty array if the output_index does not correspond to any input indices.

Arguments

• stride::Integer: the size of the operating blocks along the active dimension.

pool(input, params)

Computes the result of applying the pooling function params.pooling_function to non-overlapping cells of input with sizes specified in params.strides.

poolmap(f, input_array, strides)

Returns output from applying f to subarrays of input_array, with the windows determined by the strides.