Summary of Input Data Processing
Our model uses a dual hourglass architecture. The output of the first hourglass block is summed with the pre-processed data (via skip connection), and this sum is then fed into the second block.
The convolution operation restores the data to its original form (a three-channel image). The function of the hourglass blocks is to identify features at various levels of abstraction. Starting from high resolution, pixels and their neighbors are convolved. The size of the convolution remains constant, but the resolution decreases, allowing the convolution to cover a larger portion of the image. This process is repeated until reaching the minimum resolution. Then, the reverse process begins, where each pixel is given new neighbors based on its value. The resolution is gradually increased back to the original.
conv_args = {
"kernel_initializer": tf.random_normal_initializer(0., 0.02),
#"kernel_regularizer": tf.keras.regularizers.L2(1e-5),
"padding": "same",
}
def block(ft=2, ks=3):
def her(in_x):
temp = tf.keras.layers.Conv2D(filters=ft,
kernel_size=ks,
strides=1,
**conv_args)(in_x)
temp = tf.keras.layers.BatchNormalization()(temp)
temp = tf.keras.layers.ReLU()(temp)
return temp
return her
def hourglass(fr=1):
def her(in_x):
k = 2
store = {}
x = tf.keras.layers.Conv2D(filters=fr*k,
kernel_size=3,
strides=2,
**conv_args)(in_x)
x = tf.keras.layers.BatchNormalization()(x)
x = block(fr*k)(x)
for i in range(3,11-1):
k = i
x = tf.keras.layers.Conv2D(filters=fr*k,
kernel_size=3,
strides=2,
**conv_args)(x)
x = tf.keras.layers.BatchNormalization()(x)
x = block(fr*k)(x)
store[i] = x
x = block(fr*k)(x)
temp = x
x = tf.math.add(store[9], x)
x = block(fr*k)(x)
keypoint = x
x = tf.math.add(temp, x)
del temp
x = block(fr*k)(x)
for i in range(9-1,1,-1):
k = i
x = tf.math.add(x, store.pop(i+1))
x = tf.keras.layers.Conv2DTranspose(filters=fr*k,
kernel_size=3,
strides=2,
**conv_args)(x)
x = tf.keras.layers.BatchNormalization()(x)
x = block(fr*k)(x)
keypoint = tf.keras.layers.Conv2DTranspose(filters=fr*k,
kernel_size=2,
strides=2,
**conv_args)(keypoint)
keypoint = tf.keras.layers.BatchNormalization()(keypoint)
keypoint = tf.math.add(keypoint, x)
x = tf.keras.layers.Conv2DTranspose(filters=fr*1,
kernel_size=3,
strides=2,
**conv_args)(keypoint)
x = tf.keras.layers.BatchNormalization()(x)
x = block(fr*1)(x)
return x
return her
def Generator():
input = tf.keras.Input(shape=[None, None, 3],
ragged=True)
input = input.to_tensor(name='input_image')
x = tf.keras.layers.Conv2D(filters=1,
kernel_size=1,
strides=1,
**conv_args)(input)
x = tf.keras.layers.BatchNormalization()(x)
x1 = x
x = hourglass(1)(x)
x = tf.math.add(x1,x)
x = hourglass(1)(x)
x = tf.keras.layers.Conv2D(filters=3,
kernel_size=3,
strides=1,
**conv_args)(x)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.ReLU(max_value=1)(x)
return tf.keras.Model(inputs=input, outputs=x, name="generator")
Initial Iterations
We conducted the initial 10 iterations with 100 (1) steps and 1 (2) image (a batch size of 1) to assess whether the model is functioning and learning efficiently. We chose parameters to yield relatively quick results.
(1) 800 steps were chosen to increase the likelihood of covering all possible images. We started with 50 original images, with a 0.5 probability of conversion to black and white, 0.5 probability of horizontal reflection, and 0.5 probability of compression distortion. In this scenario, the probability of obtaining an unchanged image is 0.125, while for all other transformations, it is 0.875. Hence, by generating 8 (400) times more images from the generator than untouched images, we aimed for an ideal situation where all images and their derivatives are included in the training dataset.
We also applied randomized adjustments to brightness and contrast within specific ranges. Ideally, we should have considered our probabilities from a Gaussian distribution perspective, aiming for the ideal scenario to occur in the majority of cases. However, we opted to double the number of pulls from the generator, resulting in a total of 800 steps.
(2) One image was used to assess the model’s processing speed. Following this, 30 iterations of 800 steps each were conducted with one image.
Initial Results
The processing speed was measured at 58ms, and the error graph showed consistent performance across epochs, which was satisfactory. However, the images produced by the model were not optimal. Since no clear insights were obtained from the statistical analysis module, we decided to continue training the model.
Source pictures:
Results:
Model loss:
Additional Iterations:
Another 60 iterations were conducted with modified parameters, where the neural network was fed a batch of 2 images to assess scaling based on batch size. Consequently, the number of steps was adjusted, halved to cover the probability distribution of our input data transformations adequately. However, we maintained a buffer due to concerns about unaccounted distributions, as later iterations are more sensitive to them.
Upon initiating training, an expected error spike occurred. The processing time per step increased to 76ms. Increasing the total number of images per iteration smoothed the error graph, and the larger batch improved the output image quality by promoting more generalized processing.
As a result, favorable outcomes were observed on the training data. However, the validation data only remotely resembled the expected results, suggesting potential overfitting or excessive conformity to the training data. The issue may lie within the model itself, the weight optimizer, or even the error function. To address this, we conducted an additional 100 iterations with an increased batch size and total number of pulls from the generator to verify the persistence of this trend.
Author — Egor Zyryanov
Leave a Reply