You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
tf 2.16.1
Custom code
Yes
OS platform and distribution
Windows 10
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
When I set the global seed, applied the @tf.function decorator, and performed gradient updates every two periods, I observed an unexpected phenomenon: the error values were randomly sampled from two consecutive periods with identical values. However, my expectation was that the error values should differ when sampled in each period instead of being equal in adjacent periods. Furthermore, I noticed that removing the @tf.function decorator from the model function prevented this issue of having identical error values in consecutive periods. What could be causing this phenomenon? How should one handle this situation when using @tf.function?"
Standalone code to reproduce the issue
import os
import random
import numpy as np
import tensorflow as tf
# set seeds
SEED = 0
random.seed(SEED)
np.random.seed(SEED)
tf.random.set_seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
tf.keras.utils.set_random_seed(SEED)
tf.config.experimental.enable_op_determinism()
@tf.function
def model(x):
err = tf.random.uniform(shape=(1,))
loss = x + err
return err, loss
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
training_periods = 10
# initialize x
x = tf.Variable(tf.random.uniform(shape=(1,)), trainable=True)
forttin range(training_periods):
if tt % 2 == 0:
with tf.GradientTape() as tape:
err, loss = model(x)
gradients = tape.gradient(loss, [x]) # suppose x need to be optimized
optimizer.apply_gradients(zip(gradients, [x])) # update x
print(f"Period: {tt}, err (trained): {err.numpy()}")
else:
err, loss = model(x)
print(f"Period: {tt}, err (not trained): {err.numpy()}")
# outcomes
Period: 0, err (trained): [0.01975703]
Period: 1, err (not trained): [0.01975703]
Period: 2, err (trained): [0.5400312]
Period: 3, err (not trained): [0.5400312]
Period: 4, err (trained): [0.51667833]
Period: 5, err (not trained): [0.51667833]
Period: 6, err (trained): [0.4683528]
Period: 7, err (not trained): [0.4683528]
Period: 8, err (trained): [0.14856052]
Period: 9, err (not trained): [0.14856052]
Relevant log output
No response
The text was updated successfully, but these errors were encountered:
@tonyherschel Could you try to consider explicitly clearing the cache every period.
You could also try to control the caching behavior with the jit_compile option within @tf.function by setting jit_compile to False as follows;
Hi, @sushreebarsa, thanks for your reply! I tried setting jit_compile to False in the @tf.function, but the results do not seem to have changed.
import os
import random
import numpy as np
import tensorflow as tf
# set seeds
SEED = 0
random.seed(SEED)
np.random.seed(SEED)
tf.random.set_seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
tf.keras.utils.set_random_seed(SEED)
tf.config.experimental.enable_op_determinism()
@tf.function(jit_compile=False)
def model(x):
err = tf.random.uniform(shape=(1,))
loss = x + err
return err, loss
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
training_periods = 10
# initialize x
x = tf.Variable(tf.random.uniform(shape=(1,)), trainable=True)
for tt in range(training_periods):
if tt % 2 == 0:
with tf.GradientTape() as tape:
err, loss = model(x)
gradients = tape.gradient(loss, [x]) # suppose x need to be optimized
optimizer.apply_gradients(zip(gradients, [x])) # update x
print(f"Period: {tt}, err (trained): {err.numpy()}")
else:
err, loss = model(x)
print(f"Period: {tt}, err (not trained): {err.numpy()}")
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
tf 2.16.1
Custom code
Yes
OS platform and distribution
Windows 10
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
When I set the global seed, applied the @tf.function decorator, and performed gradient updates every two periods, I observed an unexpected phenomenon: the error values were randomly sampled from two consecutive periods with identical values. However, my expectation was that the error values should differ when sampled in each period instead of being equal in adjacent periods. Furthermore, I noticed that removing the @tf.function decorator from the model function prevented this issue of having identical error values in consecutive periods. What could be causing this phenomenon? How should one handle this situation when using @tf.function?"
Standalone code to reproduce the issue
Relevant log output
No response
The text was updated successfully, but these errors were encountered: