Does every message have to end with a dot to be printed immediately?

Spent several hours trying to understand why the output is not flushed for a python script, it appeared that Octopus prints a message immediately only if it ends with a dot, otherwise Console output not displaying until step is finished happens.

Is this an expected behavior?

Hi @user10,

Thanks for reaching out!

From my understanding, Python will write redirected output to a buffer before outputting it. If you flush this cache the output should be written immediately!

I’ve just tested writing every three seconds using the flush parameter in the print function:

import time, sys
for i in range(10):
    print('test', flush=True)
    time.sleep(3)

Feel free to reach out if you have any questions or run into any issues!

Best Regards,

Thanks for a quick answer!

I tried using

    sys.stdout.write(text)
    sys.stdout.write('\n')
    sys.stdout.flush()

and
print(text, flush=True)
none of which worked, until I added a dot at the end of the printed text.

If the text were buffered by the Python runtime, Octopus wouldn’t be able to show it immediately disregarding the presence of a dot. To prove my point I’ve tested combinations flush/noflush x dot/nodot.

The test step:

echo "running"
script_path="/home/deploy/.octopus/Applications/OctopusServer"
python3 -W ignore  $script_path/test.py

image

The test Python script:

import time
import sys

def wo(f):
    for x in 'qwerty':
        print(time.ctime() + ' test without dot', flush=f)
        time.sleep(1)

def wi(f):
    for x in 'qwerty':
        print(time.ctime() + ' test with dot.', flush=f)
        time.sleep(1)

print('started', flush=True)
wo(True)
wi(True)
wo(False)
wi(False)
print('finished', flush=True)
exit(-1)

The result:

Notice how Octopus timestamps differ only inside of the output for the flush+dot case.

UPD:
Octopus 3.17.13
I guess it’s a long ago fixed bug, but Octopus is company-managed

Hi @user10,

Thanks for the detailed reply!

I tested this on the latest version of Octopus and found that when flushing the buffer the log would feedback with updated timestamps regardless of if there was a trailing “.”.

I also tried this on 2020.1.10 and found that when flushing the buffer with or without a trailing “.” there was relatively good feedback but it seemed a bit sluggish.

I’m uncertain if using the built-in Python source editor in these versions played a part in increasing the responsiveness.

I’m not entirely certain whether or not the increase in the responsiveness on these versions will be adequate enough for your use case, would you be able to set up a test instance with a newer version to see for yourself?

I’ll await your response; however, we stopped updating Octopus version 3.X and it may be best to consider an upgrade if possible to see if this improves your experience.

Thanks and Kind Regards,
Adam

Since the Octopus server I’m dealing with is owned by the company I’m working for, I don’t really think migration is currently an option.

Just wanted to find our whether that behavior was intentional and was documented anywhere.
Looks like it’s not, and as 3.X is out of support, adding trailing “.” is a working solution for now.

Thank you!

Hey @user10,

Just jumping in here for Finnian and Adam as they are both offline.

I will reach out to our developers and ask if what we’re seeing on our end is expected.

I am glad to hear that you at least have a workaround going forward til you can upgrade.

Please feel free to reach out in the meantime while we wait on a response.

Best,
Jeremy

Hi @user10,

Our engineers have had a look through the code and can confirm that because we are reading from stdout, Python will only output once the buffer is flushed. It’s curious that the ‘.’ is flushing the buffer as you’ve seen, I wonder if this would also be the case for other characters that often terminate sentences such as ‘\n’ or ‘!’.

As you mentioned, v3 Octopus is not supported anymore and so it is unlikely that we will backport the fix, however I believe there is a workaround that should work for your entire environment without needing to modify your scripts!

I’ve seen that it’s possible to configure python to run in unbuffered mode by setting an environment variable PYTHONUNBUFFERED to true. This should allow you to write to the console immediately, and not have to update any of your scripts.

Feel free to let me know if this doesn’t help or you have any questions.

Best Regards,

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.