Strange bug when saving animations (with streamplot)

Development matplotlib-devel
1

I’m having a tough time figuring this out: Saving an animation seems to hang when using streamplot. The exact same animation runs without issue when calling show.

On my system, the example below hangs consistently at frame 173. However, the number of saved frames (before hanging) varies with density of stream lines (i.e. number of line segments in streamplot): More frames saved for lower density.

Seems to be a memory or file-size issue, but

  • Memory use doesn’t seem to grow. (this suggests that ffmpeg adds frames to disk as opposed to memory.)

  • Saving the animation up to the frame just before it hangs gives modest (~1MB) videos.

Maybe this is a limitation of ffmpeg?

Can someone confirm that the example hangs on their system? (Somehow, I often run into bugs that are not reproducible.)

In case this is system dependent:

  • OSX 10.6

  • Python 2.6.1 (system install)

  • matplotlib master

  • ffmpeg 0.10

Simple plots seem to work fine, but I should probably try to reproduce using something other than streamplot (maybe create the equivalent number of line and arrow collections).

-Tony

#~~~ Example

import numpy as np

import matplotlib.pyplot as plt

import matplotlib.animation as animation

fig, ax = plt.subplots()

do nothing function that prints frame

def update(frame_num):

print frame_num

return []

Y, X = np.mgrid[:1:100j, :1:100j]

U = np.ones_like(X)

V = np.ones_like(X)

ax.streamplot(X, Y, U, V, density=1)

ani = animation.FuncAnimation(fig, update, save_count=500)

save hangs at frame 173 (this probably varies by system, if it’s reproducible).

ani.save(‘animation.avi’)

show animates all frames w/o issue.

#plt.show()

2

I finally had some time to revisit this issue. It seems that subprocess PIPEs have fairly limited buffers [1] (also see docs for subprocess.Popen.communicate [2]) and ffmpeg tends to generate a decent amount of output to stderr. A simple solution is to redirect stderr to a temporary file. This change fixes the issue on my system.

If this bug is reproducible, I can submit a PR. The temporary file here could be created using tempfile so it gets cleaned up automatically, or maybe a more permanent log file for debugging. Thoughts?

-Tony

[1] http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/

[2] http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

···

On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu <tsyu80@…149…> wrote:

I’m having a tough time figuring this out: Saving an animation seems to hang when using streamplot. The exact same animation runs without issue when calling show.

On my system, the example below hangs consistently at frame 173. However, the number of saved frames (before hanging) varies with density of stream lines (i.e. number of line segments in streamplot): More frames saved for lower density.

Seems to be a memory or file-size issue, but

  • Memory use doesn’t seem to grow. (this suggests that ffmpeg adds frames to disk as opposed to memory.)
  • Saving the animation up to the frame just before it hangs gives modest (~1MB) videos.

Maybe this is a limitation of ffmpeg?

Can someone confirm that the example hangs on their system? (Somehow, I often run into bugs that are not reproducible.)

In case this is system dependent:

  • OSX 10.6
  • Python 2.6.1 (system install)
  • matplotlib master
  • ffmpeg 0.10

Simple plots seem to work fine, but I should probably try to reproduce using something other than streamplot (maybe create the equivalent number of line and arrow collections).

-Tony

#~~~ Example

import numpy as np

import matplotlib.pyplot as plt

import matplotlib.animation as animation

fig, ax = plt.subplots()

do nothing function that prints frame

def update(frame_num):

print frame_num

return

Y, X = np.mgrid[:1:100j, :1:100j]

U = np.ones_like(X)

V = np.ones_like(X)

ax.streamplot(X, Y, U, V, density=1)

ani = animation.FuncAnimation(fig, update, save_count=500)

save hangs at frame 173 (this probably varies by system, if it’s reproducible).

ani.save(‘animation.avi’)

show animates all frames w/o issue.

#plt.show()

3

And just to clarify: My original email mentioned that saving would hang only when streamplot was used to create the figure. It turns out that ffmpeg prints more output (keyframes?) for more complex images, so when I ran simpler plots, they didn’t produce enough output to fill the subprocess PIPE (although they would eventually).

Also, theres’s a simpler fix than piping ffmpeg output to a file: Just set -loglevel quiet in the ffmpeg command.

-Tony

···

On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu <tsyu80@…149…> wrote:

On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu <tsyu80@…149…> wrote:

I’m having a tough time figuring this out: Saving an animation seems to hang when using streamplot. The exact same animation runs without issue when calling show.

On my system, the example below hangs consistently at frame 173. However, the number of saved frames (before hanging) varies with density of stream lines (i.e. number of line segments in streamplot): More frames saved for lower density.

Seems to be a memory or file-size issue, but

  • Memory use doesn’t seem to grow. (this suggests that ffmpeg adds frames to disk as opposed to memory.)
  • Saving the animation up to the frame just before it hangs gives modest (~1MB) videos.

Maybe this is a limitation of ffmpeg?

Can someone confirm that the example hangs on their system? (Somehow, I often run into bugs that are not reproducible.)

In case this is system dependent:

  • OSX 10.6
  • Python 2.6.1 (system install)
  • matplotlib master
  • ffmpeg 0.10

Simple plots seem to work fine, but I should probably try to reproduce using something other than streamplot (maybe create the equivalent number of line and arrow collections).

-Tony

#~~~ Example

import numpy as np

import matplotlib.pyplot as plt

import matplotlib.animation as animation

fig, ax = plt.subplots()

do nothing function that prints frame

def update(frame_num):

print frame_num

return

Y, X = np.mgrid[:1:100j, :1:100j]

U = np.ones_like(X)

V = np.ones_like(X)

ax.streamplot(X, Y, U, V, density=1)

ani = animation.FuncAnimation(fig, update, save_count=500)

save hangs at frame 173 (this probably varies by system, if it’s reproducible).

ani.save(‘animation.avi’)

show animates all frames w/o issue.

#plt.show()

I finally had some time to revisit this issue. It seems that subprocess PIPEs have fairly limited buffers [1] (also see docs for subprocess.Popen.communicate [2]) and ffmpeg tends to generate a decent amount of output to stderr. A simple solution is to redirect stderr to a temporary file. This change fixes the issue on my system.

If this bug is reproducible, I can submit a PR. The temporary file here could be created using tempfile so it gets cleaned up automatically, or maybe a more permanent log file for debugging. Thoughts?

-Tony

[1] http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/

[2] http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

4

It's not a bad idea to have it logged to a file in a temp directory.
We could potentially just do this if debug is set to verbose, however,
and just use -loglevel quiet otherwise. Can you manage PR for this or
do I need to?

Ryan

···

On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu <tsyu80@...149...> wrote:

On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu <tsyu80@...149...> wrote:

On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu <tsyu80@...149...> wrote:

I'm having a tough time figuring this out: Saving an animation seems to
hang when using `streamplot`. The exact same animation runs without issue
when calling show.

On my system, the example below hangs consistently at frame 173. However,
the number of saved frames (before hanging) varies with density of stream
lines (i.e. number of line segments in streamplot): More frames saved for
lower density.

Seems to be a memory or file-size issue, but

   * Memory use doesn't seem to grow. (this suggests that ffmpeg adds
frames to disk as opposed to memory.)
   * Saving the animation up to the frame just before it hangs gives
modest (~1MB) videos.
Maybe this is a limitation of `ffmpeg`?

Can someone confirm that the example hangs on their system? (Somehow, I
often run into bugs that are not reproducible.)

In case this is system dependent:
* OSX 10.6
* Python 2.6.1 (system install)
* matplotlib master
* ffmpeg 0.10

Simple plots seem to work fine, but I should probably try to reproduce
using something other than streamplot (maybe create the equivalent number of
line and arrow collections).

-Tony

#~~~ Example
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation

fig, ax = plt.subplots()

# do nothing function that prints frame
def update(frame_num):
    print frame_num
    return

Y, X = np.mgrid[:1:100j, :1:100j]
U = np.ones_like(X)
V = np.ones_like(X)
ax.streamplot(X, Y, U, V, density=1)

ani = animation.FuncAnimation(fig, update, save_count=500)

# save hangs at frame 173 (this probably varies by system, if it's
reproducible).
ani.save('animation.avi')
# show animates all frames w/o issue.
#plt.show()

I finally had some time to revisit this issue. It seems that subprocess
PIPEs have fairly limited buffers [1] (also see docs for
`subprocess.Popen.communicate` [2]) and ffmpeg tends to generate a decent
amount of output to stderr. A simple solution is to redirect stderr to a
temporary file. This change fixes the issue on my system.

If this bug is reproducible, I can submit a PR. The temporary file here
could be created using tempfile so it gets cleaned up automatically, or
maybe a more permanent log file for debugging. Thoughts?

-Tony

[1]
Subprocess Hanging: PIPE is your enemy

[2]
subprocess — Subprocess management — Python 3.12.0 documentation

And just to clarify: My original email mentioned that saving would hang only
when `streamplot` was used to create the figure. It turns out that ffmpeg
prints more output (keyframes?) for more complex images, so when I ran
simpler plots, they didn't produce enough output to fill the subprocess
`PIPE` (although they would eventually).

Also, theres's a simpler fix than piping ffmpeg output to a file: Just set
`-loglevel quiet` in the ffmpeg command.

-Tony

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

5

Hey Ryan,

If you have time, that’d be great. Otherwise, I should have some time at the end of the week to submit a PR.

-Tony

···

On Tue, Jul 3, 2012 at 10:22 AM, Ryan May <rmay31@…149…> wrote:

On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu <tsyu80@…149…> wrote:

On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu <tsyu80@…149…> wrote:

On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu <tsyu80@…149…> wrote:

I’m having a tough time figuring this out: Saving an animation seems to

hang when using streamplot. The exact same animation runs without issue

when calling show.

On my system, the example below hangs consistently at frame 173. However,

the number of saved frames (before hanging) varies with density of stream

lines (i.e. number of line segments in streamplot): More frames saved for

lower density.

Seems to be a memory or file-size issue, but

  • Memory use doesn’t seem to grow. (this suggests that ffmpeg adds

frames to disk as opposed to memory.)

  • Saving the animation up to the frame just before it hangs gives

modest (~1MB) videos.

Maybe this is a limitation of ffmpeg?

Can someone confirm that the example hangs on their system? (Somehow, I

often run into bugs that are not reproducible.)

In case this is system dependent:

  • OSX 10.6
  • Python 2.6.1 (system install)
  • matplotlib master
  • ffmpeg 0.10

Simple plots seem to work fine, but I should probably try to reproduce

using something other than streamplot (maybe create the equivalent number of

line and arrow collections).

-Tony

#~~~ Example

import numpy as np

import matplotlib.pyplot as plt

import matplotlib.animation as animation

fig, ax = plt.subplots()

do nothing function that prints frame

def update(frame_num):

print frame_num

return []

Y, X = np.mgrid[:1:100j, :1:100j]

U = np.ones_like(X)

V = np.ones_like(X)

ax.streamplot(X, Y, U, V, density=1)

ani = animation.FuncAnimation(fig, update, save_count=500)

save hangs at frame 173 (this probably varies by system, if it’s

reproducible).

ani.save(‘animation.avi’)

show animates all frames w/o issue.

#plt.show()

I finally had some time to revisit this issue. It seems that subprocess

PIPEs have fairly limited buffers [1] (also see docs for

subprocess.Popen.communicate [2]) and ffmpeg tends to generate a decent

amount of output to stderr. A simple solution is to redirect stderr to a

temporary file. This change fixes the issue on my system.

If this bug is reproducible, I can submit a PR. The temporary file here

could be created using tempfile so it gets cleaned up automatically, or

maybe a more permanent log file for debugging. Thoughts?

-Tony

[1]

http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/

[2]

http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

And just to clarify: My original email mentioned that saving would hang only

when streamplot was used to create the figure. It turns out that ffmpeg

prints more output (keyframes?) for more complex images, so when I ran

simpler plots, they didn’t produce enough output to fill the subprocess

PIPE (although they would eventually).

Also, theres’s a simpler fix than piping ffmpeg output to a file: Just set

-loglevel quiet in the ffmpeg command.

-Tony

It’s not a bad idea to have it logged to a file in a temp directory.

We could potentially just do this if debug is set to verbose, however,

and just use -loglevel quiet otherwise. Can you manage PR for this or

do I need to?

Ryan

6

I might be able to squeeze it in. If you don't see anything by the end
of the week, hit me up.

Ryan

···

On Tue, Jul 3, 2012 at 10:03 AM, Tony Yu <tsyu80@...149...> wrote:

On Tue, Jul 3, 2012 at 10:22 AM, Ryan May <rmay31@...149...> wrote:

On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu <tsyu80@...149...> wrote:
> On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu <tsyu80@...149...> wrote:
>> On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu <tsyu80@...149...> wrote:
>>>
>>> I'm having a tough time figuring this out: Saving an animation seems
>>> to
>>> hang when using `streamplot`. The exact same animation runs without
>>> issue
>>> when calling show.
>>>
>>> On my system, the example below hangs consistently at frame 173.
>>> However,
>>> the number of saved frames (before hanging) varies with density of
>>> stream
>>> lines (i.e. number of line segments in streamplot): More frames saved
>>> for
>>> lower density.
>>>
>>> Seems to be a memory or file-size issue, but
>>>
>>> * Memory use doesn't seem to grow. (this suggests that ffmpeg adds
>>> frames to disk as opposed to memory.)
>>> * Saving the animation up to the frame just before it hangs gives
>>> modest (~1MB) videos.
>>> Maybe this is a limitation of `ffmpeg`?
>>>
>>> Can someone confirm that the example hangs on their system? (Somehow,
>>> I
>>> often run into bugs that are not reproducible.)
>>>
>>> In case this is system dependent:
>>> * OSX 10.6
>>> * Python 2.6.1 (system install)
>>> * matplotlib master
>>> * ffmpeg 0.10
>>>
>>> Simple plots seem to work fine, but I should probably try to reproduce
>>> using something other than streamplot (maybe create the equivalent
>>> number of
>>> line and arrow collections).
>>>
>>> -Tony
>>>
>>> #~~~ Example
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import matplotlib.animation as animation
>>>
>>> fig, ax = plt.subplots()
>>>
>>> # do nothing function that prints frame
>>> def update(frame_num):
>>> print frame_num
>>> return
>>>
>>> Y, X = np.mgrid[:1:100j, :1:100j]
>>> U = np.ones_like(X)
>>> V = np.ones_like(X)
>>> ax.streamplot(X, Y, U, V, density=1)
>>>
>>> ani = animation.FuncAnimation(fig, update, save_count=500)
>>>
>>> # save hangs at frame 173 (this probably varies by system, if it's
>>> reproducible).
>>> ani.save('animation.avi')
>>> # show animates all frames w/o issue.
>>> #plt.show()
>>>
>>>
>>
>> I finally had some time to revisit this issue. It seems that subprocess
>> PIPEs have fairly limited buffers [1] (also see docs for
>> `subprocess.Popen.communicate` [2]) and ffmpeg tends to generate a
>> decent
>> amount of output to stderr. A simple solution is to redirect stderr to
>> a
>> temporary file. This change fixes the issue on my system.
>>
>> If this bug is reproducible, I can submit a PR. The temporary file here
>> could be created using tempfile so it gets cleaned up automatically, or
>> maybe a more permanent log file for debugging. Thoughts?
>>
>> -Tony
>>
>> [1]
>>
>> Subprocess Hanging: PIPE is your enemy
>>
>> [2]
>>
>> subprocess — Subprocess management — Python 3.12.0 documentation
>>
>
> And just to clarify: My original email mentioned that saving would hang
> only
> when `streamplot` was used to create the figure. It turns out that
> ffmpeg
> prints more output (keyframes?) for more complex images, so when I ran
> simpler plots, they didn't produce enough output to fill the subprocess
> `PIPE` (although they would eventually).
>
> Also, theres's a simpler fix than piping ffmpeg output to a file: Just
> set
> `-loglevel quiet` in the ffmpeg command.
>
> -Tony

It's not a bad idea to have it logged to a file in a temp directory.
We could potentially just do this if debug is set to verbose, however,
and just use -loglevel quiet otherwise. Can you manage PR for this or
do I need to?

Ryan

Hey Ryan,

If you have time, that'd be great. Otherwise, I should have some time at the
end of the week to submit a PR.

-Tony

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

7

Hey Ryan,

I didn’t see a PR for this issue, so I just added a fix:

https://github.com/matplotlib/matplotlib/pull/989

As mentioned before, the PR suppresses logging in ffmpeg under normal circumstances. When the verbosity level is set to debug, however, the subprocess output (stdout + stderr) are piped to sys.stdout. This is where verbose reports are posted, so I figured that was the best place (oh, actually, I guess could have piped to verbose.fileo).

If you feel the debug output should go to a file instead, let me know.

Cheers,

-Tony

···

On Tue, Jul 3, 2012 at 11:18 AM, Ryan May <rmay31@…149…> wrote:

On Tue, Jul 3, 2012 at 10:03 AM, Tony Yu <tsyu80@…149…> wrote:

On Tue, Jul 3, 2012 at 10:22 AM, Ryan May <rmay31@…149…> wrote:

On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu <tsyu80@…149…> wrote:

On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu <tsyu80@…149…> wrote:

I finally had some time to revisit this issue. It seems that subprocess

PIPEs have fairly limited buffers [1] (also see docs for

subprocess.Popen.communicate [2]) and ffmpeg tends to generate a

decent

amount of output to stderr. A simple solution is to redirect stderr to

a

temporary file. This change fixes the issue on my system.

If this bug is reproducible, I can submit a PR. The temporary file here

could be created using tempfile so it gets cleaned up automatically, or

maybe a more permanent log file for debugging. Thoughts?

-Tony

[1]

http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/

[2]

http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

And just to clarify: My original email mentioned that saving would hang

only

when streamplot was used to create the figure. It turns out that

ffmpeg

prints more output (keyframes?) for more complex images, so when I ran

simpler plots, they didn’t produce enough output to fill the subprocess

PIPE (although they would eventually).

Also, theres’s a simpler fix than piping ffmpeg output to a file: Just

set

-loglevel quiet in the ffmpeg command.

-Tony

It’s not a bad idea to have it logged to a file in a temp directory.

We could potentially just do this if debug is set to verbose, however,

and just use -loglevel quiet otherwise. Can you manage PR for this or

do I need to?

Ryan

Hey Ryan,

If you have time, that’d be great. Otherwise, I should have some time at the

end of the week to submit a PR.

-Tony

I might be able to squeeze it in. If you don’t see anything by the end

of the week, hit me up.

Ryan