Requesting advice on multi-threaded matplotlib session + grids displaying data

John, That makes sense. Tks vm.

    > I am still confused on what approach needs to be taken to
    > get the data grids to be *interactive* from the python
    > command prompt. It seems to me that such interactivity
    > requires threading (ie, creating any new data grid gui
    > window in its own thread). Is that correct, or is there
    > some other way to do it?

OK, I appreciate that it's confusing. It's not an easy subject. I
suggest you give http://matplotlib.sourceforge.net/interactive.html
and http://matplotlib.sourceforge.net/faq.html#SHOW another reading
because your comments below indicate you haven't fully digested that
material. I take full responsibility for this, however, because I
wrote the docs!

I'll elaborate somewhat.

First a reminder which I think you already understand: matplotlib has
an interactive mode and a non-interactive mode. You want to use the
interactive mode, set 'interactive : True' in your rc file. In
interactive mode do not call show, ever. OK.

Your real problem is how to start the mainloop vis-a-vis threading.
The answer is (for gtk/gtkagg), use a custom python shell for pygtk.
The idea here is that the custom shell starts the mainloop for you.
When you import matplotlib with interactive on from one of these
shells, the mainloop is already started in a separate thread, and you
can issue plot commands and get immediate responses without losing
control of the prompt. Of course, you can launch other gtk windows
and widgets too.

There are a variety of python shells out there for pygtk, but I
strongly recommend you start with Fernando Perez's pyint-gtk.py.
If you notice in my last post to you, I wrote

  > Here is an example using Fernando's ipython-gtk
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  > hunter:~/python/examples/pygtk> python ~/tmp/ip/ipython-gtk.py
  > Python 2.3.2 (#1, Oct 13 2003, 11:33:15)
  > [GCC 3.3.1] on linux2
  > Type "copyright", "credits" or "license" for more information.
  > (MatplotlibShell)

  > 1 >>> from matplotlib.numerix import rand
  > 2 >>> from array_to_grid import view_array
  > 3 >>> view_array( rand(20,10))
  > Out[3]: <ArrayView object (GtkWindow) at 0x4136db94>

I would have emphasized that but I wrongly assumed you knew you needed
a threaded GTK shell. I guess I've been doing this too long :slight_smile:

pyint-gtk.py is just a temporary proof-of-concept to lay the
groundwork for ipython-gtk, which will be a full featured ipython
shell with gtk/matplotlib support. If you haven't checked out
ipython, you should, as it's *very nice* for interactive work.
Nonetheless, the current release of ipython doesn't support
gtk/matplotlib, and pyint-gtk.py is as good as the standard shell.
Plus we need testers, because the guts of this shell will become the
guts of ipython-gtk.

If you start it with

  > python pyint-gtk.py -mplot

it will not only start the gtk mainloop but will import all of
matplotlib, so if the first command you type is

  >>> plot([1,2,3)

a plot will pop up. And you can import and use the view_array code I
posted before as well.

I'll include the src for pyint-gtk.py here, as it is short

Hope this helps,
JDH

#!/usr/bin/env python
"""Multithreaded interactive interpreter with GTK and Matplotlib support.

Usage:

  pyint-gtk.py -> starts shell with gtk thread running separately

  pyint-gtk.py -mplot [filename] -> initializes matplotlib, optionally running
  the named file. The shell starts after the file is executed.

Threading code inspired by:
Building GTK GUIs interactively « Python recipes « ActiveState Code, by Brian
McErlean and John Finlay.

Matplotlib support taken from interactive.py in the matplotlib distribution.

Also borrows liberally from code.py in the Python standard library."""

__author__ = "Fernando Perez <Fernando.Perez@...179...>"

import sys
import code
import threading

import pygtk
pygtk.require("2.0")
import gtk

try:
    import readline
except ImportError:
    has_readline = False
else:
    has_readline = True

class MTConsole(code.InteractiveConsole):
    """Simple multi-threaded shell"""

    def __init__(self,on_kill=None,*args,**kw):
        code.InteractiveConsole.__init__(self,*args,**kw)
        self.code_to_run = None
        self.ready = threading.Condition()
        self._kill = False
        if on_kill is None:
            on_kill =
        # Check that all things to kill are callable:
        for _ in on_kill:
            if not callable(_):
                raise TypeError,'on_kill must be a list of callables'
        self.on_kill = on_kill
        # Set up tab-completer
        if has_readline:
            import rlcompleter
            try: # this form only works with python 2.3
                self.completer = rlcompleter.Completer(self.locals)
            except: # simpler for py2.2
                self.completer = rlcompleter.Completer()
                
            readline.set_completer(self.completer.complete)
            # Use tab for completions
            readline.parse_and_bind('tab: complete')
            # This forces readline to automatically print the above list when tab
            # completion is set to 'complete'.
            readline.parse_and_bind('set show-all-if-ambiguous on')
            # Bindings for incremental searches in the history. These searches
            # use the string typed so far on the command line and search
            # anything in the previous input history containing them.
            readline.parse_and_bind('"\C-r": reverse-search-history')
            readline.parse_and_bind('"\C-s": forward-search-history')

    def runsource(self, source, filename="<input>", symbol="single"):
        """Compile and run some source in the interpreter.

        Arguments are as for compile_command().

        One several things can happen:

        1) The input is incorrect; compile_command() raised an
        exception (SyntaxError or OverflowError). A syntax traceback
        will be printed by calling the showsyntaxerror() method.

        2) The input is incomplete, and more input is required;
        compile_command() returned None. Nothing happens.

        3) The input is complete; compile_command() returned a code
        object. The code is executed by calling self.runcode() (which
        also handles run-time exceptions, except for SystemExit).

        The return value is True in case 2, False in the other cases (unless
        an exception is raised). The return value can be used to
        decide whether to use sys.ps1 or sys.ps2 to prompt the next
        line.
        """
        try:
            code = self.compile(source, filename, symbol)
        except (OverflowError, SyntaxError, ValueError):
            # Case 1
            self.showsyntaxerror(filename)
            return False

        if code is None:
            # Case 2
            return True

        # Case 3
        # Store code in self, so the execution thread can handle it
        self.ready.acquire()
        self.code_to_run = code
        self.ready.wait() # Wait until processed in timeout interval
        self.ready.release()

        return False

    def runcode(self):
        """Execute a code object.

        When an exception occurs, self.showtraceback() is called to display a
        traceback."""

        self.ready.acquire()
        if self._kill:
            print 'Closing threads...',
            sys.stdout.flush()
            for tokill in self.on_kill:
                tokill()
            print 'Done.'

        if self.code_to_run is not None:
            self.ready.notify()
            code.InteractiveConsole.runcode(self,self.code_to_run)

        self.code_to_run = None
        self.ready.release()
        return True

    def kill (self):
        """Kill the thread, returning when it has been shut down."""
        self.ready.acquire()
        self._kill = True
        self.ready.release()

class GTKInterpreter(threading.Thread):
    """Run a gtk mainloop() in a separate thread.
    Python commands can be passed to the thread where they will be executed.
    This is implemented by periodically checking for passed code using a
    GTK timeout callback.
    """
    TIMEOUT = 100 # Milisecond interval between timeouts.
    
    def __init__(self,banner=None):
        threading.Thread.__init__(self)
        self.banner = banner
        self.shell = MTConsole(on_kill=[gtk.mainquit])

    def run(self):
        self.pre_interact()
        self.shell.interact(self.banner)
        self.shell.kill()

    def mainloop(self):
        self.start()
        gtk.timeout_add(self.TIMEOUT, self.shell.runcode)
        try:
            if gtk.gtk_version[0] >= 2:
                gtk.threads_init()
        except AttributeError:
            pass
        gtk.mainloop()
        self.join()

    def pre_interact(self):
        """This method should be overridden by subclasses.

        It gets called right before interact(), but after the thread starts.
        Typically used to push initialization code into the interpreter"""
        
        pass

class MatplotLibInterpreter(GTKInterpreter):
    """Threaded interpreter with matplotlib support."""

    def __init__(self,banner=None):
        banner = """\nWelcome to matplotlib, a matlab-like python environment.
    help(matlab) -> help on matlab compatible commands from matplotlib.
    help(plotting) -> help on plotting commands.
    """
        GTKInterpreter.__init__(self,banner)
        
    def pre_interact(self):
        """Initialize matplotlib before user interaction begins"""

        push = self.shell.push
        # Code to execute in user's namespace
        lines = ["import matplotlib",
                 "matplotlib.use('GTKAgg')",
                 "matplotlib.interactive(1)",
                 "import matplotlib.matlab as matlab",
                 "from matplotlib.matlab import *"]

        map(push,lines)
        
        # Execute file if given.
        if len(sys.argv)>1:
            import matplotlib
            matplotlib.interactive(0) # turn off interaction
            fname = sys.argv[1]
            try:
                inFile = file(fname, 'r')
            except IOError:
                print '*** ERROR *** Could not read file <%s>' % fname
            else:
                print '*** Executing file <%s>:' % fname
                for line in inFile:
                    if line.lstrip().find('show()')==0: continue
                    print '>>', line,
                    push(line)
                inFile.close()
            matplotlib.interactive(1) # turn on interaction

if __name__ == '__main__':
    # Quick sys.argv hack to extract the option and leave filenames in sys.argv.
    # For real option handling, use optparse or getopt.
    if len(sys.argv) > 1 and sys.argv[1]=='-mplot':
        sys.argv = [sys.argv[0]]+sys.argv[2:]
        MatplotLibInterpreter().mainloop()
    else:
        GTKInterpreter().mainloop()

John,

Tks again for your help. I thought that you were running that script in
plain old ipython. I didn't realize ipython-gtk was different from ipython.
After reading your last message, I downloaded and began using ipython 0.6.2
for python 2.3. I thought ipython-gtk was some linux variant of ipython (im
on windows xp). I appreciate now that it is a different animal entirely.
My mix-up on that. Sorry for the confusion.

There is no mention of ipython-gtk or pyint-gtk on the ipython.scipy.org web
site. Is there any documentation yet that I can see, perhaps tucked away in
a cvs branch somewhere?

I have enough to get going now.

Tks again,

Tom

···

-----Original Message-----
From: John Hunter [mailto:jdhunter@…4…]
Sent: Friday, August 20, 2004 8:36 AM
To: Thomas Barket
Cc: matplotlib-users@lists.sourceforge.net
Subject: Re: [Matplotlib-users] Requesting advice on multi-threaded
matplotlib session + grids displaying data

    > John, That makes sense. Tks vm.

    > I am still confused on what approach needs to be taken to
    > get the data grids to be *interactive* from the python
    > command prompt. It seems to me that such interactivity
    > requires threading (ie, creating any new data grid gui
    > window in its own thread). Is that correct, or is there
    > some other way to do it?

OK, I appreciate that it's confusing. It's not an easy subject. I suggest
you give http://matplotlib.sourceforge.net/interactive.html
and http://matplotlib.sourceforge.net/faq.html#SHOW another reading because
your comments below indicate you haven't fully digested that material. I
take full responsibility for this, however, because I wrote the docs!

I'll elaborate somewhat.

First a reminder which I think you already understand: matplotlib has an
interactive mode and a non-interactive mode. You want to use the
interactive mode, set 'interactive : True' in your rc file. In interactive
mode do not call show, ever. OK.

Your real problem is how to start the mainloop vis-a-vis threading.
The answer is (for gtk/gtkagg), use a custom python shell for pygtk.
The idea here is that the custom shell starts the mainloop for you.
When you import matplotlib with interactive on from one of these shells, the
mainloop is already started in a separate thread, and you can issue plot
commands and get immediate responses without losing control of the prompt.
Of course, you can launch other gtk windows and widgets too.

There are a variety of python shells out there for pygtk, but I strongly
recommend you start with Fernando Perez's pyint-gtk.py.
If you notice in my last post to you, I wrote

  > Here is an example using Fernando's ipython-gtk
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  > hunter:~/python/examples/pygtk> python ~/tmp/ip/ipython-gtk.py
  > Python 2.3.2 (#1, Oct 13 2003, 11:33:15)
  > [GCC 3.3.1] on linux2
  > Type "copyright", "credits" or "license" for more information.
  > (MatplotlibShell)

  > 1 >>> from matplotlib.numerix import rand
  > 2 >>> from array_to_grid import view_array
  > 3 >>> view_array( rand(20,10))
  > Out[3]: <ArrayView object (GtkWindow) at 0x4136db94>

I would have emphasized that but I wrongly assumed you knew you needed a
threaded GTK shell. I guess I've been doing this too long :slight_smile:

pyint-gtk.py is just a temporary proof-of-concept to lay the groundwork for
ipython-gtk, which will be a full featured ipython shell with gtk/matplotlib
support. If you haven't checked out ipython, you should, as it's *very
nice* for interactive work.
Nonetheless, the current release of ipython doesn't support gtk/matplotlib,
and pyint-gtk.py is as good as the standard shell.
Plus we need testers, because the guts of this shell will become the guts of
ipython-gtk.

If you start it with

  > python pyint-gtk.py -mplot

it will not only start the gtk mainloop but will import all of matplotlib,
so if the first command you type is

  >>> plot([1,2,3[])

a plot will pop up. And you can import and use the view_array code I posted
before as well.

I'll include the src for pyint-gtk.py here, as it is short

Hope this helps,
JDH

#!/usr/bin/env python
"""Multithreaded interactive interpreter with GTK and Matplotlib support.

Usage:

  pyint-gtk.py -> starts shell with gtk thread running separately

  pyint-gtk.py -mplot [filename] -> initializes matplotlib, optionally
running
  the named file. The shell starts after the file is executed.

Threading code inspired by:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65109, by Brian
McErlean and John Finlay.

Matplotlib support taken from interactive.py in the matplotlib distribution.

Also borrows liberally from code.py in the Python standard library."""

__author__ = "Fernando Perez <Fernando.Perez@...179...>"

import sys
import code
import threading

import pygtk
pygtk.require("2.0")
import gtk

try:
    import readline
except ImportError:
    has_readline = False
else:
    has_readline = True

class MTConsole(code.InteractiveConsole):
    """Simple multi-threaded shell"""

    def __init__(self,on_kill=None,*args,**kw):
        code.InteractiveConsole.__init__(self,*args,**kw)
        self.code_to_run = None
        self.ready = threading.Condition()
        self._kill = False
        if on_kill is None:
            on_kill = []
        # Check that all things to kill are callable:
        for _ in on_kill:
            if not callable(_):
                raise TypeError,'on_kill must be a list of callables'
        self.on_kill = on_kill
        # Set up tab-completer
        if has_readline:
            import rlcompleter
            try: # this form only works with python 2.3
                self.completer = rlcompleter.Completer(self.locals)
            except: # simpler for py2.2
                self.completer = rlcompleter.Completer()
                
            readline.set_completer(self.completer.complete)
            # Use tab for completions
            readline.parse_and_bind('tab: complete')
            # This forces readline to automatically print the above list
when tab
            # completion is set to 'complete'.
            readline.parse_and_bind('set show-all-if-ambiguous on')
            # Bindings for incremental searches in the history. These
searches
            # use the string typed so far on the command line and search
            # anything in the previous input history containing them.
            readline.parse_and_bind('"\C-r": reverse-search-history')
            readline.parse_and_bind('"\C-s": forward-search-history')

    def runsource(self, source, filename="<input>", symbol="single"):
        """Compile and run some source in the interpreter.

        Arguments are as for compile_command().

        One several things can happen:

        1) The input is incorrect; compile_command() raised an
        exception (SyntaxError or OverflowError). A syntax traceback
        will be printed by calling the showsyntaxerror() method.

        2) The input is incomplete, and more input is required;
        compile_command() returned None. Nothing happens.

        3) The input is complete; compile_command() returned a code
        object. The code is executed by calling self.runcode() (which
        also handles run-time exceptions, except for SystemExit).

        The return value is True in case 2, False in the other cases (unless
        an exception is raised). The return value can be used to
        decide whether to use sys.ps1 or sys.ps2 to prompt the next
        line.
        """
        try:
            code = self.compile(source, filename, symbol)
        except (OverflowError, SyntaxError, ValueError):
            # Case 1
            self.showsyntaxerror(filename)
            return False

        if code is None:
            # Case 2
            return True

        # Case 3
        # Store code in self, so the execution thread can handle it
        self.ready.acquire()
        self.code_to_run = code
        self.ready.wait() # Wait until processed in timeout interval
        self.ready.release()

        return False

    def runcode(self):
        """Execute a code object.

        When an exception occurs, self.showtraceback() is called to display
a
        traceback."""

        self.ready.acquire()
        if self._kill:
            print 'Closing threads...',
            sys.stdout.flush()
            for tokill in self.on_kill:
                tokill()
            print 'Done.'

        if self.code_to_run is not None:
            self.ready.notify()
            code.InteractiveConsole.runcode(self,self.code_to_run)

        self.code_to_run = None
        self.ready.release()
        return True

    def kill (self):
        """Kill the thread, returning when it has been shut down."""
        self.ready.acquire()
        self._kill = True
        self.ready.release()

class GTKInterpreter(threading.Thread):
    """Run a gtk mainloop() in a separate thread.
    Python commands can be passed to the thread where they will be executed.
    This is implemented by periodically checking for passed code using a
    GTK timeout callback.
    """
    TIMEOUT = 100 # Milisecond interval between timeouts.
    
    def __init__(self,banner=None):
        threading.Thread.__init__(self)
        self.banner = banner
        self.shell = MTConsole(on_kill=[gtk.mainquit])

    def run(self):
        self.pre_interact()
        self.shell.interact(self.banner)
        self.shell.kill()

    def mainloop(self):
        self.start()
        gtk.timeout_add(self.TIMEOUT, self.shell.runcode)
        try:
            if gtk.gtk_version[0] >= 2:
                gtk.threads_init()
        except AttributeError:
            pass
        gtk.mainloop()
        self.join()

    def pre_interact(self):
        """This method should be overridden by subclasses.

        It gets called right before interact(), but after the thread starts.
        Typically used to push initialization code into the interpreter"""
        
        pass

class MatplotLibInterpreter(GTKInterpreter):
    """Threaded interpreter with matplotlib support."""

    def __init__(self,banner=None):
        banner = """\nWelcome to matplotlib, a matlab-like python
environment.
    help(matlab) -> help on matlab compatible commands from matplotlib.
    help(plotting) -> help on plotting commands.
    """
        GTKInterpreter.__init__(self,banner)
        
    def pre_interact(self):
        """Initialize matplotlib before user interaction begins"""

        push = self.shell.push
        # Code to execute in user's namespace
        lines = ["import matplotlib",
                 "matplotlib.use('GTKAgg')",
                 "matplotlib.interactive(1)",
                 "import matplotlib.matlab as matlab",
                 "from matplotlib.matlab import *"]

        map(push,lines)
        
        # Execute file if given.
        if len(sys.argv)>1:
            import matplotlib
            matplotlib.interactive(0) # turn off interaction
            fname = sys.argv[1]
            try:
                inFile = file(fname, 'r')
            except IOError:
                print '*** ERROR *** Could not read file <%s>' % fname
            else:
                print '*** Executing file <%s>:' % fname
                for line in inFile:
                    if line.lstrip().find('show()')==0: continue
                    print '>>', line,
                    push(line)
                inFile.close()
            matplotlib.interactive(1) # turn on interaction

if __name__ == '__main__':
    # Quick sys.argv hack to extract the option and leave filenames in
sys.argv.
    # For real option handling, use optparse or getopt.
    if len(sys.argv) > 1 and sys.argv[1]=='-mplot':
        sys.argv = [sys.argv[0]]+sys.argv[2:]
        MatplotLibInterpreter().mainloop()
    else:
        GTKInterpreter().mainloop()