NST - New Source Tree

Posted by machine on 2010-10-12 06:52 | Staff Pick!
0.61.1
Adam Lyskawa
Wed, 2013-09-25 15:29
  • : 4.1 - 8.*
  • : 4.1 - 8.*

Code browser / source tree panel

NST screenshot

Based on great Komodo Source Tree by Iván Campaña. Now completely rewritten to implement all new features like unlimited nesting, reverse maping, persistent buttons, definition blocks and more. Tested on as big code samples as: itself (JavaScript, RDF, XUL, CSS), jQuery (JavaScript), Komodo source code (Python), MACHINE application framework (PHP, JavaScript, XHTML, HTML, CSS), DVDRip (Perl), various Komodo add-ons.

So why would you need it? If you work on big projects, it's very hard not to get lost in your code without a decent map. NST provides it, in a manner similar to Komodo Code Browser. One great feature I've discovered just tonight is it gives a hint which params functions in current file accept, in case Komodo Code Intel gets lost, and sometimes in JS it does.

Please report any bugs found, thank you.

New in NST 0.61.1

  • FIXED: Komodo 8.5 compatibility issues (pushed by ToddW - ActiveState)

New in NST 0.61

  • FIXED: Komodo 8 compatibility issues
  • ADDED: Ruby and C++ support (experimental, not tested yet)

New in NST 0.59

  • NEW: Komodo 7.0 Support
  • FIXED: internal dependencies removed

New in NST 0.58

  • FIXED: icons exception bug
  • FIXED: bug triggered with clicking on empty tree

New in NST 0.57

  • FIXED: tree column bug with another Komodo window
  • FIXED: moving between panels
  • FIXED: startup issues (initial selected state)

New in NST 0.56

  • FIXED: Python support (code level calculation, block literals handling)

New in NST 0.55

At last!

NST is on GitHub. There are 2 branches, stable and testing, the last one is default. The stable branch is the current release, testing - next release.

How to use the new repo on Linux:

cd ~/.komodoedit/6.1/XRE/extensions
rm -Rf NST@nisza.org # to remove old version first
git clone git://github.com/HTD/NST.git NST@nisza.org

For write access use

git clone git@github.com:HTD/NST.git NST@nisza.org

instead.

Behave, preserve code formatting, document your changes, and make things simple and fast. If it's too complicated - it's probably wrong. KISS should be the golden rule not limited to web developers. Complying to this rule made whole thing possible and maintainable. Let's keep it this way.

chodorowicz | Wed, 2010-10-13 15:06

many many many thanks! that was one of the top things missing in Komodo Edit (cannot afford IDE for now)

gigi | Fri, 2010-10-15 21:13

Thanks very much for adding the 'mark current function' feature!! NST is a great extension.

machine | Mon, 2010-10-18 02:28

I used all samples I found online, and they seems to look ok in NST. Ok, in fact I tested the latest version only with

http://www.exit1.org/dvdrip/index.cipp

Python is on its way, but there is still a bug with nesting, so I disabled it for now.

rachline | Mon, 2010-10-18 02:33

it looks great and it is working great.
thank you for all the hard work you have done on this extension :)

rsyring | Fri, 2010-10-22 13:32

All I can say is wow, thank you!

rachline | Sun, 2010-10-24 01:40

getting the cursor in the desired function is a welcome addition.
i think that you just made it perfect;)

thanks you.

rsyring | Wed, 2010-10-27 19:00

Thanks for both of the 0.5.x updates.

fdisk | Thu, 2010-11-11 12:29

Smarty template engine for PHP is actually html file with css and javascripts. The file extension is usually an *.tpl
But user can specify other extension.
Could you please add some configuration. For example specify a file extension and assign the parser.

machine | Fri, 2010-11-12 11:00

I'll add HTML/CSS/JS in one file support, but not very soon. It's not easy to implement.

Extensions configuration is already handled by Komoodo. If you select a language from menu, NST would use this language regardless of file extension.

stickman | Tue, 2010-11-16 02:36

Not sure if this is intended behaviour, but every time I save a file or move between tabs the source tree collapses. So I have to expand it again to see the tree. Is there any chance you could stop it from doing this (or provide an option to do so)?

Thanks for this really useful add-on.

(Win XP; Komodo Edit 6.0.1, build 6535; NST 0.52)

machine | Tue, 2010-11-16 02:38

You can click to expand tree by default, it remember its state, but I think you mean to save branches which was manualy expanded. I'll see what I can do and queue for the next release.

stickman | Tue, 2010-11-16 02:47

I work in PHP and mainly in classes, one class per file, so if I change tab or save the tree collapses and all I see is the class name. It's not the end of the world but it would help me if it didn't do this...

cn01 | Thu, 2010-12-02 03:36

Hi Adam,

Thanks for this great extension - it makes Comodo Edit way more usable to my mind.

I noticed that with the default install on version 0.52 the menu View -> Tabs&Sidebars does not list NST. As soon as I tweak the file tree.xul a little it appears fine :-) Here are my changes of the menu declaration section (not sure if ko.uilayout is better than just uilayout - I saw both ways before, but I followed the layout in Dafi's klint here)

...
    <menupopup id="menu_view_tabs_popup">
      <menuitem id="show_NST_tab2"
                 observes="show_NST_tab"
                insertafter="show_codebrowser_tab2"
                class="menuitem-iconic-wide"
                oncommand="ko.uilayout_ensureTabShown('NST_tab', true)"
                label="NST"
                 type="checkbox"/>
    </menupopup>

  <menupopup id="tabPicker_popup">
    <menuitem id="show_NST_tab"
              insertafter="show_codebrowser_tab"
              oncommand="ko.uilayout_ensureTabShown('NST_tab', true)"
              label="NST"
              type="checkbox"/>
  </menupopup>
  <tabs id="right_toolbox_tabs">
    <tab id="NST_tab" label="Source"/>
  </tabs>
...

The same thing as diff to have the full picture:

--- D:\download\NST-0.52.ko\content\tree.xul
+++ C:\Users\CN01\AppData\Local\ActiveState\KomodoIDE\6.0\XRE\extensions\NST@nisza.org\content\tree.xul
@@ -2,20 +2,23 @@
 <!DOCTYPE overlay PUBLIC "-//MOZILLA//DTD XUL V1.0//EN" "http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
 <?xml-stylesheet href="chrome://NST/skin/default.css" type="text/css"?>
 <overlay id="NSTOverlay" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
-  <popupset id="extensionPopupSet">
+
     <menupopup id="menu_view_tabs_popup">
       <menuitem id="show_NST_tab2"
+                 observes="show_NST_tab"
                 insertafter="show_codebrowser_tab2"
                 class="menuitem-iconic-wide"
-                oncommand="uilayout_ensureTabShown('NST_tab', true)"
-                label="NST"/>
+                oncommand="ko.uilayout_ensureTabShown('NST_tab', true)"
+                label="NST"
+                 type="checkbox"/>
     </menupopup>
-  </popupset>
+
   <menupopup id="tabPicker_popup">
     <menuitem id="show_NST_tab"
               insertafter="show_codebrowser_tab"
-              oncommand="uilayout_ensureTabShown('NST_tab', true)"
-              label="NST"/>
+              oncommand="ko.uilayout_ensureTabShown('NST_tab', true)"
+              label="NST"
+              type="checkbox"/>
   </menupopup>
   <tabs id="right_toolbox_tabs">
     <tab id="NST_tab" label="Source"/>

For personal use, I also wrote a little macro to allow activating NST via a keyboard shortcut. It might not be the most efficient way, but it was the only way I found so far, having no direct cmd_... support.
Just in case it helps anyone else, here it is:

// ui_helper functions - modified to not do any logging, but otherwise identical to the ones in uilayout.js
toggleSplitter = function (aCommandID) {
        var elt = document.getElementById(aCommandID);
        if (!elt) {
                return;
        }
        var boxId = elt.getAttribute('box');
        var box = document.getElementById(boxId)
        if (!box) {
                return;
        }
        var splitterId = elt.getAttribute('splitter')
        var splitter = document.getElementById(splitterId)
        if (!splitter) {
                return;
        }
        if (!box.hasAttribute('collapsed') || box.getAttribute("collapsed") == "false") {
                box.setAttribute('collapsed', 'true');
                splitter.setAttribute('collapsed', 'true');
                elt.removeAttribute('checked');
        } else {
                box.setAttribute('collapsed', 'false');
                splitter.setAttribute('collapsed', 'false');
                elt.setAttribute('checked', 'true');
        }
        if (aCommandID == "cmd_viewLeftPane") {
                CodeIntel_UpdateCodeBrowserVisibility();
        }
}
ensurePaneShown = function (tabs) {
        var splitterId = tabs.getAttribute('splitterId');
        var splitterWidget = document.getElementById(splitterId);
        var splitterCmdId = splitterWidget.getAttribute('splitterCmdId');
        if (splitterWidget.hasAttribute('collapsed') && splitterWidget.getAttribute('collapsed') == 'true') {
                toggleSplitter(splitterCmdId);
        }
}
ensureTabShown = function (tabId, focusToo) {
        try {
                if (typeof(focusToo) == 'undefined') focusToo = false;
                var wm = Components.classes["@mozilla.org/appshell/window-mediator;1"].getService(Components.interfaces.nsIWindowMediator);
                var mainWindow = wm.getMostRecentWindow('Komodo');
                var tab = mainWindow.document.getElementById(tabId);
                if (!tab) {
                        return;
                }
                var tabs = tab.parentNode;
                // First make sure that the pane the tab is in is visible
                ensurePaneShown(tabs);
                tabs.selectedItem = tab;
                if (focusToo) {
                        tab.focus();
                }
        } catch (e) {}
}
// end ui_helper_functions
komodo.assertMacroVersion(3);
if (komodo.view) {
        komodo.view.setFocus();
}
// show NST tab and activate it
ensureTabShown('NST_tab', true);
// trigger a refresh just to be sure
extensions.NST.refresh('button');
// focus back to edit window
if (komodo.view) {
        komodo.view.setFocus();
}

machine | Thu, 2010-12-02 04:53

Thanks, I'll include changes in next version, the macro seems useful too.
I have a great idea of better source tracking, thanks to Komodo team, but I have too much work right now to continue with NST. But (as Terminators say) I'll be back :)

seanmonstar | Wed, 2010-12-08 18:19

By not showing keys that have quotes around them, this pattern gets left behind :(

var A = new Class({
    'protected doStuff': function() {
        //do things
    }
});

I don't expect the "protected" to be parsed off, I just expected to see "protected doStuff"() in the tree view.

cn01 | Fri, 2010-12-17 13:21

Hi Adam,

I'm not sure how much time you have to spend on this at the moment and how much priority a better JS parser is for you. However I recently stumbled across a perhaps useful JS only Narcissus implementation.

The original Narcissus was written by Brendan Eich - the father of JS who developed it for Netscape originally and it's a bit more forgiving and also slightly faster than e.g. JSLint. Guillaume Lathoud build the JS only version, used it for a JSLint like test suite and it's downloadable here. It also seems to run fine inside Komodo according to the test suite he provided.

The parser returns a tree which could perhaps be used to include such nice stuff like variable names as well as a more complete function set for JS in NST.

Once I'm less swamped I might also play with it myself, but not sure about the century this will happen in at the moment, so I rather decided to provide the idea and hope for the Terminator's return or another kind soul in the community :-)

machine | Fri, 2010-12-17 13:41

The problem is - NST uses a common parser for several languages. It's good because it provides help for some languages Komodo IDE's Code Browser doesn't.

But to be honest - there is no problem in making separate parser for JS. I plan to develop NST, so this parser could be really great. If someone else wanted to try - good luck. All you have to do is to build an array of node items, I left some comments in code, so it should be quite easy to figure out how it works.

The real problem is time. Too much work, constant overtime because of missed deadlines. But there are some things to do with NST. I'll have to gather all bugs reported here, fix them, add better mapping between code and tree (Komodo guys answered me how to do it quite long time ago, and I didn't even have time to check this out). I'll definitely check this parser. Komodo is too good to have missing features ;).

sven | Sun, 2011-01-02 04:39

Hi,

thanks for this great addon. I had hacked the old source tree addon a little bit and maybe put some of these features into this addon.
Is there a public repository for this addon? Github or something like that?
Then i could build my patches against the HEAD.

Thanks,
Sven

machine | Sat, 2011-01-15 06:53

You're right, I have to set up public repository. I just don't know Git well enough to use it. I plan to make it work on February. Now I have too much work to even touch it. So - all reported bugs and repository in February :)

If you have fixed version, fixed nice and clean, tested - please send it to me machine at nisza.org, I'll upload it here as new version.

sven | Fri, 2011-08-26 08:48

I created a git repository from the latest release and put added a Makefile:

https://github.com/sni/NST

I also added basic support for c++:

https://github.com/sni/NST/commit/e203b437d744132106f10d0a5df05af79b83e1...

wesamly | Wed, 2011-01-12 03:54

First of all, thank you for this great extension.
One thing I noticed, when viewing css document that contains brackets
within the definition block. My English is not that good, so I took a screen shot:

http://img402.imageshack.us/img402/8692/snapshot1o.png

Keep up the great job ;)

machine | Thu, 2011-02-03 10:26

It's due to ",\n" sequence is special and used for multiple selectors per definition. I didn't know this syntax, its Mozilla specific CSS - I'll try to implement it in 0.54. It could be great if you could provide a sample CSS file, use jsfiddle for it, it's a great sketchpad for web code.

matic | Sat, 2011-01-15 06:36

Hey,

first of all: just awesome!! Amazing Plugin for Komodo Edit.

But I wonder if it is possbile to add php class attributes to the NST:

class AnalyseHelper extends AppHelper {
   
    var $helpers = array('Html','Form','Js' => array('Jquery'));

the NST doesn't show "helpers" in the tree. Is it possible to add this feature?

matic

machine | Sat, 2011-01-15 07:10

I didn't intend to add properties to the tree, but in February I plan to set up a public repository - so maybe someone will add this feature. I used NetBeans source tree and Code Browser from Komodo IDE for some months and more often I found I prefer shorter tree. If we add more and more items to the tree, searching it could lose the advantage of being so quick.

Technicaly - no problem with this. Just new icons and rules for the parser. We need public, protected and private variables, then public static, protected static and private static. 6 new icons, and a few new rules to match cases. Well, at least for PHP. I don't know other languages well enough to write rules for them. But before this - an option to remember which nodes was collapsed last time must be added.

The icons in NST I took are from Visual Studio, edited with GIMP :) I think they are modified enough to use them :)

oneman | Thu, 2011-01-27 01:24

Excellent add-on! I discovered however that when xml tags contain underscores, only the first part of the tag name is displayed and worse, the whole hierarchical structure of the file is lost, every tag is interpreted as a child tag of the former one.

Komodo Edit, version 6.0.3, build 6811
NST 0.52

mykes | Wed, 2011-02-02 08:13

Awesome how it works with Javascript for my purposes.

Does it make sense to parse JSDoc or some other documentation/commenting system to provide more info in the tree than just the function names and variable names?

jones | Fri, 2011-02-18 03:57

This is a great extension, thanks.

I wonder if when looking at an html file, instead of just having the tag structure, it could show IDs and classes of elements too. Would really make this a killer extension!

machine | Fri, 2011-02-18 04:23

But the HTML without IDs and classes must be handled too, you can't always have IDs :) So we have to add another button for HTML. But how would you make the structure? The same as it is, only fitered to elements having id or class? I'm not sure what to do with something like this

<div id="a">
  <div><!-- this one has no attribute, but its child has -->
    <span class="a1">...</span>
  </div>
</div>

I think it should go like this:

a-|_
    a1

jones | Fri, 2011-02-18 07:32

How about:

div#a -|_
        div -|_
              span.a1

If you had this html:

<div id="a" class="b">
        <div>
                <span class="c d"></span>
        </div>
</div>

Then I would suggest:

div#a.b -|_
        div -|_
              span.c.d

OK
machine | Fri, 2011-02-18 07:41

Scheduled for the 0.54. Maybe next week I'll find some time. I think reverse map would make it a little faster too - with huge form scrolling with arrow key caused refresh problems.

jones | Fri, 2011-02-18 08:19

Thanks for being so responsive. With this added, your plugin will really make my life significantly easier. Thanks.

machine | Sun, 2011-02-20 23:52

It's done, works exactly like it should. First I've added tag attributes to the tree.

It affects whole XML parser and I think it's a good thing. The bad thing is it could be a little bit slower, since tag matching uses a little bit more complex regex. Then again - I can't notice any difference on my machine with ca. 1000 lines of bloated HTML.

Next I've implemented HTML filter as follows:
- if the tag has neither name, id, nor class attribute AND
- has no child nodes with special attributes
the tag is filtered (node removed). It means html node will be filtered (leaving nothing) only if there is no node with special attributes inside. The head node will always be filtered, since it doesn't contain any node with special attributes.

Next I've added new button for toggling this feature, which appears instead of "toggle sort" button for HTML. Sorting for HTML makes no sense anyway.

The last thing to do is reverse mapping to gain some speed.

jones | Mon, 2011-02-21 07:18

Just checked out the new version and yes, you are awesome! Thanks.

quebor | Wed, 2011-02-23 12:08

Hi,

I just started using NST after using the Komodo Source Tree for some time, and so far its been great. Thanks a lot!

One thing I was hoping this would have that the other source tree doesn't is support for functions and classes that span multiple lines. For example:

def myFunc(arg1, arg2, arg3,
                     arg4):

This does not show up in NST.

I'm using version 0.54 of NST and Komodo 5.2.

Thanks!

machine | Mon, 2011-02-28 15:55

My job is killing me right now, no time to even reply an email...
I'll try to fix it in next release, Python parser is a very tricky one, I'm not satisfied with how it works, but at the moment I have no idea how to fix it.

There's still a bug visible with one of Komodo sources :)

joel.cairney | Wed, 2011-03-02 08:51

Hi,

I have some javaScript code that is mixing public and private function declarations, and the functions that do this are not showing up in the source tree as either.

function myClass() {
   
    this.myFunction = function myFunction() { ... } // this doesn't show.

}

Would it be possible to fix this in a future version?

barryvan | Thu, 2011-03-03 00:57

If you have XML fragments (that aren't well-formed), NST seems to go somewhat haywire. For example, something like this:

<alpha>
  <beta>pony</beta>
</alpha>
<gamma>
  <delta>pony</delta>
</gamma>

comes out with a tree that looks something like this:

-> alpha
  -> beta
    -> gamma
      -> delta

It'd be great if NST could handle this more elegantly, so that the output is

-> alpha
  -> beta
-> gamma
  -> delta

Perhaps an invisible parent node?

jixor | Wed, 2011-03-23 16:51

In PHP it is possible to span the function over multiple lines, e.g.

public function example(
    array    $variable      = array('foobar'),
    stdClass $somethingElse = null
    )
{
    return false;
}

However this syntax causes the function to not appear in the source tree.

jixor | Wed, 2011-03-23 16:56

Small CSS bug, if the last rule in a set is not terminated with a ; it causes unwanted indentation.

Incidentally there used to be a great bug in the css parser that allowed me to group and title sections of css making it a lot easier to navigate large files. If you could bring in some comment syntax that would intentionally allow this behaviour that would be fantastic.

jumpingGrendel@... | Fri, 2011-04-15 14:03

Hi,

I installed this without issue, but couldn't get it to appear in the UI. Closer examination showed tree.xul referencing a nonexistant function, "ko.uilayout_ensureTabShown". I changed it locally to ko.uilayout.ensureTabShown('NST_tab', true) and it started working. Maybe the Komodo API changed in recent versions? Anyway, hope this helps.

John

tito | Sat, 2011-04-16 09:14

Did you plan to setup a repository then we can submit fixes?
Regards,
Tito

davestewart | Tue, 2011-04-19 17:35

I've been hoping and waiting for about a year now that you'll implement a proper visual hierarchy for JavaScript classes.

At the moment, I still get

class.prototype.function1()
class.prototype.function2()
class.prototype.function3()

as opposed to

class.prototype
    function1()
    function2()
    function3()

Is this on the radar to be done? It would make laying out and navigating larger classes a lot easier!

Cheers,
Dave

chriswa | Sat, 2011-04-30 01:34

I wouldn't be using Komodo Edit if it wasn't for this extension. Thank you for all your hard work! <3

Do you want some help setting up a source repository? If you upload a zip of your source code somewhere and post a link, myself or someone else will find it a proper home for you. Do you have a source control preference? Google Code is pretty nice and uses SVN.

I'd love to have access to the source so I can make little customizations.

I hope you're not hesitating because you want to pretty up your code for us - we won't judge! :)

- Chris

machine | Sat, 2011-09-03 04:43

I don't know why I don't get notification about new comments, I'm checking them right now, after releasing a new version.

I think I have to state it clear on main page: NST has the BSD license, so it's fully open source and can be modified, but to release derived work you need a permission.

I think I should change it to MPL (Mozilla Public License) to make it more "compatible" but I'll have to read more first.

Anyway, I encourage everyone to modify the source and send me updated code so I can upload it here, so everyone will get the updated version automagicaly. Autoupdate is here to stay :)

I kindly ask you only to document your changes in JSDoc style and remain formating rules used in its files (no tabs, 2 spaces per indent, UTF-8 encoding, no BOM, LF as line ending, opening brace in the same line and so on...) Please test it with some bigger files (Komodo sources are great for testing).

As I said, it will be on Github, just not right now. Oh, you already have the source, it's in xpi file - it's the most common zip on Earth.

Shame on me, but I haven't used source control yet, but Git seems OK to me, Goolge... AFAIK they are evil now, and I will not endorse evil.

ivanhoe011 | Thu, 2011-06-09 19:17

Great extension, the only thing really missing is a support for SQL files, the old source tree had it and it was extremely useful feature.

SQL
machine | Sat, 2011-09-03 04:44

Scheduled for the next release.

meter | Sat, 2011-06-18 08:39

Awesome add-on. A bug: With Python (the only language I tested), function and method headers that span more than one line are not listed by NST, e.g.:

    def foo(self, blah=42,
            thing=True, gob=0):

robert.joy@xirr... | Wed, 2012-02-01 08:13

Ivan's sourcetree supported these languages. It appears this does not. Can you please add support for these? Thanks!

codejumper | Wed, 2012-02-01 09:24

can you add (restore) support ruby files (*.rb)?

thank you

machine | Wed, 2012-02-01 23:49

There are only 2 things I reused from the old Source Tree: TreeView component and the idea to parse code line by line. The parser is new and completely different. No parsing code is compatible with old Source Tree, so it has to be written from scratch.

Languages we need: TCL, SQL, Ruby, C#, C, C++.

Well, NST is Open Source, so feel free to help me with this :) But don't forget to test your code good before committing.

There's another way you can help: please provide code samples to me. Perfect code sample should be long and contain all language structures you want to see mapped.

I speak only JS, PHP, XML (all breads), C#, SQL and Python (a little). So without your help it will be very difficult to me to implement new languages. Even when I'll get the basics - it's still not enough to write a good parser for them. The ActionScript case: I knew the syntax (at least I thought I knew). But there was a guy who sent me some very good samples and we made it.

There is a GitHub project, anyone can add a new language there, of course I can add it too, but I have a lot of work right now so it has to wait for a week or two.

I strongly advise you to wait for version 0.60 - it will probably have better code separation (parsers in separate file or even files). There will be some major changes. Estimated time: 2 weeks.