ActiveState Powered by ActiveState

ActiveState Community


TextMate-style Snippets + Questions for the Komodo developers (UPDATE 2)

Posted by Stan on 2008-03-14 18:15

Please make sure to read all updates to this thread:
UPDATE 1
UPDATE 2

Demo is available here: http://www.psp-webtech.co.uk/komodo/demo/sample.html

Background Information

I have been using Komodo for some time now. I switched from Zend Studio in a hope for a faster and better editor. Komodo is fantastic and the JavaScript macros put a whole new meaning to the word “extensible”. However I found I was missing one very important feature from Zend – templates. Komodo has support for snippets, but triggering snippets from your toolbox is slow and too complicated. I would like to be able to hit TAB and have my template appear in the editor. I also didn’t like the idea of organising all my snippets inside the toolbox. I would like to be able to edit my snippets in Notepad or any other editor. I also want to be able to provide much more complicated scenarios where for example I select some number inside of the editor, I hit TAB and a menu appears allowing me to select an action e.g. convert British pounds to US dollars.
Having all of the above in mind I started working on my own Textmate-style macros/templates.

Screen grabA Better Engine

After spending two nights working on this engine, I am finally satisfied with the end result.
This JavaScript macro allows you to organise your snippets in a simple XML file which you upload on a website or on your localhost. The idea behind this is to be able to have the snippets shared across your team or between your work and home PC.
Each snippet can aggregate other snippets. For example you can combine your HEAD and BODY snippets to form the HTML snippet.
This engine also supports tab stops. When you reach the last tab stop in a given snippet and you hit TAB again the editor caret is moved back into position.

What else is there/to come?

+ Snippets based on the file type, CSS, HTML
+ Display a menu when more than one snippet is found i.e. doctype
+ Chained snippets – type ul inside an HTML document, hit TAB to expand the snippet, hit TAB again to insert a new list item
+ Support for local files was added. No need to put the snippets on a web server

At this point not much else is provided by the engine. The snippets are handled by Komodo so you get all the usual stuff. Here is a list of things I am considering for the next big update of the engine:
- Placeholders i.e. <h1 id="page-title">Page Title</h1>
- Auto-update engine
- Facility to execute remote URLs i.e. send active selection to a L10N table inside the DB and return the key back
- Dynamic snippets i.e. select a number within the editor and do [[%convert: GBP ->EUR]]

Enough information already, I want the macro

For those of you that made it through to here, I am attaching the engine as a KPZ file. To install follow these steps:
- Fire up Komodo and go to Toolbox -> Import Package…
- Locate the KPZ file on your computer and select Open
- Open your Toolbox panel (View -> Tabs -> Toolbox) and locate the Snippets entry in the list
- Right click and select Edit Macro
- Note the SNIPPET_URL at the top, make sure this is updated to point to a valid address where the snippets XML file is located at (a sample file is available here: http://www.psp-webtech.co.uk/komodo/snippets.xml)
Close Komodo down and fire it up again. Open a new PHP file, type htmlpage and hit TAB. If everything installed properly, you should have an HTML skeleton page appear in your editor.

How to create new snippets

Please refer to the sample XML file here:
http://www.psp-webtech.co.uk/komodo/snippets.xml

Ideas, comments ?

I would really like to get your opinion on the engine – does it work for you, what else you need, how flexible/fast you find it to be, etc. Any feedback would be much appreciated as I hope I will continue to spend time on this project.

Enjoy!

AttachmentSize
Snippets.kpz17.43 KB

Stan | Sat, 2008-03-15 09:59

I went to bed last night quite pleased with what I had accomplished for just two days. However, something was really bugging me. Why use a text file for all snippets? This is not “extensible” in any way. Adding more features to the engine will turn this file into a mess of code, rules and snippets.

What’s new

This update includes several new features:
- Snippets are now organised in a handy XML file. Sample is provided here: http://www.psp-webtech.co.uk/komodo/snippets.xml
- Support for language-specific snippets i.e. you can have the same trigger apply different snippets for different file types
- Support for snippet selection i.e. if you have more than one template for a given snippet, a popup menu will allow you to choose which one you want to apply

The KPZ file above has been updated to include the new version.

Stan | Sat, 2008-03-15 12:15

I have just edited my post and it seems to have also wiped out yours. I hadn’t realised it does that, sorry. Anyway, I just wanted to say thanks for giving this macro a try. The script has evolved a lot and I think now it’s even more powerful than the template engine in Zend Studio.

A questions for the Komodo devs

I plan to spend some more time working on the macro tomorrow. I want to integrate all available triggers into the autocomplete list. I did a quick search in the Komodo JS APIs, but I didn’t find any interface that would allow me to plug into this service. I am considering building CIX files on the fly, but I am not sure how to instruct Komodo to load it. Any help would be much appreciated.

ericp | Sat, 2008-03-15 13:18

Insaned, could you edit the traceback changing all "[less-thans]"s to "[ampersand]lt;"s?

I'm advocating that we let the toolbox, and toolbox "parts", be representations of filesystem hierarchies, instead of monolithic XML files. That way you could simply
drop new snippets in them. Komodo now does filesystem notifications,
(see koIFileNotificationService), so it shouldn't be too hard. Yes, I got the
idea from this obscure editor on the Mac :)

Also see the Abbreviation support we added in 4.3 --

See the ko.abbrev singleton in chrome/komodo/content/library/abbrev.js

I'll leave it to Trent to comment on integrating with the autocomplete service.

- Eric

Stan | Sat, 2008-03-15 13:48

Eric,

Many thanks for your comments. If by traceback you mean my previous post, I just fixed it.

I am using an XML file in this extension as I want to be able to share my snippets across the team i.e. whenever I make a change, I want it to propagate to all team members (and friends). I am not aware of a way to synchronise the toolbox across different installations of Komodo. Each snippet in the XML file also has a couple of extra properties which I am not sure could fit quite easily in the hierarchy of the toolbox. I am considering exploring the “Remote File” part as this could replace the XMLHttpRequest I have in there at the moment.

I’ve just had a look at abbrev.js and I think there are a couple of nice features I could borrow (like the use of origAnchor versus the offset I store at the moment).

Any help or a steer in the right direction regarding the autocomplete integration would be much appreciated.

Isaak | Sun, 2008-03-16 07:37

Have you fixed the tabStopRegEx = /(?«|\%tabstop\:)/g; issue?
It also seems that my post re-appeared :).

--
Isaak Malik
Web Developer

Stan | Tue, 2008-03-18 09:59

Hi Isaak,

I have updated this to the following (download KPZ above):

tabStopRegEx = new RegExp('(«|\%tabstop\:)', 'g');
tabStopRegEx = new RegExp('(«|\%tabstop\:)', 'g');

I am not sure why you get this error, could be the "«" character I have in there. Can you please confirm if it is OK now ?

Isaak | Mon, 2008-03-17 06:46

Ok, I downloaded the new version of your extension and the error doesn't show up anymore. I've also tested the snippets and it's working great.

Thanks again :)

--
Isaak Malik
Web Developer

Isaak | Tue, 2008-03-18 08:00

Your change did not fix the error so I had a closer look at it myself and fixed it. The bug was that you used a wrong syntax, here's the bugfix:
tabStopRegEx = /(?:«|\%tabstop\:)/g;
Notice the "(?:)" instead of "(?)".

If this is not really what you require please explain the details and I'll create the right pattern for you.

I'm loving this snippet extension more and more :)

--
Isaak Malik
Web Developer

Stan | Tue, 2008-03-18 09:59

I just had a quick look at the code and I don’t think we need the brackets in there. Try this:

tabStopRegEx = new RegExp('«|\%tabstop\:', 'g');
tabStopRegEx = new RegExp('«|\%tabstop\:', 'g');

If the one above doesn’t work for you, yours is just as good and does the work.
Note I haven’t update the KPZ this time

Stan | Thu, 2008-03-20 04:56

I have been looking into ways to integrate this extension with the auto-complete feature, but to no avail so far. Just this piece of code and an idea of how things should work:

svc = Components.classes['@activestate.com/koCodeIntelService;1'].getService(Components.interfaces.koICodeIntelService);
// some_temp_file.php will contain all available triggers as PHP functions or constants
svc.ideEvent('added_file_to_project', 'some_temp_file.php', ko.projects.active.manager.getCurrentProject());
svc = Components.classes['@activestate.com/koCodeIntelService;1'].getService(Components.interfaces.koICodeIntelService);
// some_temp_file.php will contain all available triggers as PHP functions or constants
svc.ideEvent('added_file_to_project', 'some_temp_file.php', ko.projects.active.manager.getCurrentProject());

This is to “simulate” a new file being added to the project so the CodeIntel Service will pick it up and hopefully parse it. I haven’t tested it yet, I am not a Python developer nor have I ever done anything like this before (but it sure is fun).

Are there any Komodo developers out there that might be able to help?

UPDATE
I also came across this, but it doesn’t seem to be working:

koCodeIntelCatalogSvc = Components.classes['@activestate.com/koCodeIntelCatalogsTreeView;1'].getService(Components.interfaces.koICodeIntelCatalogsTreeView);
koCodeIntelCatalogSvc.addPaths(1, ['C:\\snippets.cix']);
koCodeIntelCatalogSvc = Components.classes['@activestate.com/koCodeIntelCatalogsTreeView;1'].getService(Components.interfaces.koICodeIntelCatalogsTreeView);
koCodeIntelCatalogSvc.addPaths(1, ['C:\\snippets.cix']);

Isaak | Tue, 2008-03-18 12:55

The change that I did actually made it work but the pattern seems a bit incompatible, I haven't looked into your code so I'm not sure what the correct pattern would be so I'm going to change it into yours to be on the safe side.

--
Isaak Malik
Web Developer

Isaak | Sun, 2008-04-06 09:12

Hey Stan,

I have a small request if you don't mind, could you change your macro so that snippets are only checked and loaded at Komodo startup? I noticed that your macro checks for snippet updates regularly while Komodo is running, personally I find this unnecessary.

Thank you in advance.
--
Isaak Malik
Web Developer

Stan | Sun, 2008-04-06 16:13

I have converted this macro to an extension and it supports negative values for SNIPPETS_HTTP_RELOAD_INTERVAL (i.e. do not check for updates). By default the extension will also install the snippets.xml file in your home directory and it won’t reload it unless you change it (thanks Eric for introducing me to the koFileNotification service). There are a couple of new features as well such as the ability to pass arguments to templates (e.g. type input.hidden to get a hidden input field or input.file to get a file upload control). I plan to release the extension in the next couple of weeks. I am also working on a XUL front-end that will allow you to edit the snippets in a way similar to the toolbox.
I have also completely replaced the TAB-keyboard handler. In my development/local version I also provide support for tabchoice handlers, i.e. you get a list of predefined values for each TAB-stop. I am also trying to compile a default snippets.xml that will support all standard HTML tags. It’s a lot of work, so as I said above, I will probably release all this in the next couple of weeks or so.

Just a suggestion, go to the toolbox, locate the Snippets macro, right click and select “Edit Macro”. Set the value of SNIPPETS_HTTP_RELOAD_INTERVAL to 24 hours or so. This will disable the automatic update.

Cheers,
Stan

dreftymac | Tue, 2008-04-08 05:56

Just some preliminary questions:

1) How is this line of development an improvement over the other "TextMate" like enhancements by others? (see for example http://blogs.activestate.com/jeffg/2006/08/textmatestyle_m.html) it seems like there are multiple people doing slightly different versions of the same idea, anyone have a side-by-side comparison?

1.5) This seems to be re-inventing stuff made available in the latest release of KomodoEdit. Is this by design or did you start building this before you knew about the latest feature enhancements?

2) I've noticed a lot of snippet engines using XML. This seems like an ok choice but after a while it gets *very* cumbersome in some areas. Have you considered packaging snippets in a full-fledged programming language syntax, (such as Python?) ... [see note]

Thanks for any feedback on these.

----
Note: I've used this approach before and it has huge advantages:
i) automatic support for variable interpolation and advanced string manipulation;
ii) no need to use CDATA escapes to avoid syntax collisions (e.g., with python just use """ or ''' or pep292 Templates or printf or ... you have *many* alternatives)
iii) automatic support for "snippet transclusion" (e.g., if you define each snippet as a function and each "language library" as a class, the language already includes built-in support for cross-referencing from one place to another, no need to re-invent this)
iv) no hassles with XML kludges such as escaping greater-than signs or un-nestable CDATA symbols and other ugly and inelegant stuff
v) you can design snippets to be as simple or as complex as you want, on the simple side you don't even have to know Python (or Ruby or Perl, or whatever native syntax you use)

Stan | Tue, 2008-04-08 06:22

Nice to have some feedback on this, really appreciated.

RE 1 and 1.5) The main reason I am doing this is to have a better tool that extends far beyond the default abbreviation implementation made available in Komodo 4.3 (which is more or less a port of Jeff’s initial trigger macro). As you said, there are already a couple of implementations available in the community forum, but what I am trying to do is to have one “better” that captures all the basics and provides many new features.

RE 2) I am aware the snippets.xml file could grow and with each new addition it only gets harder and harder to maintain. But look at this from another angle – having a single file that you could xcopy over to a server and have it shared among your colleagues and friends is better than dealing with updates or syncing multiple files.

RE i) The snippets.xml file is just a representation of the snippets you would normally put in your toolbox. You still have the ability to use special variables, such as [[%date]] or any other already available in Komodo.
RE ii) I find the use of CDATA sections much easier than dealing with HTML entities. Also new users won’t have to worry about valid XML.
RE iii) Not sure I follow in this one, but if you look at the sample XML file here: http://www.psp-webtech.co.uk/komodo/snippets.xml you will see the [[%include]] command used in a couple of places.
RE iv) See comments below on new improvements.
RE v) I am afraid I don’t understand this one.

You make some good points in your post. Looking back to my previous reply you will see I am trying to overcome some of the, let’s call them, major downsides of this extension. I am working on an UI to allow you to edit your snippets, so no XML skills would be required.

I appreciate your feedback and let’s keep the discussion going : )

Cheers,
Stan

dreftymac | Tue, 2008-04-08 07:50

I'll try to explain a bit more why I think XML is not the best choice for defining snippets.

Nothing speaks like an example, so here is one to illustrate a bit of what I am talking about. Consider how readable this is compared to XML. Also notice that "piggybacking" on a full-fledged programming language allows us to do a bunch of custom things that cannot already be done with XML, and yet there's *nothing* that we can do in XML that we cannot do here.

    ...

    ### begin: define pysnip_entry
        oEntry                = core_pysnip.snipentry()
        oEntry.sigil          =  '___'    ### the user can redefine the 'placeholder' syntax
        oEntry.caption        =  'new blank simpletable';
        oEntry.abbrev         =  'simpletable';
        oEntry.desc           =  """new simpletable for html or xml, with a secret message thrown in""";
        oEntry.keycombo       =  None;
    ### ------------------------------------------
        oEntry.body =r"""
            {table}
                {tr}
                    {td}___cell1{/td}
                    {td}___cell2{/td}
                    {td}___cell3{/td}
                {/tr}
            {/table}

            {p}Secret message: ___{String.rot13('HelloWorld!')} {/p}
        """
    ### ------------------------------------------

### begin_: do cool stuff with snippets
    mysnips = html.getsnips(abbrev.startswith('simple')); ## get more than one
    mysnip  = gui_chooserdialog(mysnips);  ## use a textmate-style chooser
    page    = html.getsnip(name.equals('newdoc')); ## outer enclosing page
    mysnip  = gui_fillintheblanks(mysnip); ## popup a gui dialog box auto-populated with every variable in our snippet
    page.body = mysnip.body;
    page.textarea_paste();

### begin_: do cool stuff with entire snippet libraries
    myexport = core_pysnip.export(libname.match('perl')).to_xml;
    n2export = core_pysnip.export(libname.match('*')).to_textmate_bundle;
    n3export = core_pysnip.export(libname.match('x')).to_yaml;
    core_pysnip.export('*').to_base64encoded().upload_to_website();

    ### NOTE: curly-braces were put in to keep Drupal happy.
    ### Normally we would just use whatever syntax we want
    ### with ZERO ugly syntax hacks or escapes.

    ### NOTE: we have a bunch of snippets that look just like this
    ### one, all defined using python syntax. But you could do it
    ### using ruby or perl quite easily also, as these languages are
    ### all best-of-breed as far as manipulating strings goes.

    ### NOTE: the user can extend this however she wants without "breaking" 
    ### the syntax, add a custom attribute, add custom functions, metadata
    ### whatever. There's no need to reverse engineer the snippet file
    ### format, or processing language because it's all just python.
    ...

Stan | Tue, 2008-04-08 08:11

I love thinking in code, you are absolutely right, nothing speaks better.

I don’t really see the advantages of using Python, Ruby or any other programming language to store the snippets. Whether it is XML, text files or anything else it’s just a container. Nothing stops you from writing layers on top of it, whether they would trigger functions from Python, provide additional parsing and act like plug-ins, etc. I think either way you just need a way to store the snippets. It’s more of a personal choice whether you would stuff snippets in XML and then write plug-ins on top of it to support various new tasks or you are going to rely on Python and what’s already available in Python to do the work. Personally I find XML better and easier. It’s widely used by everyone, even a junior web developer already knows how to write and read XML. It has a clear model of describing chunks of data and is extensible.

Eric from Active State has also raised this issue in his post. At the end of the day nothing works better than a UI on top of, be it XML, Python/Ruby files, the toolbox or whatever. I think we all agree on this one.

Looking forward to your comments,
Stan

dreftymac | Tue, 2008-04-08 10:35

Yeah, pretty much we're in agreement. I guess the primary motivator for me is I've tried both approaches (I've lost count of how many snippet engines i've built on top of editors, in an effort to "textmate-ize" them since before textmate even existed) and the "language-based" approach won hands down simply because XML doesn't really "do" anything by itself, eventually you have to resort to a language to process it, handle includes, handle interpolation, handle escapes, ... yadda yadda ... and before you know it you've re-invented TemplateToolkit, which itself gets processed by a language. (jeez, how many "middlemen" are we inviting to this party?)

I can see how some prefer XML though, I'm just stating some of my rationale for being less favorable towards it, based on prior experiences.

You are, however, absolutely spot-on with the part about UI. I don't know XUL well enough right now, else I'd be helping with coding instead of just commenting, but it looks like you're on the right track.

1) dont force me to memorize key combos
2) dont make me use the mouse to select snippets
3) DO allow me to choose from a list using a "filter as you type" interface
4) DO allow me to assign custom key combos for those instances were I *am* willing to memorize combos
5) DO allow me to easily jump into and out of any language regardless of filetype
6) DO allow me to get a lot of this done without having to take my fingers off the home row
7) DO allow me to easily manage snippets and translate them from different formats, so I can: i)import and export all the stuff I've done from other editors; and ii) use or create a tool that allows me to "convert" into and out of work done by other communities (e.g., mass convert of Textmate bundles and my own python/perl/ruby stuff into and out of Komodo).

This kind of thing is not easy to do (right) which is why I've admired those few editors and IDE's that either "get it" ... or else are easily extensible enough to get out of the way and let you do it right yourself. So far I've seen only one or two that even come close in windows-land.

I'm not sure where Komodo fits in that spectrum, but it does look somewhat promising. Work like what you've done here seems to help push it that much further.

====
UNRELATED: The only other apparent "minus" of Komodo (for me anyway) is it seems way too complicated to add custom syntax coloring for my own filetype. E.g., what if I want to add *very simple* syntax coloring on text files? Luddite looks nice, but there's gotta be an easier way to simply say "make all non-alphanumerics a different color".

Isaak | Fri, 2008-04-11 15:05

I'll be waiting for the new version then, in the meanwhile I'll be using the suggested.

Thanks :)
--
Isaak Malik
Web Developer

Florent V. | Mon, 2008-04-21 04:24

Hello,

A few weeks ago I worked on a HTML+CSS snippet toolbox with sharing (and maybe inclusion in a future Komodo release, who knows?) in mind.

So far it's still incomplete, but I'm completing the set of HTML and CSS snippets. There are no keyboard shortcuts, so you have to activate snippets using the mouse (I know...). I used to have shortcuts, but providing a toolbox with like one hundred keyboard shortcuts that will inevitably clash with user preferences is not that sensible, so I just killed them.

I thought that I would duplicate those snippets (and modify some of them) for use with the Abbreviations mechanism that was included in 3.3. But now I wonder if I should go with this extension instead.
Are there plans to enhance the Abbreviations feature using some of the improvements in this extension, or did the Komodo team decide to keep that feature simple and let people work with extensions for more advances functionality?

Stan | Mon, 2008-04-21 05:25

This is a great question and I would like to take the time to comment on it. If you are looking for a simple snippet triggering facility, then the Abbreviations feature introduced in 4.3 is pretty much all you need. Snippets are organised by language and you have a nice interface (the toolbox) to manage them.
If you are looking for more functionality then this extensions comes to the rescue. I don’t want to repeat myself, but the list of planned features is pretty much listed in this thread so if you are interested I encourage you to have a look at the discussion above.

So think about it, do you need more out of the facility already available in Komodo? Does it work for you? Do you need any of the features this extension provides? These questions should pretty much guide you in the right direction.

Hope this helps.

Florent V. | Mon, 2008-04-21 06:45

Thank you Stan for this answer.

I will need some time to review both your macro and the built-in feature. Personally I quite like the way your macro works, It's powerful but doesn't feel bloated or hard to handle (speaking of the snippet file format... I'm not a developer and I have limited knowledge of XML, but it's mostyly clear to me). It has great functionality, including the drop-down list that enables you to choose from various options and the way some snippets (such as the unordered list one in your example) behave.

Do you intend to create a XPI extension once it's more polished?
For my private use, I would be glad with using your macro as it is right now. But my goal is to contribute a great HTML/CSS toolbox as well, so if it relies on an easy-to-install extension (or if it's included in it ;)), that's better. :)

...

Answering my question above: ok, you're already working on an extension. :)

“I am also trying to compile a default snippets.xml that will support all standard HTML tags. It’s a lot of work, so as I said above, I will probably release all this in the next couple of weeks or so.”

Need help with that?

If you have some time, check out the toolbox I shared (link in my previous post). It doesn't have all of the 91 standard HTML 4.01 tags, but it's a start. It also has some page templates that might be useful. Then it has some links to online ressources, and examples of code, that I don't intend to convert to Abbreviations or Snippets.

By the way, I think you should consider (if you didn't already ;)) support of multiple Snippets files. What if you have your own Snippets file, and want to add snippets for a different language your learning, or combine your own snippets and some snippets shared by the whole team?

Stan | Tue, 2008-04-29 03:09

Sorry for my late reply, I have been quite busy at work recently. Yes, I intend to package this macro as an extension. I am currently reworking the macro; in fact I have started from scratch and I am building a very modular snippet engine.
Your post got me thinking. It’s really powerful to have a programming language behind the snippets. What I am currently trying to accomplish it to have a set of “triggers” that behave in the same manner, but are configured differently. So for example you can have your snippets in an XML file (called Container). You can also have a JavaScript file that would take a snippet such as this one “table/head/body2x2/foot” and would output a 2x2 table with header and footer. Additionally you can have triggers that would work like GhostDoc for Visual Studio. Currently I am far away from having this feature in the engine, but the idea is to be able to have phpDoc and JDoc blocks auto-generated based on the signature of a method/function.

I would be quite happy to have a look at your current work. Please send over any HTML/CSS snippets you’ve got to insaned [ at ] abv [ dot ] bg.

tdegrunt | Wed, 2008-06-25 02:47

Is it possible to create more complex snippets like below, whereby there's nesting:

<%= f.submit [[%tabstop:"Submit"]] [[%tabstop:, :disable_with => '[[%tabstop:Submitting]]']] %>

This in analogy to TextMate's:
${TM_RAILS_TEMPLATE_START_RUBY_EXPR}f.submit "${1:Submit}"${2:, :disable_with => '${3:$1ing...}'}${TM_RAILS_TEMPLATE_END_RUBY_EXPR}

TextMate's even more complex, namely it'll replicate whatever you type at ${1:Sumbit} at ${3:$1ing...}.

roamzero | Mon, 2008-07-07 01:59

There is this weird issue where if you ctrl-z out of the snippet it caused the extension to stop working and act buggy for a bit before you can invoke it again. Is there a way to fix this?

Stan | Mon, 2008-07-07 02:24

@tdegrunt: Unfortunately nesting tabstops is not supported. I plan to include this support in future versions, however I can’t commit to any dates as things are getting quite complicated and frankly I don’t get much spare time :(

@roamzero: I am aware there are some issues when using the Undo command. Unfortunately at this point the snippet engine does not monitor the state of the buffer. To get the snippets running again, hit TAB as many times as the number of tabstops you’ve got in your latest snippet. Sorry.

Cheers,
Stan

-->