Under Cover

And here is part 2!

After my disappointment with XQuery, I looked at a template engine for Python. While Python can do some basic templating itself, Cheetah is much more powerful. So here’s an example for a Cheetah template:

Anyone who has looked at a Scribus preferences file (scribus.rc) will be aware that the sheer amount of settings is daunting. There’s a C++ file named prefsstructs.h that defines on 470 lines of code 29 data structures too hold various settings. And these are just dumb structs, without any code to read/write to XML (or even getters/setters or default values or scripter support). So how can we clean up this mess?

While we recently released into the wild, one of the tools we used for the first time is the new OpenSuse Build Server. So why the rant ?

In a word it is terrific. πŸ™‚ It still is in “beta” phase and there are features to be added. However, it is a major helper for time stretched developers to package their application in a sane way and then automagically push them to rotating mirrors worldwide without a lot of fuss.

It is all fine and well if you write the best code in the world if no one can use it. And that means packaging. For Linux, that means creating a .deb or creating rpm packages which many Linux distributions use.

The magic is one can upload the source and a spec file which is a recipe to build rpms. Select which platforms to build, push a button and voila. In an hour or two you have all your packages built in a carefully controlled manner. This allowed us to supply for all Suse releases 9.3+ including the Enterprise Desktop 10 for the first time. Even better for users, they can use what ever tool be it Yast, Smart or Yum to add these repositories to get the latest greatest Scribus with all dependencies automatically satisfied. Push button packaging for users too πŸ™‚

In part two, I’ll explain the master plan and I’ll rant a bit more about why this is all a Good Thing β„’

So a big thanks to darix, adrianS and the Build Server team for making such a nice tool and allowing us to play.

We rant and rave about Scribus’ PDF quality and its ability to make commercial grade PDF by mere mortals. A design decision made early on to focus on robust PDF export was in hindsight a brilliant one by Franz. OK, now that is one side of the PDF coin. The other side is parsing and editing PDF. A task even more complex in some cases and in no case trivial.

We often get asked when Scribus will edit PDF and if there are tools which will do it easily and are open source. The first answer is eventually – but it will not be easy, given the team’s collective knowhow about PDF. The second is nothing open source beyond some minor manipulation like splitting, merging and some imposition. And while Acrobat Professional can do some limited PDF editing, it cannot do everything. PDF editing in the way most people perceive is the realm of pre-press specialist Acrobat plugins which are expensive and require some knowhow of pre-press and commercial printing. They also require Acrobat Professional.

Today, I had a chance to get a cvs build of PoDoFo and its GUI browser running which uses cmake and Qt4. PoDoFo is the first serious attempt to write an open source pdf library and tools which can both parse correctly the internals of a PDF and edit PDF. πŸ™‚


Podofo Browser walking through the internals of a Scribus created PDF

Above is the PoDoFo browser tool. To the untrained eye, you’d say “So what?” To the trained eye what you see is the ability to parse, select and even edit the contents of a PDF. Things like extracting and replacing images. Or eventually re-embedding fonts. Or editing the text within the PDF. Or …
So what it is about PDF that makes it so difficult to edit? Simply parsing PDF correctly is difficult enough. One, it is a non-linear format. You can put the fonts after the text and put the images above them if you want. Content is embedded in what are known as streams and one needs to decipher on the fly the type of stream it is, how it is compressed and where it belongs in the file, along with things like how it is displayed. Oh and objects are reusable too. You cannot simply fire up your preferred editor and start hacking.
PDF was always meant to be “Electronic paper” if you will. In final finished form the idea is it will display the same everywhere no matter what platform or device. (You can display PDF in a Palm Pilot easily.) It was never ever meant to be edited once created.
As I said, not trivial. Then, even if you figure all this out, then you need to reassemble all these bits in a PDF coherent way which won’t break when reloaded in a viewer.

So, PoDoFo is the first real open source PDF editor. Plus, it comes with some other very useful tools for PDF. It will be in the future I hope a PDF library to allow lots of applications to create high quality PDF with the same kind of simplicity and robustness Scribus has now. The key thing is now it is able to get to the much of the low-level bits inside of a PDF. From this lots of magic will come in the future I am sure.

Ready to do some surgery on a PDF ?

It is not uncommon that when a program like Scribus grows so quickly and gets so many features added in such a short amount of time that some cleaning is necessary now and then. Throughout the 1.3.x series we have cleaned a lot of code up, fixed a lot of bugs uncovered by the cleaning, as well as continuing to improve the whole internal view of things.

So far in the 1.3.4cvs timeframe I have not had a lot of spare time as I started a new job (yay!) so the major rewrites of the 1.3.0->1.3.2 that I was able to do before were put on the backburner. In this time Andreas has been going nuts with the text system rewrite.. amazing stuff is happening there. So, in my spare time I’ve been testing, discussing etc but that gets a little boring after awhile, so this week I decided to remove one of the bugbears I’ve been hanging to do for a long time. That is, remove the global extern ScribusMainWindow* ScMW and then fix up all the resulting mess. This was used for easy access to the main window and its code in a previous lifetime where a lot of the main document code was held within the main window itself as well as a general easy (read nasty) way of finding your way back to the main application.

So, having just committed updates to about 90% of the Scribus files in 1.3.4cvs :), what does this aim to achieve? Primarily, moving the bulk of access down a layer into the document. Theres still a lot of leftover code that now just goes doc->scMW() however once this is eliminated we can start to run more operations without a GUI (read command line) or with multiple GUIs (read SDI mode). Hopefully I removed all the uninitiated pointers around and added ones where necessary :). Still works ok here, but we never said the changes in 1.3.4cvs were not going to get a little bumpy at times :). Heres to a big change, that took a week of spare hours at nights. On ya guys.. 1.3.4 will be a major step forward with all the cool transparency stuff and text and the to-come style manager from Riku and.. and.. and…

Well, that might depend on how you do it. Using DOM is quite easy, although it might be a bit tedious if you have to traverse the whole DOM tree to create your application specific data structure (and of course it’s slow). Using SAX is faster, but programming all those handlers can be just as tedious. In any case you’ll likely have a lot of code which looks like this:

if (tagname == “SCRIBUSXML”) {
   doc = new ScribusDoc();
else if (tagname == “PAGE”) {
   doc.addPage(new Page());
else if (tagname == “STYLE”) {

Not nice at all. Of course Java developers will smirk now and say “that’s no problem for us; we can use Jakarta Commons Digester.” The curious C++ programmer might check out their webpage or the introductory article on JavaWorld and get envious. Really envious.

An example program with Digester might look like this:

Digester digester = new Digester();
digester.setValidating( false );

digester.addObjectCreate(“SCRIBUSXML”, ScribusDoc.class);

digester.addObjectCreate(“STYLE”, ParagraphStyle.class);
digester.addSetNext(“STYLE”, “addStyle”);

digester.addObjectCreate(“PAGE”, Page.class);
digester.addSetNext(“PAGE”, “addPage”);
digester.addSetProperties(“PAGE”, “NUM”, “number”);

ScribusDoc result =
   (ScribusDoc) digester.parse( “example.sla” );

So instead of programming all those tedious if-cases, you just add rules which tell
Digester what element tags and attribute names to look for and what to do if found: create new objects, call setter methods, or do some other stuff. Then you call the parse method and at the correct times the digester uses your rules to create your datastructure. At the end you just query the root object and there you go.

Unfortunately, my current pet projects are hooked on C++, so I tried to achieve a similar solution with C++ templates. This is what I came up with:

Digester digester = new Digester();

digester.addRule(“SCRIBUSXML”, Factory<ScribusDoc>());

digester.addRule(“STYLE”, Factory<ParagraphStyle>());
digester.addRule(“STYLE”, Setter( & ScribusDoc::addStyle ));

digester.addRule(“PAGE”, Factory<Page>());
digester.addRule(“PAGE”, Setter( & ScribusDoc::addPage ));
    SetAttribute( & Page::setNumber, “NUM” ));

ScribusDoc result = digester.result<ScribusDoc>();

Do you like it? Me too. πŸ™‚
The initial version of the library is available in current CVS in the “desaxe” directory. I’ll add some documentation and examples soon.

Some internals:
The deSAXe library uses a SAX parser which reads the XML document and creates events each time a start tag, end tag or a string of characters is read.
Setter/Factory/SetAttribute are examples of Actions. When a rule fires, the SAX events are forwarded to the corresponding Action. If more than one rule fires, all Actions reveive the events in the order they were added to the digester. Internally deSAXe uses a finite automaton to keep track on which rules are active, so it’s pretty fast (unlike Jakarta Commons Digester πŸ™‚ ).

Looking forward to your comments (please use the mailing list since atm comments are disabled here).


P.S. Yes, I do enjoy parsing XML, now! πŸ™‚
P.P.S. Apart from the pun on SAX the library is named after the coolest Scribus user in Australia, Elaine de Saxe. πŸ™‚ Hi Elaine!

Finally, loading of 0.x, 1.x, 1.2.x and 1.3.x=>1.3.3 file formats is now coded with the FileLoader plugin. Work can begin fairly quickly on the file format code for 1.3.4, OR at least which was my main urge to get this code done: massive format changes for Andreas and Riku to support inherited text styles, Unicode reading and writing instead of control characters, etc etc. This will mean that we will no longer need to safely change 1.3.3 loading code.. its handled separately now. To do: File saving of 1.3 docs via the plugins, and.. if anyone got adventurous, saving back to 1.2 format.

Update: Saving of docs to 1.3.x format is done.
Update: Created a new 1.3.4 plugin for loading and saving. This is now the default plugin used.

After much wailing, gnashing of teeth and general procrastination by many.. today’s show is brought to you by the letters w, o, r, k, i, n and g! The aging (read, since 1.3.1) FileLoader plugin system can finally load a Scribus 1.2.x document without the AGES old scribusXml.cpp anymore. All very technical for those that just want to do publishing, however what it does mean is that once the work is complete, we can bring you a cleaner loading and saving system. Fewer, or preferably no modifications into old loader code for old format docs when adding new features to the new formats. Most importantly, this means that finally we can begin the 1.3.4+ format to handle the new text features. I think that can start next week.