And here is part 2!

After my disappointment with XQuery, I looked at a template engine for Python. While Python can do some basic templating itself, Cheetah is much more powerful. So here’s an example for a Cheetah template:
(more…)

Anyone who has looked at a Scribus preferences file (scribus.rc) will be aware that the sheer amount of settings is daunting. There’s a C++ file named prefsstructs.h that defines on 470 lines of code 29 data structures too hold various settings. And these are just dumb structs, without any code to read/write to XML (or even getters/setters or default values or scripter support). So how can we clean up this mess?
(more…)

You know how it is, idly chatting in #scribus, compiling a little, surfing here and there… and suddenly you want to know the distance from Montreal (site of LGM2 ) to Victoria (site of FOSS4G 2007 ). Nothing as easy as that, just ask Google Maps for directions and it’ll give you the km number for free. Unfortunately there are quite a lot of Victorias around the world, so you have to choose the correct one (in British Columbia). But seeing other locations listed, from Texas to Australia, you obviously try some more, just for the fun of it (maps.google.com sadly only offers destinations on the North-American continent). Ok, no route to Victoria, Australia, nor to Victora, Seychellen — not really surprising.
The distance to Victoria, Texas is even shorter than to Victoria, BC, that’s a (small) surprise. But now: Google Maps Deutschland does have a route to Victoria, UK. Yippee!

google-map-montreal-victoria.png(click to enlarge)

Of course you wonder how to take the longer, middle leg of this journey, but — don’t fear! — Google Maps Deutschland has a solution in its itinerary for that, too. Read for yourself. 😀

/Andreas

P.S. And here’s what we really need: the route from Lyon to Montreal (this time in English language):
http://maps.google.fr/maps?f=d&hl=en&saddr=Lyon&daddr=Montreal

One of the oldest feature requests for Scribus have been character styles and hierarchical styles. These are now finally implemented in Scribus 1.3.4cvs. Also Tsoots has done some good work on the new Style Manager, which replaces the old Paragraph Styles dialog, and he will hopefully describe it in a blog entry soon. I’ll take some time here to explain the new concepts behind styles.
Before 1.3.4 there were only named styles for paragraphs and direct fromatting for character styles. Now there are named styles for both paragraph styles and charstyles; and also direct formatting for paragraph styles and charstyles (Yes, this means you finally don’t have to define a new paragraph style if you just want to change the margins for one paragraph).

In the following I’ll describe the logic behind hierarchical styles as it is implemented in Scribus 1.3.4. Paragraph styles, charstyles and line styles behave much the same in this respect. Also other styles which are implemented in the future will share this behaviour. I’ll use paragraph styles as an example in this explanation.

What does “hierarchical” style mean? It means that each style does not just define a bunch of attributes, but also has a flag for each attribute which tells if it is defined in this style or inherited. So consider a style A has as parent a style B:

Style A Style B
parent: “B” parent: “”
font: “Courier” defined font: “Palatino” defined
fontsize: “12” inherited fontsize: “12” defined
color: “black” inherited color: “black” defined

If you now format some text with style A, it will take the attributes (“Courier”, 12, black). If you change the fontsize of style B, then style A will automatically pick up the new fontsize. On the other hand, changing the font from “Palatino” to “Times” will not affect style A.
The new stylemanager displays attributes in bold if they are defined locally and in regular font if they are inherited.
It’s also possible to have chains of inheritance, eg. A has parent B, B has parent C, and so on. Only cycles are not allowed.

So what about direct formatting? That is implemented as unnamed styles. I also call those “physical styles” and the named ones “logical styles”.
Physical styles are attached to some content, eg. a run of characters, a paragraph or a story (that’s the combined text of a chain of textframes).
A physical style can have a logical style as a parent and then inherits all attributes which are not set directly.

There’s not only a logical hierarchy but also a physical hierarchy of styles. The physical hierarchy depends on the nesting of elements, so a physical style will pick its attributes from character run, then paragraph, then story, then document. See figure for how those contexts are nested.

Figure: Nesting of contexts

nesting of contexts

So how do I know which attribute value ends up in the style I use? Well, it follows these rules:

1. if the attribute is locally defined (not inherited) in this style, the local value is taken.
2. if the style has a named style as parent, the attribute is searched in this named style(yellow line). If neither the named style nor its ancestors (red lines) define the attribute, go on with 3.
3. the attribute is searched in the next physical style in the hierarchy (follow blue lines). If it’s not defined in any of those styles, go to 4.
4. Use a compiled-in default value.

The documents default style will define all attributes, so rule 4. is never reached.

Ok, that’s all. Complicated? Probably, but I still hope you like it 🙂

You might know that being an Open Source developer in general, and a Scribus developer in particular, allows you to travel all over the world, meet interesting people and have a nice dinner with them. This time I attended the GNOME Live! summit in Boston at the MIT. Thanks to Behdad Esfabod from Pango and Ed Trager from Unifont it also hosted the first FLOSS Text Layout Summit this year; and that was the reason for me being there (not  to prepare for porting Scribus to GNOME and not  for a tea party 🙂 ). Also attending where people from Qt, KWord, fontconfig, SuSE, DejaVu fonts, and SIL. So what was this about? You probably all had your share of font related problems, so it should be no surprise we talked about that — a little. The major part was concerned with international text layout. Ed Trager wrote a nice introduction on what this is about. If you are only familiar with Latin script, read in this article all about writing directions and how characters change their shape according to context. International layout usually includes the following steps:

  1. Itemize — find runs of characters which have the same script and font
  2. Shape — apply script specific substitutions and positioning
  3. Find linebreaks — most applications use the simple and straightforward approach, but Scribus will do something more sophisticated soon.
  4. Justify lines — by extending spaces, scaling glyphs, substituting glyphs or by script specific means, e.g. Kashida in Arabic.
  5. Render glyphs — that’s easy once you have all glyphs with their exact position and size.

Currently there exist three different libraries for international text layout (Pango, Qt and ICU) which all have their pros and cons. At the text layout summit we found out that we all share the common vision of a unified library. Why is that? Well, first of all, writing such a library is hard. Also you need a lot of people to check the results for each existing script. Second, there are still many scripts which are not supported yet, and the hope is to support scripts faster if we combine our efforts. Finally, users don’t appreciate if their native language works in some applications but not in others, just because they use different libraries. You can see this within Scribus if you compare for example Arabic text in StoryEditor and on canvas (The SE uses Qt and displays Arabic correctly).
But we also reached some decisions. As a first step we will write a unified shaping engine. I was surprised how fast the Qt and Pango developers agreed on a common datastructure for such a library (well, that’s only tentative, but I expected some long discussions there). Trolltech said they would consider relicensing their shaping code in Qt so it could be included in a common library. Finally we decided to develop the unified shaping engine on top of HarfBuzz. HarfBuzz is a library which handles OpenType tables and is already used by both Qt and Pango. The unified shaping engine will also be called HarfBuzz and will initially support OpenType fonts. Legacy Type1 and TTF fonts can be used for basic shaping in Latin script and for Arabic if they contain the necessary Arabic presentation forms. Later we also want to integrate Graphite, a new open source font technology from SIL, and platform specific glue code for AAT (Macintosh) and UniScribe (Windows).

Ah yes, in the future Scribus will use this unified shaping engine directly. KWord decided on using Qt, and most other applications will also use either Qt or Pango. There was also some interest from FireFox, OpenOffice, Inkscape and Abiword developers, but so far I can’t tell if they would use the new unified shaper directly or not.

Well, what else? We have a combined planning/agenda/minutes webpage. There are also links to some slides and audio recordings of most talks.

The organisation of the summit was not as good as LGM, though. Ok, there was free lunch (American pizza for all), but we had trouble to use a projector, had to organize audio recording ourselves, could only find (near dry) green and yellow pens for the whiteboard and worst of all: no coffee! At first I hoped I had discovered a very geeky coffee machine in one corner of the MIT media lab, but that was just a liquid nitrogen tank. 🙁

But all in all, very nice people, very geeky setting, very good dining, and interesting and productive talks. And I had ExScite as my private sightseeing guide on my last day, thanks ExScite! 🙂

It’s a common misconception that a font is a bunch of characters. That’s not so. Fonts (or, as Adobe likes to call them, font programs) contain glyphs. Glyphs are the shapes which are painted to screen or on paper. Characters are more abstract and convey some meaning, eg. the latin character ‘A’ or the diigit ‘7’. Even a single font can have more than one glyph for those characters, eg. a swash ‘A’ or an oldstyle numeral ‘7’.

Until now Scribus had the same sloppyness regarding characters and glyphs. Internally Scribus stores Unicode characters and for output it used to map each character to exactly one glyph. I’ve started to change that, though. Here are some use cases where it might be necessary to do something else:

  • using different glyphs depending on style, eg. SmallCaps or OldStyleNumerals
  • mapping the pagenumber placeholder to several digits
  • smart hyphens don’t have an associated character stored with them
  • ligatures
  • …

So here’s the new scheme:

Scribus still stores a sequence of Unicode characters. For output, two more sequences are generated: a sequence of items and a sequence of glyphs. The items provide a glue between the unicode chars and the glyphs. The smallest possible item is a cluster, that is a character and all its marks, e.g. ‘a’ + ‘^’. This would be connected to the glyph ‘â’. The other way is also possible: the unicode ‘â’ is mapped to two glyphs ‘a’ and ‘^’ which are positioned in just the right way. Usually an item will correspond to a syllable, though. The details of items are only relevant to Scribus’s layout engine, so I won’t go into that here. Just keep in mind that items are used to break the 1:1 correspondence between characters (what you type) and glyphs (what you see).

The item and glyph sequences are created during layout. At the same time the exact position and size of each glyph is calculated. A sideeffect of this is that Scribus now uses the same data for screen, PS and PDF output (before that the PS and PDF output routines did part of the layout themselves — bad idea).

The current code is still limited. Currently it’s only used for hyphens and pagenumbers. The next step will be to use the new NLS data structures and take advantage of OpenType features. So stay tuned: true smallcaps, ligatures and oldstyle figures are on their way…

Two days ago I filed issue #4000 on Scribus’s bug tracker. 🙂
I also wanted to file #3000 but mrdocs beat me on that this January.
Ugh? Another 1000 bugs in merely six months??

Yep. This shows that the Scribus project is not only active at the development front but also has people who care about its quality and keep track of issues. Fortunately, most of these 1000 new bugs are already closed:

scribus-bugs-cummulative
Another diagram shows that the majority of issues are feature requests or minor issues:


Currently there are only 116 unresolved issues with severity major, block or crash. So what happens to issues which get resolved? While we also have resolutions like “no change required” and “won’t fix”, most issues get fixed:

Which shows we are not only working hard, but we’ve also got the statistics to prove it! 😉 For those who are interested in more statistics, they should check out the “Summary” and “Advanced summary” sections at http://bugs.scribus.net.
Keep on feeding us those bugs, we love it!

/Andreas

Well, that might depend on how you do it. Using DOM is quite easy, although it might be a bit tedious if you have to traverse the whole DOM tree to create your application specific data structure (and of course it’s slow). Using SAX is faster, but programming all those handlers can be just as tedious. In any case you’ll likely have a lot of code which looks like this:

if (tagname == “SCRIBUSXML”) {
   doc = new ScribusDoc();
}
else if (tagname == “PAGE”) {
   doc.addPage(new Page());
}
else if (tagname == “STYLE”) {
   …

Not nice at all. Of course Java developers will smirk now and say “that’s no problem for us; we can use Jakarta Commons Digester.” The curious C++ programmer might check out their webpage or the introductory article on JavaWorld and get envious. Really envious.

An example program with Digester might look like this:

Digester digester = new Digester();
digester.setValidating( false );

digester.addObjectCreate(“SCRIBUSXML”, ScribusDoc.class);

digester.addObjectCreate(“STYLE”, ParagraphStyle.class);
digester.addSetNext(“STYLE”, “addStyle”);

digester.addObjectCreate(“PAGE”, Page.class);
digester.addSetNext(“PAGE”, “addPage”);
digester.addSetProperties(“PAGE”, “NUM”, “number”);

ScribusDoc result =
   (ScribusDoc) digester.parse( “example.sla” );

So instead of programming all those tedious if-cases, you just add rules which tell
Digester what element tags and attribute names to look for and what to do if found: create new objects, call setter methods, or do some other stuff. Then you call the parse method and at the correct times the digester uses your rules to create your datastructure. At the end you just query the root object and there you go.

Unfortunately, my current pet projects are hooked on C++, so I tried to achieve a similar solution with C++ templates. This is what I came up with:

Digester digester = new Digester();

digester.addRule(“SCRIBUSXML”, Factory<ScribusDoc>());

digester.addRule(“STYLE”, Factory<ParagraphStyle>());
digester.addRule(“STYLE”, Setter( & ScribusDoc::addStyle ));

digester.addRule(“PAGE”, Factory<Page>());
digester.addRule(“PAGE”, Setter( & ScribusDoc::addPage ));
digester.addRule(“PAGE”,
    SetAttribute( & Page::setNumber, “NUM” ));

digester.parse(“example.sla”);
ScribusDoc result = digester.result<ScribusDoc>();

Do you like it? Me too. 🙂
The initial version of the library is available in current CVS in the “desaxe” directory. I’ll add some documentation and examples soon.

Some internals:
The deSAXe library uses a SAX parser which reads the XML document and creates events each time a start tag, end tag or a string of characters is read.
Setter/Factory/SetAttribute are examples of Actions. When a rule fires, the SAX events are forwarded to the corresponding Action. If more than one rule fires, all Actions reveive the events in the order they were added to the digester. Internally deSAXe uses a finite automaton to keep track on which rules are active, so it’s pretty fast (unlike Jakarta Commons Digester 🙂 ).

Looking forward to your comments (please use the mailing list since atm comments are disabled here).

/Andreas

P.S. Yes, I do enjoy parsing XML, now! 🙂
P.P.S. Apart from the pun on SAX the library is named after the coolest Scribus user in Australia, Elaine de Saxe. 🙂 Hi Elaine!

And a “Thank you, Peter” for the great work you are doing for Scribus. If there are two things most software developers (and FOSS developers in particular) really don’t like to do, it’s writing documentation and regression testing. The Scribus project is very lucky to have Peter, who does not only do a fantastic job with both, but also is an excellent promoter for the Scribus case, and — last not least — provides invaluable background information on the pre-press business to us less experienced developers.

So, a great round of cheers for Peter! We’re all looking forward to another exciting year of team work together with him.

Last week I implemented “optical margins” in 1.3.4cvs. So I’ll use my first blog entry to describe what that’s all about.

When you look at a newspaper, or a Scribus document with justfied text, you might notice that the left margin looks much straighter than the right one, even if all chars line up geometrically. That’s because most characters line up nicely at the beginning of a line, but a lot of characters leave an uneven appearance at the end of the line. Most notorious are punctuation characters: hyphen, comma, dot, quotation mark, and others.

An old technique to repair this is called “hanging punctuation”. This moves the offending punctuation marks into the margin. Unfortunately, this doesn’t look quite right (you can verify this if you activate that option in InDesign). The solution is to move the punctuation marks only to 50%-70% into the margin. This is how it works in Scribus 1.3.4cvs (PDF, sla). I copied the exact values from Hàn Thê´ Thành’s PhD thesis “Microtypographic extensions to the TEX typesetting system”. He implemented this technique in pdftex.

But also characters like ‘f’ and ‘r’ contribute to the uneven appearance of right margins. Moving those characters (very slightly) into the margin is called “margin kerning”. Since characters might have very different shapes in different fonts, I don’t use fixed percentages to move those into the margin. Instead I use a trick: I use the font’s kerning information for the pairs “f.”, “r.”, and so on. Maybe in the future font designers will include information for margin kerning explicitly in their fonts, but for now I’m quite content with my implementation. I only use margin kerning for the right margin, though. The left margin is usually quite smooth, except for characters like ‘A’, ‘W’ and ‘V’.

Currently there’s no option to deactivate optical margins in Scribus. I’ll add that later, and probably also options to finetune it (change the percentages for hanging punctuation per font, activate margin kerning for the left margin).

So tell me what you think about it and how I can improve it further.

Is that all? No. First, I fixed a bug in Scribus 1.3.3 which messed up right margins by placing hyphens irregularly. Hey, all that work with optical margins is for nothing if Scribus places hyphens randomly anyway! 😉
Then, Hàn Thê´ Thành describes another technique called “glyph extension”. This stretches or shrinks characters by -2% to +2% in a line, in order to avoid large white spaces between the words. This glyph extension won’t be visible to the naked eye but will help to get a more even word spacing.
Now, speaking of word spacing, isn’t TEX supposed to have a much better algorithm for linebreaking, which was also adapted to InDesign? Yep. TEX and InDesign optimize linebreaks for a whole paragraph. That results in a much more even word spacing than is achieved by the straight forward algorithm used by Scribus and other layout programs. Used to be used by Scribus, I should say, since we have a prototype of a new text layout system which supports this optimisation. Currently, the prototype is lacking some standard features of the old layout system (like non-justified alignment, tabs, drop caps, and some linespacing options). Once this is done we’ll merge it into cvs, so there’s something you can look forward to!

Ciao for now
/Andreas