Text & Fonts


Now that 1.3.5(1) is released, development has regained speed. Here’s a short overview of what has happened recently in 1.5svn and Scribus in general:

  • Scribus founder Franz Schmid returned from his well-deserved break with a big bang, the latter being experimental code to embed 3D objects in Scribus-created PDFs.
  • Franz also wrote import filters for Calamus Vector Graphics (CVG), WordPerfect Graphics (WPG) and Apple PICT files. The quality of those filters is already superb. Along the way he also created an import filter for colour palettes from Viva Designer.
  • The code of a successful GSoC 2009 project has finally found its way into 1.5svn, namely export to PDF/X-1a and X4.
  • The first part of another student project, an import filter for QuarkXPress Tags, has also been finished, and we expect to add the filter to trunk soon. The student, Jain Basil, is now working on a second import filter for InDesign’s IDML format and hopes to tackle the older InDesign INX format, as well as Xtags, later.

We are also glad that the release of 1.3.5(1) has found an overwhelmingly positive reception in the news outlets, from the Free Software press to magazines for professional publishers.

Finally, we are glad that more companies decided to support our project:

  • The German colour vendor Marabu provided us with colour data for the company’s colour sets, which have been exclusively measured under different lighting conditions for Scribus.
  • dtp studio Oldenburg, another German company, which is in the business of measuring, creating and commercially distributing colour palettes, colour software and colour atlases, donated a huge chunk of colour palettes and will also officially support Scribus, which means that Scribus users will be able to buy additional palettes that work with Scribus from dtp studio.
  • There’s also good news for those who use Scribus on OS/2 or eComStation, as Serenity Systems donated free (as in gratis) licenses of eComStation to the Scribus Team to help with porting Scribus and Fontmatrix to this platform.

And let’s not forget to mention the operator of Vector Portal who agreed to help us with the creation of attractive templates for business cards and presentation templates. Lest we forget: We’re also working on other sets of templates for magazines, brochures etc.

Does this sound good? There’s more to come, stay tuned!

One of the things which is perceived as a stumbling block to using Linux as a designer’s platform is the lack of font management tools. Most graphics professionals started or at some time worked on a Mac and often enough MacOS 9 or earlier.

So, while font management is nowhere as difficult on Linux as it might be assumed since the arrival of Keith Packard’s fontconfig, it is a perception thing. Moreover, KDE3 for a while, now has a simple and very effective font manager/installer called Kfontinstaller. I have commented elsewhere that I think it is the simplest and most user GUI friendly font manager on any platform. It is a font manager with no extra bells and whistles. To boot, Linux can run with lots of fonts installed with no large performance hit – unlike earlier graphics apps and older Windows versions.

Getting back to Macs, and in my experience with production workstations on Win32 , a 100% certainty was the need for a third party font manager. Most of the time either Suitcase or ATM sufficed. They made it manageable to use and select from more than a few hundred fonts.

Plus, fonts on pre-OSX Macs had a nasty tendency to corrupt themselves every once and a while. Most of the older font managers had tools to try and fix this too. Something which one should never worry about happening on Linux. 😉 As you can imagine over time doing layout professionally you tended to end up with hundreds if not thousands of fonts installed. Not good for performance and a nightmare to manage without third party tools.

Given all that, one can see why graphics and layout folks are asking: “Where is the font manager ?” In a sign of the increasing maturity of open source as a graphics platform, there is now a real font manager built with the kind of features and abilities graphic designers, layout pros. along with others have felt necessary, but modernized with some new touches. Introducing Fontmatrix. 🙂

fontmatrix1.png

One thing where Fontmatrix has the older proprietary managers beat is the concept of ‘tagging’. This makes it really nice to group fonts and even sub-group them logically for use in say a book. It also has extensive support to give you a nice gui to show all the glyphs in a font, previews of sample text, variable sizing and also tell you what kinds of advanced Open Type features are inside the guts of your font. That is something I’ve never seen outside of a font editor. And not least, it creates a nice PDF catalogue of your fonts for printing or reference. All in all, a real pro’s font manager, but still nice and user friendly.
Fontmatrix is still young, but it does not crash for me and it works really well already. For installing ease, I have packaged it for Suse 10.2+ and Fedora 8 on the OpenSuse Build Service which I have talked about before. Enjoy.

One of the oldest feature requests for Scribus have been character styles and hierarchical styles. These are now finally implemented in Scribus 1.3.4cvs. Also Tsoots has done some good work on the new Style Manager, which replaces the old Paragraph Styles dialog, and he will hopefully describe it in a blog entry soon. I’ll take some time here to explain the new concepts behind styles.
Before 1.3.4 there were only named styles for paragraphs and direct fromatting for character styles. Now there are named styles for both paragraph styles and charstyles; and also direct formatting for paragraph styles and charstyles (Yes, this means you finally don’t have to define a new paragraph style if you just want to change the margins for one paragraph).

In the following I’ll describe the logic behind hierarchical styles as it is implemented in Scribus 1.3.4. Paragraph styles, charstyles and line styles behave much the same in this respect. Also other styles which are implemented in the future will share this behaviour. I’ll use paragraph styles as an example in this explanation.

What does “hierarchical” style mean? It means that each style does not just define a bunch of attributes, but also has a flag for each attribute which tells if it is defined in this style or inherited. So consider a style A has as parent a style B:

Style A Style B
parent: “B” parent: “”
font: “Courier” defined font: “Palatino” defined
fontsize: “12” inherited fontsize: “12” defined
color: “black” inherited color: “black” defined

If you now format some text with style A, it will take the attributes (“Courier”, 12, black). If you change the fontsize of style B, then style A will automatically pick up the new fontsize. On the other hand, changing the font from “Palatino” to “Times” will not affect style A.
The new stylemanager displays attributes in bold if they are defined locally and in regular font if they are inherited.
It’s also possible to have chains of inheritance, eg. A has parent B, B has parent C, and so on. Only cycles are not allowed.

So what about direct formatting? That is implemented as unnamed styles. I also call those “physical styles” and the named ones “logical styles”.
Physical styles are attached to some content, eg. a run of characters, a paragraph or a story (that’s the combined text of a chain of textframes).
A physical style can have a logical style as a parent and then inherits all attributes which are not set directly.

There’s not only a logical hierarchy but also a physical hierarchy of styles. The physical hierarchy depends on the nesting of elements, so a physical style will pick its attributes from character run, then paragraph, then story, then document. See figure for how those contexts are nested.

Figure: Nesting of contexts

nesting of contexts

So how do I know which attribute value ends up in the style I use? Well, it follows these rules:

1. if the attribute is locally defined (not inherited) in this style, the local value is taken.
2. if the style has a named style as parent, the attribute is searched in this named style(yellow line). If neither the named style nor its ancestors (red lines) define the attribute, go on with 3.
3. the attribute is searched in the next physical style in the hierarchy (follow blue lines). If it’s not defined in any of those styles, go to 4.
4. Use a compiled-in default value.

The documents default style will define all attributes, so rule 4. is never reached.

Ok, that’s all. Complicated? Probably, but I still hope you like it 🙂

You might know that being an Open Source developer in general, and a Scribus developer in particular, allows you to travel all over the world, meet interesting people and have a nice dinner with them. This time I attended the GNOME Live! summit in Boston at the MIT. Thanks to Behdad Esfabod from Pango and Ed Trager from Unifont it also hosted the first FLOSS Text Layout Summit this year; and that was the reason for me being there (not  to prepare for porting Scribus to GNOME and not  for a tea party 🙂 ). Also attending where people from Qt, KWord, fontconfig, SuSE, DejaVu fonts, and SIL. So what was this about? You probably all had your share of font related problems, so it should be no surprise we talked about that — a little. The major part was concerned with international text layout. Ed Trager wrote a nice introduction on what this is about. If you are only familiar with Latin script, read in this article all about writing directions and how characters change their shape according to context. International layout usually includes the following steps:

  1. Itemize — find runs of characters which have the same script and font
  2. Shape — apply script specific substitutions and positioning
  3. Find linebreaks — most applications use the simple and straightforward approach, but Scribus will do something more sophisticated soon.
  4. Justify lines — by extending spaces, scaling glyphs, substituting glyphs or by script specific means, e.g. Kashida in Arabic.
  5. Render glyphs — that’s easy once you have all glyphs with their exact position and size.

Currently there exist three different libraries for international text layout (Pango, Qt and ICU) which all have their pros and cons. At the text layout summit we found out that we all share the common vision of a unified library. Why is that? Well, first of all, writing such a library is hard. Also you need a lot of people to check the results for each existing script. Second, there are still many scripts which are not supported yet, and the hope is to support scripts faster if we combine our efforts. Finally, users don’t appreciate if their native language works in some applications but not in others, just because they use different libraries. You can see this within Scribus if you compare for example Arabic text in StoryEditor and on canvas (The SE uses Qt and displays Arabic correctly).
But we also reached some decisions. As a first step we will write a unified shaping engine. I was surprised how fast the Qt and Pango developers agreed on a common datastructure for such a library (well, that’s only tentative, but I expected some long discussions there). Trolltech said they would consider relicensing their shaping code in Qt so it could be included in a common library. Finally we decided to develop the unified shaping engine on top of HarfBuzz. HarfBuzz is a library which handles OpenType tables and is already used by both Qt and Pango. The unified shaping engine will also be called HarfBuzz and will initially support OpenType fonts. Legacy Type1 and TTF fonts can be used for basic shaping in Latin script and for Arabic if they contain the necessary Arabic presentation forms. Later we also want to integrate Graphite, a new open source font technology from SIL, and platform specific glue code for AAT (Macintosh) and UniScribe (Windows).

Ah yes, in the future Scribus will use this unified shaping engine directly. KWord decided on using Qt, and most other applications will also use either Qt or Pango. There was also some interest from FireFox, OpenOffice, Inkscape and Abiword developers, but so far I can’t tell if they would use the new unified shaper directly or not.

Well, what else? We have a combined planning/agenda/minutes webpage. There are also links to some slides and audio recordings of most talks.

The organisation of the summit was not as good as LGM, though. Ok, there was free lunch (American pizza for all), but we had trouble to use a projector, had to organize audio recording ourselves, could only find (near dry) green and yellow pens for the whiteboard and worst of all: no coffee! At first I hoped I had discovered a very geeky coffee machine in one corner of the MIT media lab, but that was just a liquid nitrogen tank. 🙁

But all in all, very nice people, very geeky setting, very good dining, and interesting and productive talks. And I had ExScite as my private sightseeing guide on my last day, thanks ExScite! 🙂

It’s a common misconception that a font is a bunch of characters. That’s not so. Fonts (or, as Adobe likes to call them, font programs) contain glyphs. Glyphs are the shapes which are painted to screen or on paper. Characters are more abstract and convey some meaning, eg. the latin character ‘A’ or the diigit ‘7’. Even a single font can have more than one glyph for those characters, eg. a swash ‘A’ or an oldstyle numeral ‘7’.

Until now Scribus had the same sloppyness regarding characters and glyphs. Internally Scribus stores Unicode characters and for output it used to map each character to exactly one glyph. I’ve started to change that, though. Here are some use cases where it might be necessary to do something else:

  • using different glyphs depending on style, eg. SmallCaps or OldStyleNumerals
  • mapping the pagenumber placeholder to several digits
  • smart hyphens don’t have an associated character stored with them
  • ligatures

So here’s the new scheme:

Scribus still stores a sequence of Unicode characters. For output, two more sequences are generated: a sequence of items and a sequence of glyphs. The items provide a glue between the unicode chars and the glyphs. The smallest possible item is a cluster, that is a character and all its marks, e.g. ‘a’ + ‘^’. This would be connected to the glyph ‘â’. The other way is also possible: the unicode ‘â’ is mapped to two glyphs ‘a’ and ‘^’ which are positioned in just the right way. Usually an item will correspond to a syllable, though. The details of items are only relevant to Scribus’s layout engine, so I won’t go into that here. Just keep in mind that items are used to break the 1:1 correspondence between characters (what you type) and glyphs (what you see).

The item and glyph sequences are created during layout. At the same time the exact position and size of each glyph is calculated. A sideeffect of this is that Scribus now uses the same data for screen, PS and PDF output (before that the PS and PDF output routines did part of the layout themselves — bad idea).

The current code is still limited. Currently it’s only used for hyphens and pagenumbers. The next step will be to use the new NLS data structures and take advantage of OpenType features. So stay tuned: true smallcaps, ligatures and oldstyle figures are on their way…

Last week I implemented “optical margins” in 1.3.4cvs. So I’ll use my first blog entry to describe what that’s all about.

When you look at a newspaper, or a Scribus document with justfied text, you might notice that the left margin looks much straighter than the right one, even if all chars line up geometrically. That’s because most characters line up nicely at the beginning of a line, but a lot of characters leave an uneven appearance at the end of the line. Most notorious are punctuation characters: hyphen, comma, dot, quotation mark, and others.

An old technique to repair this is called “hanging punctuation”. This moves the offending punctuation marks into the margin. Unfortunately, this doesn’t look quite right (you can verify this if you activate that option in InDesign). The solution is to move the punctuation marks only to 50%-70% into the margin. This is how it works in Scribus 1.3.4cvs (PDF, sla). I copied the exact values from Hàn Thê´ Thành’s PhD thesis “Microtypographic extensions to the TEX typesetting system”. He implemented this technique in pdftex.

But also characters like ‘f’ and ‘r’ contribute to the uneven appearance of right margins. Moving those characters (very slightly) into the margin is called “margin kerning”. Since characters might have very different shapes in different fonts, I don’t use fixed percentages to move those into the margin. Instead I use a trick: I use the font’s kerning information for the pairs “f.”, “r.”, and so on. Maybe in the future font designers will include information for margin kerning explicitly in their fonts, but for now I’m quite content with my implementation. I only use margin kerning for the right margin, though. The left margin is usually quite smooth, except for characters like ‘A’, ‘W’ and ‘V’.

Currently there’s no option to deactivate optical margins in Scribus. I’ll add that later, and probably also options to finetune it (change the percentages for hanging punctuation per font, activate margin kerning for the left margin).

So tell me what you think about it and how I can improve it further.

Is that all? No. First, I fixed a bug in Scribus 1.3.3 which messed up right margins by placing hyphens irregularly. Hey, all that work with optical margins is for nothing if Scribus places hyphens randomly anyway! 😉
Then, Hàn Thê´ Thành describes another technique called “glyph extension”. This stretches or shrinks characters by -2% to +2% in a line, in order to avoid large white spaces between the words. This glyph extension won’t be visible to the naked eye but will help to get a more even word spacing.
Now, speaking of word spacing, isn’t TEX supposed to have a much better algorithm for linebreaking, which was also adapted to InDesign? Yep. TEX and InDesign optimize linebreaks for a whole paragraph. That results in a much more even word spacing than is achieved by the straight forward algorithm used by Scribus and other layout programs. Used to be used by Scribus, I should say, since we have a prototype of a new text layout system which supports this optimisation. Currently, the prototype is lacking some standard features of the old layout system (like non-justified alignment, tabs, drop caps, and some linespacing options). Once this is done we’ll merge it into cvs, so there’s something you can look forward to!

Ciao for now
/Andreas